市值: $2.1773T -4.14%
成交额(24h): $113.037B 39.30%
  • 市值: $2.1773T -4.14%
  • 成交额(24h): $113.037B 39.30%
  • 恐惧与贪婪指数:
  • 市值: $2.1773T -4.14%
加密货币
话题
百科
资讯
加密话题
视频
热门新闻
加密货币
话题
百科
资讯
加密话题
视频
bitcoin
bitcoin

$64289.441012 USD

-4.30%

ethereum
ethereum

$1790.079831 USD

-5.61%

tether
tether

$0.999340 USD

0.03%

bnb
bnb

$589.806756 USD

-4.33%

usd-coin
usd-coin

$0.999720 USD

-0.01%

xrp
xrp

$0.954925 USD

-3.07%

solana
solana

$70.338246 USD

-6.67%

tron
tron

$0.307198 USD

-1.65%

hyperliquid
hyperliquid

$65.416106 USD

-5.80%

zcash
zcash

$899.101757 USD

-1.32%

monero
monero

$491.132124 USD

-7.27%

dogecoin
dogecoin

$0.057920 USD

-7.54%

unus-sed-leo
unus-sed-leo

$9.119156 USD

-1.59%

chainlink
chainlink

$9.107922 USD

-4.95%

cardano
cardano

$0.150722 USD

-7.52%

加密货币新闻

风暴:多模式LLM的时空令牌减少

2025/03/11 15:24

使用AI了解视频需要有效地处理图像序列。当前基于视频的AI模型的一个主要挑战是他们无法将视频作为连续流动,缺少重要的运动细节和破坏连续性。缺乏时间建模可以阻止追踪变化。因此,事件和互动是部分未知的。

风暴:多模式LLM的时空令牌减少

Current video-based AI models face challenges in processing videos as a continuous flow, leading to a lack of motion details and continuity. This limitation hinders the models in performing well in tasks that require temporal modeling, such as tracing changes or events, and interactions are often partially identified.

当前基于视频的AI模型在处理视频时面临着挑战,这是连续流动的,导致运动细节和连续性缺乏。这种限制会阻碍模型在需要时间建模的任务中表现良好,例如追踪变化或事件,并且相互作用通常会部分识别。

Moreover, processing long videos becomes computationally demanding, necessitating techniques like frame skipping to reduce the burden. However, these techniques discard valuable information and decrease accuracy. Furthermore, the dense structure of data within frames does not compress well, resulting in redundancy and inefficient use of computational resources.

此外,处理较长的视频在计算范围内变得要求很高,需要采用诸如帧跳过以减轻负担的技术。但是,这些技术丢弃了有价值的信息并降低准确性。此外,帧中数据的密集结构并不能很好地压缩,从而导致冗余和计算资源的效率低下。

Existing video-language models typically treat videos as static frame sequences with image encoders and vision-language projectors. This approach struggles to represent motion and continuity effectively. It also places the burden of inferring temporal relations on the language model, leading to partial comprehension of the video content.

现有的视频语言模型通常将视频视为带有图像编码器和视觉图像投影仪的静态框架序列。这种方法难以有效地表示运动和连续性。它还将推断时间关系的负担放在语言模型上,从而部分理解视频内容。

To address the computational complexity of processing long videos, researchers have explored subsampling of frames, which reduces the workload at the expense of removing useful details that affect accuracy. Token reduction methods, such as recursive KV cache compression and frame selection, have also been investigated but add complexity without significant performance improvement.

为了解决处理长视频的计算复杂性,研究人员探索了框架的子采样,这减少了工作量,而牺牲了影响准确性的有用细节。也已经研究了递归KV缓存压缩和框架选择之类的令牌减少方法,但也已被研究,但增加了复杂性而没有显着改善。

Advanced video encoders and pooling methods have been developed to extract richer spatiotemporal features and facilitate efficient processing. However, these techniques are not yet sufficiently efficient and scalable, rendering the processing of long videos computationally intensive.

已经开发出高级视频编码器和合并方法来提取更丰富的时空特征并促进有效的处理。但是,这些技术尚未足够高效且可扩展,从而使长期视频的处理在计算上进行了密集。

In their quest to overcome these challenges, researchers from NVIDIA, Rutgers University, UC Berkeley, MIT, Nanjing University, and KAIST present STORM (Spatiotemporal Token Reduction for Multimodal LLMs), a Mamba-based temporal projector architecture. Unlike prior methods where temporal relations are inferred separately on each video frame and language models are utilized for this purpose, STORM integrates temporal information at the video tokens level to eliminate computational redundancy and maximize efficiency. The model enhances video representations with a bidirectional spatiotemporal scanning mechanism, effectively mitigating the burden of temporal reasoning from the LLM.

为了克服这些挑战,NVIDIA,Rutgers University,UC Berkeley,MIT,Nanjing University和Kaist现在的Storm(多模式LLMS时空令牌减少),基于MAMBA的临时投影仪架构。与先前的方法不同的方法在每个视频框架上分别推断出时间关系,而语言模型则用于此目的,Storm在视频令牌级别集成了时间信息,以消除计算冗余并最大程度地提高效率。该模型通过双向时空扫描机制增强了视频表示,有效地减轻了LLM的时间推理负担。

The framework utilizes Mamba layers for efficient temporal modeling, incorporating a bidirectional scanning module to capture dependencies across spatial and temporal dimensions. The temporal encoder processes the image and video inputs differently, acting as a spatial scanner for images to integrate global spatial context and as a spatiotemporal scanner for videos to capture temporal dynamics.

该框架利用MAMBA层进行有效的时间建模,并结合了双向扫描模块来捕获跨空间和时间维度的依赖项。时间编码器对图像和视频输入进行了不同的处理,作为图像的空间扫描仪,用于整合全局空间上下文,并作为视频的时空示例来捕获时间动力学。

During training, token compression techniques are applied to improve computational efficiency while maintaining essential information, enabling the inference of large models on a single GPU. At test time, a training-free token subsampling technique is introduced to further reduce computational burdens while preserving important temporal details. This technique facilitates efficient processing of long videos without requiring specialized equipment or deep adaptations of the model.

在训练过程中,将令牌压缩技术应用于提高计算效率的同时维持基本信息,从而可以在单个GPU上推断大型模型。在测试时,引入了一种无培训的令牌亚采样技术,以进一步减少计算负担,同时保留重要的时间细节。这种技术可促进长期视频的有效处理,而无需专门的设备或模型的深层改编。

To evaluate the STORM model for video understanding, the researchers perform experiments with an efficient setup. Training is performed using pre-trained SigLIP models, and a temporal projector is introduced through random initialization. The process is divided into two stages: an alignment stage, where the image encoder and LLM are frozen while only the temporal projector is trained using image-text pairs, and a supervised fine-tuning (SFT) stage with a diverse dataset of 12.5 million samples, including text, image-text, and video-text data.

为了评估风暴模型以进行视频理解,研究人员通过有效的设置进行实验。使用预先训练的siglip模型进行训练,并通过随机初始化引入颞投影仪。该过程分为两个阶段:一个对齐阶段,其中图像编码器和LLM被冷冻,而仅使用图像文本对培训了临时投影仪,以及有监督的微调(SFT)阶段,具有1,250万个样品的多样化数据集,包括文本,图像text和Video-Text数据。

Token compression methods, such as temporal and spatial pooling, are applied to decrease the computational burden. The final model is evaluated on several long-video benchmarks, including EgoSchema, MVBench, MLVU, LongVideoBench, and VideoMME, and its performance is compared with other prominent video LLMs, such as VILA, MERLOT, and CLIP.

将令牌压缩方法(例如时间和空间池)应用以减轻计算负担。最终模型将在几个长期基准测试中进行评估,包括Egoschema,MVBench,MLVU,LongVideObench和VideMomme,并将其性能与其他著名的视频LLM(例如Vila,Merlot和Clip)进行了比较。

The evaluation results demonstrate that STORM outperforms existing models, achieving state-of-the-art results on standard benchmarks. The Mamba module significantly improves efficiency by compressing visual tokens while preserving key information, reducing the inference time by up to 65.5%. Among the tested token reduction methods, temporal pooling shows the best performance on long videos, enabling the optimization of performance with a small number of tokens.

评估结果表明,风暴的表现优于现有模型,在标准基准上取得了最新的结果。 MAMBA模块通过压缩视觉令牌在保留关键信息的同时,可以显着提高效率,从而将推理时间最多减少65.5%。在经过测试的令牌减少方法中,时间池在长视频中显示出最佳性能,从而可以使用少量令牌优化性能。

In particular, STORM shows a significant improvement over the VILA baseline model, especially in tasks that require understanding the global context of the video. The analysis of the results highlights the crucial role of Mamba for optimized token compression, with performance boosts increasing proportionally with the video length, from 8 to 128 frames.

特别是,暴风雨比Vila基线模型有了显着改善,尤其是在需要了解视频全球环境的任务中。对结果的分析突出了Mamba在优化令牌压缩方面的关键作用,并且性能的提升与视频长度的增长率从8帧增加到128帧。

In conclusion, the researchers propose STORM, a Mamba-based temporal encoder architecture, for efficient processing of long videos. This model integrates temporal information at the video tokens level and introduces a bidirectional scanning module in the Mamba layer to capture spatiotemporal dependencies.

总之,研究人员建议Storm是一种基于MAMBA的时间编码器架构,用于有效地处理长视频。该模型在视频令牌级别集成了时间信息,并在Mamba层中引入了双向扫描模块,以捕获时空依赖性。

The researchers apply token compression techniques during training and a training-free token subsampling method during test time to optimize computational efficiency. Their experiments demonstrate that STORM outperforms existing models and achieves state-of-the-art results on various long-video benchmarks.

研究人员在训练过程中应用令牌压缩技术,并在测试时间内采用无训练的令牌子采样方法来优化计算效率。他们的实验表明,风暴的表现优于现有模型,并在各种长期基准测试基准上实现了最先进的结果。

原文来源:marktechpost

免责声明:info@kdj.com

所提供的信息并非交易建议。根据本文提供的信息进行的任何投资,kdj.com不承担任何责任。加密货币具有高波动性,强烈建议您深入研究后,谨慎投资!

如您认为本网站上使用的内容侵犯了您的版权,请立即联系我们(info@kdj.com),我们将及时删除。

2026年11月16日 发表的其他文章