|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Cryptocurrency News Articles
STORM: Spatiotemporal Token Reduction for Multimodal LLMs
Mar 11, 2025 at 03:24 pm
Understanding videos with AI requires handling sequences of images efficiently. A major challenge in current video-based AI models is their inability to process videos as a continuous flow, missing important motion details and disrupting continuity. This lack of temporal modeling prevents tracing changes; therefore, events and interactions are partially unknown.

Current video-based AI models face challenges in processing videos as a continuous flow, leading to a lack of motion details and continuity. This limitation hinders the models in performing well in tasks that require temporal modeling, such as tracing changes or events, and interactions are often partially identified.
Moreover, processing long videos becomes computationally demanding, necessitating techniques like frame skipping to reduce the burden. However, these techniques discard valuable information and decrease accuracy. Furthermore, the dense structure of data within frames does not compress well, resulting in redundancy and inefficient use of computational resources.
Existing video-language models typically treat videos as static frame sequences with image encoders and vision-language projectors. This approach struggles to represent motion and continuity effectively. It also places the burden of inferring temporal relations on the language model, leading to partial comprehension of the video content.
To address the computational complexity of processing long videos, researchers have explored subsampling of frames, which reduces the workload at the expense of removing useful details that affect accuracy. Token reduction methods, such as recursive KV cache compression and frame selection, have also been investigated but add complexity without significant performance improvement.
Advanced video encoders and pooling methods have been developed to extract richer spatiotemporal features and facilitate efficient processing. However, these techniques are not yet sufficiently efficient and scalable, rendering the processing of long videos computationally intensive.
In their quest to overcome these challenges, researchers from NVIDIA, Rutgers University, UC Berkeley, MIT, Nanjing University, and KAIST present STORM (Spatiotemporal Token Reduction for Multimodal LLMs), a Mamba-based temporal projector architecture. Unlike prior methods where temporal relations are inferred separately on each video frame and language models are utilized for this purpose, STORM integrates temporal information at the video tokens level to eliminate computational redundancy and maximize efficiency. The model enhances video representations with a bidirectional spatiotemporal scanning mechanism, effectively mitigating the burden of temporal reasoning from the LLM.
The framework utilizes Mamba layers for efficient temporal modeling, incorporating a bidirectional scanning module to capture dependencies across spatial and temporal dimensions. The temporal encoder processes the image and video inputs differently, acting as a spatial scanner for images to integrate global spatial context and as a spatiotemporal scanner for videos to capture temporal dynamics.
During training, token compression techniques are applied to improve computational efficiency while maintaining essential information, enabling the inference of large models on a single GPU. At test time, a training-free token subsampling technique is introduced to further reduce computational burdens while preserving important temporal details. This technique facilitates efficient processing of long videos without requiring specialized equipment or deep adaptations of the model.
To evaluate the STORM model for video understanding, the researchers perform experiments with an efficient setup. Training is performed using pre-trained SigLIP models, and a temporal projector is introduced through random initialization. The process is divided into two stages: an alignment stage, where the image encoder and LLM are frozen while only the temporal projector is trained using image-text pairs, and a supervised fine-tuning (SFT) stage with a diverse dataset of 12.5 million samples, including text, image-text, and video-text data.
Token compression methods, such as temporal and spatial pooling, are applied to decrease the computational burden. The final model is evaluated on several long-video benchmarks, including EgoSchema, MVBench, MLVU, LongVideoBench, and VideoMME, and its performance is compared with other prominent video LLMs, such as VILA, MERLOT, and CLIP.
The evaluation results demonstrate that STORM outperforms existing models, achieving state-of-the-art results on standard benchmarks. The Mamba module significantly improves efficiency by compressing visual tokens while preserving key information, reducing the inference time by up to 65.5%. Among the tested token reduction methods, temporal pooling shows the best performance on long videos, enabling the optimization of performance with a small number of tokens.
In particular, STORM shows a significant improvement over the VILA baseline model, especially in tasks that require understanding the global context of the video. The analysis of the results highlights the crucial role of Mamba for optimized token compression, with performance boosts increasing proportionally with the video length, from 8 to 128 frames.
In conclusion, the researchers propose STORM, a Mamba-based temporal encoder architecture, for efficient processing of long videos. This model integrates temporal information at the video tokens level and introduces a bidirectional scanning module in the Mamba layer to capture spatiotemporal dependencies.
The researchers apply token compression techniques during training and a training-free token subsampling method during test time to optimize computational efficiency. Their experiments demonstrate that STORM outperforms existing models and achieves state-of-the-art results on various long-video benchmarks.
Disclaimer:info@kdj.com
The information provided is not trading advice. kdj.com does not assume any responsibility for any investments made based on the information provided in this article. Cryptocurrencies are highly volatile and it is highly recommended that you invest with caution after thorough research!
If you believe that the content used on this website infringes your copyright, please contact us immediately (info@kdj.com) and we will delete it promptly.
-
-
-
- DTCC Embraces Multichain Future: What It Means for Solana and the XRP Ledger
- Nov 16, 2026 at 04:05 am
- DTCC is expanding its tokenization strategy across multiple blockchain networks, with Solana securing a key integration. This multichain approach reshapes the landscape for digital assets, prompting questions about the XRP Ledger's role in this evolving financial infrastructure.
-
- Sui Network, Toncoin, TRON: Navigating the Evolving Crypto Landscape
- Nov 14, 2026 at 11:55 pm
- Amidst market shifts, Sui Network, Toncoin, and TRON reveal distinct paths for growth and challenges. This report dissects their latest movements, from TRON's account surge to Sui's ecosystem adjustments and Toncoin's Telegram integration, offering insights for the discerning investor.
-
- OKX Europe Boosts Fast Redemption Limits for VIP Liquid Staking, Setting Industry Pace
- Nov 14, 2026 at 03:45 pm
- OKX Europe elevates daily fast redemption limits for VIP liquid staking token holders (BETH and OKSOL), enhancing liquidity access and user experience, while the broader liquid staking landscape sees innovations like Faroo's stPROS.
-
-
- Pyth Network, Douro Labs, and CFTC Regulation: Shaping the Future of Crypto Market Integrity
- Nov 14, 2026 at 08:05 am
- Douro Labs, a key contributor to Pyth Network, is actively engaging with CFTC on crypto market rules, advocating for multi-source pricing and robust oracle evaluation to prevent market manipulation and protect investors.
-
-

































