|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
使用AI了解視頻需要有效地處理圖像序列。當前基於視頻的AI模型的一個主要挑戰是他們無法將視頻作為連續流動,缺少重要的運動細節和破壞連續性。缺乏時間建模可以阻止追踪變化。因此,事件和互動是部分未知的。

Current video-based AI models face challenges in processing videos as a continuous flow, leading to a lack of motion details and continuity. This limitation hinders the models in performing well in tasks that require temporal modeling, such as tracing changes or events, and interactions are often partially identified.
當前基於視頻的AI模型在處理視頻時面臨著挑戰,這是連續流動的,導致運動細節和連續性缺乏。這種限制會阻礙模型在需要時間建模的任務中表現良好,例如追踪變化或事件,並且相互作用通常會部分識別。
Moreover, processing long videos becomes computationally demanding, necessitating techniques like frame skipping to reduce the burden. However, these techniques discard valuable information and decrease accuracy. Furthermore, the dense structure of data within frames does not compress well, resulting in redundancy and inefficient use of computational resources.
此外,處理較長的視頻在計算範圍內變得要求很高,需要採用諸如幀跳過以減輕負擔的技術。但是,這些技術丟棄了有價值的信息並降低準確性。此外,幀中數據的密集結構並不能很好地壓縮,從而導致冗餘和計算資源的效率低下。
Existing video-language models typically treat videos as static frame sequences with image encoders and vision-language projectors. This approach struggles to represent motion and continuity effectively. It also places the burden of inferring temporal relations on the language model, leading to partial comprehension of the video content.
現有的視頻語言模型通常將視頻視為帶有圖像編碼器和視覺圖像投影儀的靜態框架序列。這種方法難以有效地表示運動和連續性。它還將推斷時間關係的負擔放在語言模型上,從而部分理解視頻內容。
To address the computational complexity of processing long videos, researchers have explored subsampling of frames, which reduces the workload at the expense of removing useful details that affect accuracy. Token reduction methods, such as recursive KV cache compression and frame selection, have also been investigated but add complexity without significant performance improvement.
為了解決處理長視頻的計算複雜性,研究人員探索了框架的子採樣,這減少了工作量,而犧牲了影響準確性的有用細節。也已經研究了遞歸KV緩存壓縮和框架選擇之類的令牌減少方法,但也已被研究,但增加了複雜性而沒有顯著改善。
Advanced video encoders and pooling methods have been developed to extract richer spatiotemporal features and facilitate efficient processing. However, these techniques are not yet sufficiently efficient and scalable, rendering the processing of long videos computationally intensive.
已經開發出高級視頻編碼器和合併方法來提取更豐富的時空特徵並促進有效的處理。但是,這些技術尚未足夠高效且可擴展,從而使長期視頻的處理在計算上進行了密集。
In their quest to overcome these challenges, researchers from NVIDIA, Rutgers University, UC Berkeley, MIT, Nanjing University, and KAIST present STORM (Spatiotemporal Token Reduction for Multimodal LLMs), a Mamba-based temporal projector architecture. Unlike prior methods where temporal relations are inferred separately on each video frame and language models are utilized for this purpose, STORM integrates temporal information at the video tokens level to eliminate computational redundancy and maximize efficiency. The model enhances video representations with a bidirectional spatiotemporal scanning mechanism, effectively mitigating the burden of temporal reasoning from the LLM.
為了克服這些挑戰,NVIDIA,Rutgers University,UC Berkeley,MIT,Nanjing University和Kaist現在的Storm(多模式LLMS時空令牌減少),基於MAMBA的臨時投影儀架構。與先前的方法不同的方法在每個視頻框架上分別推斷出時間關係,而語言模型則用於此目的,Storm在視頻令牌級別集成了時間信息,以消除計算冗餘並最大程度地提高效率。該模型通過雙向時空掃描機制增強了視頻表示,有效地減輕了LLM的時間推理負擔。
The framework utilizes Mamba layers for efficient temporal modeling, incorporating a bidirectional scanning module to capture dependencies across spatial and temporal dimensions. The temporal encoder processes the image and video inputs differently, acting as a spatial scanner for images to integrate global spatial context and as a spatiotemporal scanner for videos to capture temporal dynamics.
該框架利用MAMBA層進行有效的時間建模,並結合了雙向掃描模塊來捕獲跨空間和時間維度的依賴項。時間編碼器對圖像和視頻輸入進行了不同的處理,作為圖像的空間掃描儀,用於整合全局空間上下文,並作為視頻的時空示例來捕獲時間動力學。
During training, token compression techniques are applied to improve computational efficiency while maintaining essential information, enabling the inference of large models on a single GPU. At test time, a training-free token subsampling technique is introduced to further reduce computational burdens while preserving important temporal details. This technique facilitates efficient processing of long videos without requiring specialized equipment or deep adaptations of the model.
在訓練過程中,將令牌壓縮技術應用於提高計算效率的同時維持基本信息,從而可以在單個GPU上推斷大型模型。在測試時,引入了一種無培訓的令牌亞採樣技術,以進一步減少計算負擔,同時保留重要的時間細節。這種技術可促進長期視頻的有效處理,而無需專門的設備或模型的深層改編。
To evaluate the STORM model for video understanding, the researchers perform experiments with an efficient setup. Training is performed using pre-trained SigLIP models, and a temporal projector is introduced through random initialization. The process is divided into two stages: an alignment stage, where the image encoder and LLM are frozen while only the temporal projector is trained using image-text pairs, and a supervised fine-tuning (SFT) stage with a diverse dataset of 12.5 million samples, including text, image-text, and video-text data.
為了評估風暴模型以進行視頻理解,研究人員通過有效的設置進行實驗。使用預先訓練的siglip模型進行訓練,並通過隨機初始化引入顳投影儀。該過程分為兩個階段:一個對齊階段,其中圖像編碼器和LLM被冷凍,而僅使用圖像文本對培訓了臨時投影儀,以及有監督的微調(SFT)階段,具有1,250萬個樣品的多樣化數據集,包括文本,圖像text和Video-Text數據。
Token compression methods, such as temporal and spatial pooling, are applied to decrease the computational burden. The final model is evaluated on several long-video benchmarks, including EgoSchema, MVBench, MLVU, LongVideoBench, and VideoMME, and its performance is compared with other prominent video LLMs, such as VILA, MERLOT, and CLIP.
將令牌壓縮方法(例如時間和空間池)應用以減輕計算負擔。最終模型將在幾個長期基準測試中進行評估,包括Egoschema,MVBench,MLVU,LongVideObench和VideMomme,並將其性能與其他著名的視頻LLM(例如Vila,Merlot和Clip)進行了比較。
The evaluation results demonstrate that STORM outperforms existing models, achieving state-of-the-art results on standard benchmarks. The Mamba module significantly improves efficiency by compressing visual tokens while preserving key information, reducing the inference time by up to 65.5%. Among the tested token reduction methods, temporal pooling shows the best performance on long videos, enabling the optimization of performance with a small number of tokens.
評估結果表明,風暴的表現優於現有模型,在標準基准上取得了最新的結果。 MAMBA模塊通過壓縮視覺令牌在保留關鍵信息的同時,可以顯著提高效率,從而將推理時間最多減少65.5%。在經過測試的令牌減少方法中,時間池在長視頻中顯示出最佳性能,從而可以使用少量令牌優化性能。
In particular, STORM shows a significant improvement over the VILA baseline model, especially in tasks that require understanding the global context of the video. The analysis of the results highlights the crucial role of Mamba for optimized token compression, with performance boosts increasing proportionally with the video length, from 8 to 128 frames.
特別是,暴風雨比Vila基線模型有了顯著改善,尤其是在需要了解視頻全球環境的任務中。對結果的分析突出了Mamba在優化令牌壓縮方面的關鍵作用,並且性能的提升與視頻長度的增長率從8幀增加到128幀。
In conclusion, the researchers propose STORM, a Mamba-based temporal encoder architecture, for efficient processing of long videos. This model integrates temporal information at the video tokens level and introduces a bidirectional scanning module in the Mamba layer to capture spatiotemporal dependencies.
總之,研究人員建議Storm是一種基於MAMBA的時間編碼器架構,用於有效地處理長視頻。該模型在視頻令牌級別集成了時間信息,並在Mamba層中引入了雙向掃描模塊,以捕獲時空依賴性。
The researchers apply token compression techniques during training and a training-free token subsampling method during test time to optimize computational efficiency. Their experiments demonstrate that STORM outperforms existing models and achieves state-of-the-art results on various long-video benchmarks.
研究人員在訓練過程中應用令牌壓縮技術,並在測試時間內採用無訓練的令牌子採樣方法來優化計算效率。他們的實驗表明,風暴的表現優於現有模型,並在各種長期基準測試基准上實現了最先進的結果。
免責聲明:info@kdj.com
所提供的資訊並非交易建議。 kDJ.com對任何基於本文提供的資訊進行的投資不承擔任何責任。加密貨幣波動性較大,建議您充分研究後謹慎投資!
如果您認為本網站使用的內容侵犯了您的版權,請立即聯絡我們(info@kdj.com),我們將及時刪除。
-
-
-
-
-
-
- 比特幣零售需求因價格下跌而暴跌,技術指標確認看跌趨勢
- 2026-11-14 15:45:01
- Bitcoin零售需求銳減近10%至3月以來最低水平,伴隨價格跌破7萬美元,技術指標指向持續看跌。
-
-
-
- XRP 價格預測:1000 美元會讓大家變得富有嗎?技術面建議值得關注的關鍵水平
- 2026-11-14 04:05:01
- 分析 XRP 以 1,000 美元創造百萬富翁的潛力以及價格預測的關鍵技術水平。

































