市值: $2.184T 0.31%
體積(24小時): $69.2158B -38.77%
  • 市值: $2.184T 0.31%
  • 體積(24小時): $69.2158B -38.77%
  • 恐懼與貪婪指數:
  • 市值: $2.184T 0.31%
加密
主題
加密植物
資訊
加密術
影片
頭號新聞
加密
主題
加密植物
資訊
加密術
影片
bitcoin
bitcoin

$64566.812908 USD

0.23%

ethereum
ethereum

$1810.474514 USD

0.98%

tether
tether

$0.999022 USD

-0.03%

bnb
bnb

$590.524043 USD

0.06%

usd-coin
usd-coin

$0.999830 USD

0.01%

xrp
xrp

$0.951385 USD

-0.53%

solana
solana

$70.791567 USD

0.57%

tron
tron

$0.302344 USD

-1.59%

hyperliquid
hyperliquid

$66.043751 USD

0.99%

zcash
zcash

$884.280716 USD

-1.46%

dogecoin
dogecoin

$0.058467 USD

0.99%

monero
monero

$482.505734 USD

-1.49%

unus-sed-leo
unus-sed-leo

$9.190973 USD

0.78%

chainlink
chainlink

$9.081451 USD

-0.25%

cardano
cardano

$0.151877 USD

0.97%

加密貨幣新聞文章

Unitok:統一的離散視覺令牌器,用於視覺生成和理解

2025/03/02 05:03

旨在將視覺生成和理解統一到單個框架中的研究人員,多模式人工智能正在迅速發展。

Unitok:統一的離散視覺令牌器,用於視覺生成和理解

Researchers are working tirelessly to unify visual generation and understanding into a single, efficient framework. However, this goal remains elusive due to the distinct requirements of these two domains. Generative models prioritize the reconstruction of fine-grained image details, while understanding models focus on aligning visual features with textual representations at a high-level semantic granularity.

研究人員正在不懈地努力將視覺產生和理解統一為一個有效的框架。但是,由於這兩個領域的不同要求,該目標仍然難以捉摸。生成模型優先考慮細粒圖像細節的重建,同時了解模型的重點是將視覺特徵與高級語義粒度的文本表示對齊。

The challenge lies in integrating both capabilities without compromising performance. As autoregressive modeling gains traction in visual tasks, the demand for a unified tokenizer that seamlessly bridges these objectives is growing. However, current approaches often specialize in either image generation or understanding. Generative models like VQVAE encode image details efficiently but struggle to align visual features with textual representations. Conversely, models like CLIP excel at semantic alignment but lack the fine-grained detail needed for high-quality image reconstruction.

挑戰在於整合兩種功能而不會損害性能。隨著自回歸建模在視覺任務中獲得吸引力,對這些目標無縫橋接的統一令牌的需求正在增長。但是,當前的方法通常專門研究圖像產生或理解。諸如VQVAE之類的生成模型有效地編碼圖像詳細信息,但很難將視覺特徵與文本表示相結合。相反,諸如夾子在語義對齊時Excel之類的模型缺少高質量圖像重建所需的細節細節。

This misalignment creates inefficiencies and makes it difficult to develop multimodal models that can generate and interpret images equally proficiently. Researchers have been exploring ways to resolve these conflicts, but existing solutions often involve increasing model complexity without fundamentally addressing the tokenization gap.

這種未對準會導致效率低下,並且使得很難開發可以同樣能夠熟練地生成和解釋圖像的多模型。研究人員一直在探索解決這些衝突的方法,但是現有的解決方案通常涉及增加模型的複雜性,而無需從根本上解決令牌化差距。

Current methods attempt to bridge this gap by integrating separate tokenization strategies for different tasks. Some models incorporate contrastive learning into generative tokenizers to enhance semantic consistency. However, these techniques introduce training conflicts that degrade model performance. Many solutions rely on large codebooks to increase token representation capacity, but excessive codebook expansion leads to underutilization and inefficiencies.

當前的方法試圖通過整合不同任務的單獨的令牌化策略來彌合這一差距。一些模型將對比度學習納入生成引物,以增強語義一致性。但是,這些技術引入了培訓衝突,以降低模型性能。許多解決方案依靠大型代碼書來提高令牌表示能力,但是過多的代碼簿擴展導致缺乏利用和效率低下。

To address these issues, a research team from The University of Hong Kong, ByteDance Inc., and Huazhong University of Science and Technology introduced UniTok, a discrete visual tokenizer designed to unify visual generation and understanding. Their method expands token representation without causing optimization instability through multi-codebook quantization, which structures vector quantization into independent sub-codebooks, enabling better representation of visual features across tasks.

為了解決這些問題,香港大學,Bytedance Inc.和Huazhong科學技術大學的研究團隊介紹了Unitok,這是一個旨在統一視覺生成和理解的離散視覺令牌。他們的方法擴展了代幣表示,而不會通過多編碼書量化引起優化的不穩定性,該量化量化了矢量量化為獨立的子編碼書,從而可以更好地表示跨任務的視覺特徵。

By addressing the fundamental bottleneck of limited discrete token capacity, UniTok achieves superior image reconstruction and semantic alignment performance. The UniTok model employs a unified training paradigm integrating reconstruction and contrastive learning objectives. The core innovation lies in multi-codebook quantization, where visual tokens are divided into multiple independent sub-codebooks instead of a single large one. This increases the representation space while maintaining computational efficiency.

通過解決有限離散令牌容量的基本瓶頸,Unitok實現了出色的圖像重建和語義對準性能。 Unitok模型採用統一的培訓範式整合重建和對比度學習目標。核心創新在於多編碼書量化,其中可視代幣分為多個獨立的子代碼書而不是單個大型代碼書。這增加了表示空間,同時保持計算效率。

UniTok also incorporates attention-based factorization, which enhances token expressiveness by preserving semantic information during compression. Unlike conventional vector quantization, this approach prevents loss conflicts and improves token utilization, ensuring visual features are accurately encoded for generative and discriminative tasks.

Unitok還結合了基於注意力的分解,從而通過在壓縮過程中保留語義信息來增強令牌表現力。與傳統的向量量化不同,此方法可以防止損失衝突並改善令牌利用率,從而確保視覺特徵被準確地編碼為生成和歧視性任務。

The researchers trained UniTok on DataComp-1B, a dataset containing 1.28 billion image-text pairs, with images resized to a resolution of 256×256. The tokenizer was tested under two conditions: with pretrained CLIP weight initialization and without it. The model’s ability to achieve high zero-shot accuracy in both cases demonstrates the effectiveness of its unified design.

研究人員在DataComp-1b上培訓了Unitok,該數據集包含12.8億張圖像文本對,圖像大小為256×256。在兩個條件下測試了令牌儀:審計的夾重量初始化,沒有它。在這兩種情況下,該模型實現高零射精精度的能力都證明了其統一設計的有效性。

Experimental evaluations on standard benchmarks confirm UniTok’s superiority over existing tokenizers. On ImageNet, UniTok achieves an rFID of 0.38 compared to 0.87 for SD-VAE, indicating a significant improvement in reconstruction quality. Further, it records a zero-shot classification accuracy of 78.6%, outperforming CLIP’s 76.2%.

對標準基準測試的實驗評估證實了Unitok對現有引物的優勢。在ImageNet上,Unitok的RFID為0.38,而SD-VAE的RFID為0.87,這表明重建質量有顯著提高。此外,它記錄了78.6%的零射擊分類精度,表現優於夾子的76.2%。

The model also excels in visual question-answering benchmarks, surpassing VILA-U in accuracy by 3.3% on TextVQA and demonstrating a significant improvement of 112 points on the MME-Perception scores. These results highlight UniTok’s ability to handle image generation and understanding effectively.

該模型在視覺提問的基準測試中還表現出色,在TextVQA上超過Vila-U的準確性超過3.3%,並在MME感知得分上證明了112分的顯著提高。這些結果突出了Unitok有效地處理圖像生成和理解的能力。

When tested on multimodal large language models, UniTok outperforms traditional VQVAE tokenizers and significantly reduces the gap between discrete and continuous tokenization approaches. The success of UniTok suggests that enhancing discrete token representation through multi-codebook quantization is a viable solution for unifying multimodal learning frameworks.

當在多模式大語言模型上進行測試時,Unitok的表現優於傳統的VQVAE令牌,並顯著減少了離散和連續令牌化方法之間的差距。 Unitok的成功表明,通過多編碼書量化增強離散令牌表示形式是統一多模式學習框架的可行解決方案。

UniTok marks a major advancement in the integration of visual generation and understanding. By resolving tokenization bottlenecks through multi-codebook quantization, researchers can achieve more effective multimodal learning. The research demonstrates that when optimized correctly, discrete tokenization can rival or even surpass continuous tokenization methods. This innovation lays the foundation for future improvements in multimodal AI, offering a scalable solution for large vision-language models.

Unitok標誌著視覺產生和理解的整合方面的重大進步。通過通過多代碼量化解決令牌化瓶頸,研究人員可以實現更有效的多模式學習。研究表明,當正確優化時,離散的令牌化可以競爭甚至超過連續的令牌化方法。這項創新為多模式AI的未來改進奠定了基礎,為大型視覺模型提供了可擴展的解決方案。

The strong experimental results validate UniTok as a promising approach for achieving seamless integration between visual generation and understanding, paving the way for more advanced multimodal models in the future.

強大的實驗結果將UNITOK驗證為實現視覺產生和理解之間無縫整合的一種有希望的方法,為將來更高級的多模型模型鋪平了道路。

原始來源:marktechpost

免責聲明:info@kdj.com

所提供的資訊並非交易建議。 kDJ.com對任何基於本文提供的資訊進行的投資不承擔任何責任。加密貨幣波動性較大,建議您充分研究後謹慎投資!

如果您認為本網站使用的內容侵犯了您的版權,請立即聯絡我們(info@kdj.com),我們將及時刪除。

2026年11月16日 其他文章發表於