|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
旨在将视觉生成和理解统一到单个框架中的研究人员,多模式人工智能正在迅速发展。

Researchers are working tirelessly to unify visual generation and understanding into a single, efficient framework. However, this goal remains elusive due to the distinct requirements of these two domains. Generative models prioritize the reconstruction of fine-grained image details, while understanding models focus on aligning visual features with textual representations at a high-level semantic granularity.
研究人员正在不懈地努力将视觉产生和理解统一为一个有效的框架。但是,由于这两个领域的不同要求,该目标仍然难以捉摸。生成模型优先考虑细粒图像细节的重建,同时了解模型的重点是将视觉特征与高级语义粒度的文本表示对齐。
The challenge lies in integrating both capabilities without compromising performance. As autoregressive modeling gains traction in visual tasks, the demand for a unified tokenizer that seamlessly bridges these objectives is growing. However, current approaches often specialize in either image generation or understanding. Generative models like VQVAE encode image details efficiently but struggle to align visual features with textual representations. Conversely, models like CLIP excel at semantic alignment but lack the fine-grained detail needed for high-quality image reconstruction.
挑战在于整合两种功能而不会损害性能。随着自回归建模在视觉任务中获得吸引力,对这些目标无缝桥接的统一令牌的需求正在增长。但是,当前的方法通常专门研究图像产生或理解。诸如VQVAE之类的生成模型有效地编码图像详细信息,但很难将视觉特征与文本表示相结合。相反,诸如夹子在语义对齐时Excel之类的模型缺少高质量图像重建所需的细节细节。
This misalignment creates inefficiencies and makes it difficult to develop multimodal models that can generate and interpret images equally proficiently. Researchers have been exploring ways to resolve these conflicts, but existing solutions often involve increasing model complexity without fundamentally addressing the tokenization gap.
这种未对准会导致效率低下,并且使得很难开发可以同样能够熟练地生成和解释图像的多模型。研究人员一直在探索解决这些冲突的方法,但是现有的解决方案通常涉及增加模型的复杂性,而无需从根本上解决令牌化差距。
Current methods attempt to bridge this gap by integrating separate tokenization strategies for different tasks. Some models incorporate contrastive learning into generative tokenizers to enhance semantic consistency. However, these techniques introduce training conflicts that degrade model performance. Many solutions rely on large codebooks to increase token representation capacity, but excessive codebook expansion leads to underutilization and inefficiencies.
当前的方法试图通过整合不同任务的单独的令牌化策略来弥合这一差距。一些模型将对比度学习纳入生成引物,以增强语义一致性。但是,这些技术引入了培训冲突,以降低模型性能。许多解决方案依靠大型代码书来提高令牌表示能力,但是过多的代码簿扩展导致缺乏利用和效率低下。
To address these issues, a research team from The University of Hong Kong, ByteDance Inc., and Huazhong University of Science and Technology introduced UniTok, a discrete visual tokenizer designed to unify visual generation and understanding. Their method expands token representation without causing optimization instability through multi-codebook quantization, which structures vector quantization into independent sub-codebooks, enabling better representation of visual features across tasks.
为了解决这些问题,香港大学,Bytedance Inc.和Huazhong科学技术大学的研究团队介绍了Unitok,这是一个旨在统一视觉生成和理解的离散视觉令牌。他们的方法扩展了代币表示,而不会通过多编码书量化引起优化的不稳定性,该量化量化了矢量量化为独立的子编码书,从而可以更好地表示跨任务的视觉特征。
By addressing the fundamental bottleneck of limited discrete token capacity, UniTok achieves superior image reconstruction and semantic alignment performance. The UniTok model employs a unified training paradigm integrating reconstruction and contrastive learning objectives. The core innovation lies in multi-codebook quantization, where visual tokens are divided into multiple independent sub-codebooks instead of a single large one. This increases the representation space while maintaining computational efficiency.
通过解决有限离散令牌容量的基本瓶颈,Unitok实现了出色的图像重建和语义对准性能。 Unitok模型采用统一的培训范式整合重建和对比度学习目标。核心创新在于多编码书量化,其中可视代币分为多个独立的子代码书而不是单个大型代码书。这增加了表示空间,同时保持计算效率。
UniTok also incorporates attention-based factorization, which enhances token expressiveness by preserving semantic information during compression. Unlike conventional vector quantization, this approach prevents loss conflicts and improves token utilization, ensuring visual features are accurately encoded for generative and discriminative tasks.
Unitok还结合了基于注意力的分解,从而通过在压缩过程中保留语义信息来增强令牌表现力。与传统的向量量化不同,此方法可以防止损失冲突并改善令牌利用率,从而确保视觉特征被准确地编码为生成和歧视性任务。
The researchers trained UniTok on DataComp-1B, a dataset containing 1.28 billion image-text pairs, with images resized to a resolution of 256×256. The tokenizer was tested under two conditions: with pretrained CLIP weight initialization and without it. The model’s ability to achieve high zero-shot accuracy in both cases demonstrates the effectiveness of its unified design.
研究人员在DataComp-1b上培训了Unitok,该数据集包含12.8亿张图像文本对,图像大小为256×256。在两个条件下测试了令牌仪:审计的夹重量初始化,没有它。在这两种情况下,该模型实现高零射精精度的能力都证明了其统一设计的有效性。
Experimental evaluations on standard benchmarks confirm UniTok’s superiority over existing tokenizers. On ImageNet, UniTok achieves an rFID of 0.38 compared to 0.87 for SD-VAE, indicating a significant improvement in reconstruction quality. Further, it records a zero-shot classification accuracy of 78.6%, outperforming CLIP’s 76.2%.
对标准基准测试的实验评估证实了Unitok对现有引物的优势。在ImageNet上,Unitok的RFID为0.38,而SD-VAE的RFID为0.87,这表明重建质量有显着提高。此外,它记录了78.6%的零射击分类精度,表现优于夹子的76.2%。
The model also excels in visual question-answering benchmarks, surpassing VILA-U in accuracy by 3.3% on TextVQA and demonstrating a significant improvement of 112 points on the MME-Perception scores. These results highlight UniTok’s ability to handle image generation and understanding effectively.
该模型在视觉提问的基准测试中还表现出色,在TextVQA上超过Vila-U的准确性超过3.3%,并在MME感知得分上证明了112分的显着改善。这些结果突出了Unitok有效地处理图像生成和理解的能力。
When tested on multimodal large language models, UniTok outperforms traditional VQVAE tokenizers and significantly reduces the gap between discrete and continuous tokenization approaches. The success of UniTok suggests that enhancing discrete token representation through multi-codebook quantization is a viable solution for unifying multimodal learning frameworks.
当在多模式大语言模型上进行测试时,Unitok的表现优于传统的VQVAE令牌,并显着减少了离散和连续令牌化方法之间的差距。 Unitok的成功表明,通过多编码书量化增强离散令牌表示形式是统一多模式学习框架的可行解决方案。
UniTok marks a major advancement in the integration of visual generation and understanding. By resolving tokenization bottlenecks through multi-codebook quantization, researchers can achieve more effective multimodal learning. The research demonstrates that when optimized correctly, discrete tokenization can rival or even surpass continuous tokenization methods. This innovation lays the foundation for future improvements in multimodal AI, offering a scalable solution for large vision-language models.
Unitok标志着视觉产生和理解的整合方面的重大进步。通过通过多代码量化解决令牌化瓶颈,研究人员可以实现更有效的多模式学习。研究表明,当正确优化时,离散的令牌化可以竞争甚至超过连续的令牌化方法。这项创新为多模式AI的未来改进奠定了基础,为大型视觉模型提供了可扩展的解决方案。
The strong experimental results validate UniTok as a promising approach for achieving seamless integration between visual generation and understanding, paving the way for more advanced multimodal models in the future.
强大的实验结果将UNITOK验证为实现视觉产生和理解之间无缝整合的一种有希望的方法,为将来更高级的多模型模型铺平了道路。
免责声明:info@kdj.com
所提供的信息并非交易建议。根据本文提供的信息进行的任何投资,kdj.com不承担任何责任。加密货币具有高波动性,强烈建议您深入研究后,谨慎投资!
如您认为本网站上使用的内容侵犯了您的版权,请立即联系我们(info@kdj.com),我们将及时删除。
-
-
-
-
-
-
- 比特币零售需求因价格下跌而暴跌,技术指标确认看跌趋势
- 2026-11-14 15:45:01
- Bitcoin零售需求锐减近10%至3月以来最低水平,伴随价格跌破7万美元,技术指标指向持续看跌。
-
-
-
- XRP 价格预测:1000 美元会让每个人都变得富有吗?技术面建议值得关注的关键水平
- 2026-11-14 04:05:01
- 分析 XRP 以 1,000 美元创造百万富翁的潜力以及价格预测的关键技术水平。

































