Market Cap: $2.184T 0.31%
Volume(24h): $69.2158B -38.77%
  • Market Cap: $2.184T 0.31%
  • Volume(24h): $69.2158B -38.77%
  • Fear & Greed Index:
  • Market Cap: $2.184T 0.31%
Cryptos
Topics
Cryptospedia
News
CryptosTopics
Videos
Top News
Cryptos
Topics
Cryptospedia
News
CryptosTopics
Videos
bitcoin
bitcoin

$64566.812908 USD

0.23%

ethereum
ethereum

$1810.474514 USD

0.98%

tether
tether

$0.999022 USD

-0.03%

bnb
bnb

$590.524043 USD

0.06%

usd-coin
usd-coin

$0.999830 USD

0.01%

xrp
xrp

$0.951385 USD

-0.53%

solana
solana

$70.791567 USD

0.57%

tron
tron

$0.302344 USD

-1.59%

hyperliquid
hyperliquid

$66.043751 USD

0.99%

zcash
zcash

$884.280716 USD

-1.46%

dogecoin
dogecoin

$0.058467 USD

0.99%

monero
monero

$482.505734 USD

-1.49%

unus-sed-leo
unus-sed-leo

$9.190973 USD

0.78%

chainlink
chainlink

$9.081451 USD

-0.25%

cardano
cardano

$0.151877 USD

0.97%

Cryptocurrency News Articles

UniTok: A Unified Discrete Visual Tokenizer for Visual Generation and Understanding

Mar 02, 2025 at 05:03 am

With researchers aiming to unify visual generation and understanding into a single framework, multimodal artificial intelligence is evolving rapidly.

UniTok: A Unified Discrete Visual Tokenizer for Visual Generation and Understanding

Researchers are working tirelessly to unify visual generation and understanding into a single, efficient framework. However, this goal remains elusive due to the distinct requirements of these two domains. Generative models prioritize the reconstruction of fine-grained image details, while understanding models focus on aligning visual features with textual representations at a high-level semantic granularity.

The challenge lies in integrating both capabilities without compromising performance. As autoregressive modeling gains traction in visual tasks, the demand for a unified tokenizer that seamlessly bridges these objectives is growing. However, current approaches often specialize in either image generation or understanding. Generative models like VQVAE encode image details efficiently but struggle to align visual features with textual representations. Conversely, models like CLIP excel at semantic alignment but lack the fine-grained detail needed for high-quality image reconstruction.

This misalignment creates inefficiencies and makes it difficult to develop multimodal models that can generate and interpret images equally proficiently. Researchers have been exploring ways to resolve these conflicts, but existing solutions often involve increasing model complexity without fundamentally addressing the tokenization gap.

Current methods attempt to bridge this gap by integrating separate tokenization strategies for different tasks. Some models incorporate contrastive learning into generative tokenizers to enhance semantic consistency. However, these techniques introduce training conflicts that degrade model performance. Many solutions rely on large codebooks to increase token representation capacity, but excessive codebook expansion leads to underutilization and inefficiencies.

To address these issues, a research team from The University of Hong Kong, ByteDance Inc., and Huazhong University of Science and Technology introduced UniTok, a discrete visual tokenizer designed to unify visual generation and understanding. Their method expands token representation without causing optimization instability through multi-codebook quantization, which structures vector quantization into independent sub-codebooks, enabling better representation of visual features across tasks.

By addressing the fundamental bottleneck of limited discrete token capacity, UniTok achieves superior image reconstruction and semantic alignment performance. The UniTok model employs a unified training paradigm integrating reconstruction and contrastive learning objectives. The core innovation lies in multi-codebook quantization, where visual tokens are divided into multiple independent sub-codebooks instead of a single large one. This increases the representation space while maintaining computational efficiency.

UniTok also incorporates attention-based factorization, which enhances token expressiveness by preserving semantic information during compression. Unlike conventional vector quantization, this approach prevents loss conflicts and improves token utilization, ensuring visual features are accurately encoded for generative and discriminative tasks.

The researchers trained UniTok on DataComp-1B, a dataset containing 1.28 billion image-text pairs, with images resized to a resolution of 256×256. The tokenizer was tested under two conditions: with pretrained CLIP weight initialization and without it. The model’s ability to achieve high zero-shot accuracy in both cases demonstrates the effectiveness of its unified design.

Experimental evaluations on standard benchmarks confirm UniTok’s superiority over existing tokenizers. On ImageNet, UniTok achieves an rFID of 0.38 compared to 0.87 for SD-VAE, indicating a significant improvement in reconstruction quality. Further, it records a zero-shot classification accuracy of 78.6%, outperforming CLIP’s 76.2%.

The model also excels in visual question-answering benchmarks, surpassing VILA-U in accuracy by 3.3% on TextVQA and demonstrating a significant improvement of 112 points on the MME-Perception scores. These results highlight UniTok’s ability to handle image generation and understanding effectively.

When tested on multimodal large language models, UniTok outperforms traditional VQVAE tokenizers and significantly reduces the gap between discrete and continuous tokenization approaches. The success of UniTok suggests that enhancing discrete token representation through multi-codebook quantization is a viable solution for unifying multimodal learning frameworks.

UniTok marks a major advancement in the integration of visual generation and understanding. By resolving tokenization bottlenecks through multi-codebook quantization, researchers can achieve more effective multimodal learning. The research demonstrates that when optimized correctly, discrete tokenization can rival or even surpass continuous tokenization methods. This innovation lays the foundation for future improvements in multimodal AI, offering a scalable solution for large vision-language models.

The strong experimental results validate UniTok as a promising approach for achieving seamless integration between visual generation and understanding, paving the way for more advanced multimodal models in the future.

Original source:marktechpost

Disclaimer:info@kdj.com

The information provided is not trading advice. kdj.com does not assume any responsibility for any investments made based on the information provided in this article. Cryptocurrencies are highly volatile and it is highly recommended that you invest with caution after thorough research!

If you believe that the content used on this website infringes your copyright, please contact us immediately (info@kdj.com) and we will delete it promptly.

Other articles published on Nov 16, 2026