Chameleon：無縫多模式文件創建的革命性模型

Meta 研究人員開發了 Chameleon，這是一種多模態基礎模型，可無縫整合圖像和文字序列。與分離模態的傳統模型不同，Chameleon 的統一架構將圖像（如文字）標記化，從而實現全面的文檔建模和跨模態的交錯推理。為了解決這種早期融合方法中的最佳化挑戰，研究人員引入了架構增強和訓練技術，從而在 Meta 的大規模資料集上成功進行了訓練。 Chameleon 的評估展示了在純文字任務中的競爭性表現，在常識推理和數學方面優於 LLaMa-2。在影像相關任務中，它在影像字幕和視覺問答方面表現出色，以更少的鏡頭超越或匹配更大的模型。

Chameleon: A Unified Foundation Model for Seamless Multimodal Document Modeling

Chameleon：無縫多模式文檔建模的統一基礎模型

Recent multimodal foundation models have revolutionized the field of artificial intelligence, enabling impressive advancements in tasks such as natural language processing, image recognition, and question answering. However, these models often treat different modalities separately, employing specific encoders or decoders for each. This approach inherently limits their ability to effectively fuse information across modalities and produce comprehensive multimodal documents that seamlessly integrate diverse sequences of images and text.

最近的多模態基礎模型徹底改變了人工智慧領域，在自然語言處理、影像辨識和問答等任務方面取得了令人矚目的進展。然而，這些模型通常單獨處理不同的模態，為每種模態採用特定的編碼器或解碼器。這種方法本質上限制了他們有效融合跨模態資訊並產生無縫整合不同影像和文字序列的全面多模態文件的能力。

Meta researchers have addressed this challenge by introducing Chameleon, a groundbreaking mixed-modal foundation model designed to facilitate seamless reasoning and generation with interleaved textual and image sequences. Unlike traditional models, Chameleon employs a unified architecture that treats both modalities equally by tokenizing images akin to text. This approach, termed early fusion, allows for seamless flow of information across modalities but also presents unique optimization challenges.

Meta 研究人員透過引入 Chameleon 解決了這項挑戰，Chameleon 是一種突破性的混合模式基礎模型，旨在促進交錯文字和圖像序列的無縫推理和生成。與傳統模型不同，Chameleon 採用統一的架構，透過將圖像標記為類似於文字的方式，平等地對待這兩種模式。這種方法被稱為早期融合，允許跨模式的資訊無縫流動，但也帶來了獨特的最佳化挑戰。

To overcome these challenges, the researchers have incorporated architectural enhancements and innovative training techniques into Chameleon's design. By adapting transformer architecture and fine-tuning strategies, they have laid the foundation for a model capable of addressing the complexities of multimodal document modeling.

為了克服這些挑戰，研究人員將架構增強和創新訓練技術融入 Chameleon 的設計中。透過調整變壓器架構和微調策略，他們為能夠解決多模式文件建模複雜性的模型奠定了基礎。

To capture the visual information effectively, they developed a novel image tokenizer that encodes high-resolution 512 × 512 images into 1024 tokens from an 8192-codebook. This tokenization process ensures that images are represented in a manner compatible with the text tokens, facilitating seamless reasoning across modalities. They further refined their tokenizer by introducing a BPE tokenizer with a 65,536-vocabulary, including image tokens, trained using the sentencepiece library. This enhanced tokenizer improved the model's performance and stability.

為了有效地捕捉視覺訊息，他們開發了一種新穎的影像標記器，將高解析度 512 × 512 影像編碼為 8192 碼本中的 1024 個標記。此標記化過程可確保圖像以與文字標記相容的方式表示，從而促進跨模式的無縫推理。他們透過引入具有 65,536 個詞彙（包括圖像標記）的 BPE 標記器，進一步完善了標記器，並使用句子庫進行了訓練。這種增強的分詞器提高了模型的性能和穩定性。

During training, they employed QK-Norm, dropout, and z-loss regularization techniques to address stability issues. Additionally, they leveraged Meta's RSC platform for efficient and scalable training. Inference in Chameleon is streamlined using PyTorch and xformers, supporting both streaming and non-streaming modes with token masking for conditional logic.

在訓練過程中，他們採用了 QK-Norm、dropout 和 z-loss 正則化技術來解決穩定性問題。此外，他們還利用 Meta 的 RSC 平台進行高效且可擴展的培訓。 Chameleon 中的推理使用 PyTorch 和 xformers 進行了簡化，支援流式和非流式模式，並具有條件邏輯的標記屏蔽。

To enhance model capabilities and safety, Chameleon underwent a rigorous alignment stage, undergoing fine-tuning on diverse datasets encompassing Text, Code, Visual Chat, and Safety. This comprehensive training regimen aimed to improve the model's generalizability and mitigate potential risks. They curated high-quality images for Image Generation using an aesthetic classifier, ensuring that the model generates visually appealing and relevant images.

為了增強模型功能和安全性，Chameleon 經歷了嚴格的調整階段，對包括文字、程式碼、視覺聊天和安全性在內的各種資料集進行了微調。這種綜合訓練方案旨在提高模型的通用性並降低潛在風險。他們使用美學分類器為圖像生成策劃了高品質的圖像，確保模型生成具有視覺吸引力且相關的圖像。

Supervised Fine-Tuning (SFT) played a crucial role in Chameleon's training process. By balancing data across modalities and employing a cosine learning rate schedule with a weight decay of 0.1, the researchers optimized the model's performance exclusively based on answer prompts. Dropout of 0.05 and z-loss regularization further enhanced the model's stability. Images in prompts were resized with border padding, while those in answers underwent center-cropping for optimal image generation quality.

有監督微調（SFT）在 Chameleon 的訓練過程中發揮了至關重要的作用。透過平衡各種模式的數據並採用權重衰減為 0.1 的餘弦學習率計劃，研究人員完全根據答案提示優化了模型的性能。 0.05 的 Dropout 和 z 損失正規化進一步增強了模型的穩定性。提示中的圖像透過邊框填充調整大小，而答案中的圖像則經過中心裁剪以獲得最佳圖像生成品質。

Chameleon's versatility extends beyond its core multimodal capabilities, as evidenced by its strong performance on various text-only tasks. The model achieved competitive results on commonsense reasoning and math tasks, surpassing LLaMa-2 on many benchmarks. This improvement can be attributed to Chameleon's robust pre-training and the inclusion of code data in its training regimen.

Chameleon 的多功能性超越了其核心多模式功能，其在各種純文字任務上的出色表現證明了這一點。該模型在常識推理和數學任務上取得了有競爭力的結果，在許多基準測試中超越了 LLaMa-2。這項改進可歸因於 Chameleon 強大的預訓練以及在其訓練方案中包含程式碼資料。

In image-to-text tasks, Chameleon excelled in image captioning, matching or surpassing larger models like Flamingo-80B and IDEFICS-80B with fewer shots. Its performance in visual question answering (VQA) approached that of top models, even though LLaMa-1.5 slightly outperformed VQA-v2.

在圖像到文字任務中，Chameleon 在圖像字幕方面表現出色，以更少的鏡頭匹配或超越了 Flamingo-80B 和 IDEFICS-80B 等較大模型。儘管 LLaMa-1.5 的表現略優於 VQA-v2，但其在視覺問答 (VQA) 方面的表現卻接近頂級模型。

Overall, Chameleon demonstrates remarkable versatility and efficiency across different tasks, requiring fewer training examples and smaller model sizes compared to its counterparts.

總體而言，Chameleon 在不同任務中表現出了卓越的多功能性和效率，與同類產品相比，它需要更少的訓練樣本和更小的模型大小。

In summary, Chameleon is a token-based model that seamlessly integrates image and text tokens, enabling superior performance in vision-language tasks. Its unified architecture facilitates joint reasoning over modalities, surpassing late-fusion models like Flamingo and IDEFICS in image captioning and visual question answering. Chameleon's early-fusion approach introduces novel techniques for stable training, addressing scalability challenges faced by previous multimodal models. It opens up new possibilities for multimodal interaction, as demonstrated by its strong performance on mixed-modal open-ended QA benchmarks.

總之，Chameleon 是一種基於 token 的模型，它無縫整合圖像和文字 token，從而在視覺語言任務中實現卓越的表現。其統一架構促進了模態聯合推理，在影像字幕和視覺問答方面超越了 Flamingo 和 IDEFICS 等後期融合模型。 Chameleon 的早期融合方法引入了穩定訓練的新技術，解決了先前多模態模型面臨的可擴展性挑戰。它為多模式互動開闢了新的可能性，正如其在混合模式開放式 QA 基準測試中的強勁表現所證明的那樣。

This groundbreaking research paves the way for the development of even more advanced multimodal models capable of handling complex tasks and producing highly coherent and informative content. Researchers, practitioners, and industry leaders alike eagerly anticipate the transformative potential of Chameleon and its potential impact on the future of AI-driven multimodal applications.

這項開創性的研究為開發更先進的多模態模型鋪平了道路，這些模型能夠處理複雜的任務並產生高度連貫和資訊豐富的內容。研究人員、從業者和行業領導者都熱切地期待 Chameleon 的變革潛力及其對人工智慧驅動的多模式應用的未來的潛在影響。