Chameleon：无缝多模式文档创建的革命性模型

Meta 研究人员开发了 Chameleon，这是一种多模态基础模型，可以无缝集成图像和文本序列。与分离模态的传统模型不同，Chameleon 的统一架构将图像（如文本）标记化，从而实现全面的文档建模和跨模态的交错推理。为了解决这种早期融合方法中的优化挑战，研究人员引入了架构增强和训练技术，从而在 Meta 的大规模数据集上成功进行了训练。 Chameleon 的评估展示了在纯文本任务中的竞争性表现，在常识推理和数学方面优于 LLaMa-2。在图像相关任务中，它在图像字幕和视觉问答方面表现出色，以更少的镜头超越或匹配更大的模型。

Chameleon: A Unified Foundation Model for Seamless Multimodal Document Modeling

Chameleon：无缝多模式文档建模的统一基础模型

Recent multimodal foundation models have revolutionized the field of artificial intelligence, enabling impressive advancements in tasks such as natural language processing, image recognition, and question answering. However, these models often treat different modalities separately, employing specific encoders or decoders for each. This approach inherently limits their ability to effectively fuse information across modalities and produce comprehensive multimodal documents that seamlessly integrate diverse sequences of images and text.

最近的多模态基础模型彻底改变了人工智能领域，在自然语言处理、图像识别和问答等任务方面取得了令人瞩目的进步。然而，这些模型通常单独处理不同的模态，为每种模态采用特定的编码器或解码器。这种方法本质上限制了他们有效融合跨模态信息并生成无缝集成不同图像和文本序列的全面多模态文档的能力。

Meta researchers have addressed this challenge by introducing Chameleon, a groundbreaking mixed-modal foundation model designed to facilitate seamless reasoning and generation with interleaved textual and image sequences. Unlike traditional models, Chameleon employs a unified architecture that treats both modalities equally by tokenizing images akin to text. This approach, termed early fusion, allows for seamless flow of information across modalities but also presents unique optimization challenges.

Meta 研究人员通过引入 Chameleon 解决了这一挑战，Chameleon 是一种突破性的混合模式基础模型，旨在促进交错文本和图像序列的无缝推理和生成。与传统模型不同，Chameleon 采用统一的架构，通过将图像标记为类似于文本的方式，平等地对待这两种模式。这种方法被称为早期融合，允许跨模式的信息无缝流动，但也带来了独特的优化挑战。

To overcome these challenges, the researchers have incorporated architectural enhancements and innovative training techniques into Chameleon's design. By adapting transformer architecture and fine-tuning strategies, they have laid the foundation for a model capable of addressing the complexities of multimodal document modeling.

为了克服这些挑战，研究人员将架构增强和创新训练技术融入 Chameleon 的设计中。通过调整变压器架构和微调策略，他们为能够解决多模式文档建模复杂性的模型奠定了基础。

To capture the visual information effectively, they developed a novel image tokenizer that encodes high-resolution 512 × 512 images into 1024 tokens from an 8192-codebook. This tokenization process ensures that images are represented in a manner compatible with the text tokens, facilitating seamless reasoning across modalities. They further refined their tokenizer by introducing a BPE tokenizer with a 65,536-vocabulary, including image tokens, trained using the sentencepiece library. This enhanced tokenizer improved the model's performance and stability.

为了有效地捕获视觉信息，他们开发了一种新颖的图像标记器，将高分辨率 512 × 512 图像编码为 8192 码本中的 1024 个标记。此标记化过程确保图像以与文本标记兼容的方式表示，从而促进跨模式的无缝推理。他们通过引入具有 65,536 个词汇（包括图像标记）的 BPE 标记器，进一步完善了标记器，并使用句子库进行了训练。这种增强的分词器提高了模型的性能和稳定性。

During training, they employed QK-Norm, dropout, and z-loss regularization techniques to address stability issues. Additionally, they leveraged Meta's RSC platform for efficient and scalable training. Inference in Chameleon is streamlined using PyTorch and xformers, supporting both streaming and non-streaming modes with token masking for conditional logic.

在训练过程中，他们采用了 QK-Norm、dropout 和 z-loss 正则化技术来解决稳定性问题。此外，他们还利用 Meta 的 RSC 平台进行高效且可扩展的培训。 Chameleon 中的推理使用 PyTorch 和 xformers 进行了简化，支持流式和非流式模式，并具有条件逻辑的标记屏蔽。

To enhance model capabilities and safety, Chameleon underwent a rigorous alignment stage, undergoing fine-tuning on diverse datasets encompassing Text, Code, Visual Chat, and Safety. This comprehensive training regimen aimed to improve the model's generalizability and mitigate potential risks. They curated high-quality images for Image Generation using an aesthetic classifier, ensuring that the model generates visually appealing and relevant images.

为了增强模型功能和安全性，Chameleon 经历了严格的调整阶段，对包括文本、代码、视觉聊天和安全性在内的各种数据集进行了微调。这种综合训练方案旨在提高模型的通用性并降低潜在风险。他们使用美学分类器为图像生成策划了高质量的图像，确保模型生成具有视觉吸引力且相关的图像。

Supervised Fine-Tuning (SFT) played a crucial role in Chameleon's training process. By balancing data across modalities and employing a cosine learning rate schedule with a weight decay of 0.1, the researchers optimized the model's performance exclusively based on answer prompts. Dropout of 0.05 and z-loss regularization further enhanced the model's stability. Images in prompts were resized with border padding, while those in answers underwent center-cropping for optimal image generation quality.

有监督微调（SFT）在 Chameleon 的训练过程中发挥了至关重要的作用。通过平衡各种模式的数据并采用权重衰减为 0.1 的余弦学习率计划，研究人员完全根据答案提示优化了模型的性能。 0.05 的 Dropout 和 z 损失正则化进一步增强了模型的稳定性。提示中的图像通过边框填充调整大小，而答案中的图像则经过中心裁剪以获得最佳图像生成质量。

Chameleon's versatility extends beyond its core multimodal capabilities, as evidenced by its strong performance on various text-only tasks. The model achieved competitive results on commonsense reasoning and math tasks, surpassing LLaMa-2 on many benchmarks. This improvement can be attributed to Chameleon's robust pre-training and the inclusion of code data in its training regimen.

Chameleon 的多功能性超出了其核心多模式功能，其在各种纯文本任务上的出色表现就证明了这一点。该模型在常识推理和数学任务上取得了有竞争力的结果，在许多基准测试中超越了 LLaMa-2。这一改进可归因于 Chameleon 强大的预训练以及在其训练方案中包含代码数据。

In image-to-text tasks, Chameleon excelled in image captioning, matching or surpassing larger models like Flamingo-80B and IDEFICS-80B with fewer shots. Its performance in visual question answering (VQA) approached that of top models, even though LLaMa-1.5 slightly outperformed VQA-v2.

在图像到文本任务中，Chameleon 在图像字幕方面表现出色，以更少的镜头匹配或超越了 Flamingo-80B 和 IDEFICS-80B 等较大模型。尽管 LLaMa-1.5 的性能略优于 VQA-v2，但其在视觉问答 (VQA) 方面的性能接近顶级模型。

Overall, Chameleon demonstrates remarkable versatility and efficiency across different tasks, requiring fewer training examples and smaller model sizes compared to its counterparts.

总体而言，Chameleon 在不同任务中表现出了卓越的多功能性和效率，与同类产品相比，它需要更少的训练样本和更小的模型大小。

In summary, Chameleon is a token-based model that seamlessly integrates image and text tokens, enabling superior performance in vision-language tasks. Its unified architecture facilitates joint reasoning over modalities, surpassing late-fusion models like Flamingo and IDEFICS in image captioning and visual question answering. Chameleon's early-fusion approach introduces novel techniques for stable training, addressing scalability challenges faced by previous multimodal models. It opens up new possibilities for multimodal interaction, as demonstrated by its strong performance on mixed-modal open-ended QA benchmarks.

总之，Chameleon 是一种基于 token 的模型，它无缝集成图像和文本 token，从而在视觉语言任务中实现卓越的性能。其统一架构促进了模态联合推理，在图像字幕和视觉问答方面超越了 Flamingo 和 IDEFICS 等后期融合模型。 Chameleon 的早期融合方法引入了稳定训练的新技术，解决了以前多模态模型面临的可扩展性挑战。它为多模式交互开辟了新的可能性，正如其在混合模式开放式 QA 基准测试中的强劲表现所证明的那样。

This groundbreaking research paves the way for the development of even more advanced multimodal models capable of handling complex tasks and producing highly coherent and informative content. Researchers, practitioners, and industry leaders alike eagerly anticipate the transformative potential of Chameleon and its potential impact on the future of AI-driven multimodal applications.

这项开创性的研究为开发更先进的多模态模型铺平了道路，这些模型能够处理复杂的任务并生成高度连贯和信息丰富的内容。研究人员、从业者和行业领导者都热切地期待 Chameleon 的变革潜力及其对人工智能驱动的多模式应用的未来的潜在影响。