Tokenformer：通过将参数视为令牌来重新思考 Transformer

Transformers 改变了人工智能，在 NLP、计算机视觉和多模式数据集成方面提供了无与伦比的性能。这些模型擅长通过注意力机制识别数据中的模式，使其成为复杂任务的理想选择。然而，由于传统结构的计算成本较高，变压器模型的快速扩展需要改进。

Transformers have revolutionized artificial intelligence, offering unparalleled performance in natural language processing (NLP), computer vision, and multi-modal data integration. These models excel at identifying patterns within data through their attention mechanisms, making them ideal for complex tasks. However, the rapid scaling of transformer models needs to be improved because of the high computational cost associated with their traditional structure. As these models grow, they demand significant hardware resources and training time, which increases exponentially with the model size.

Transformers 彻底改变了人工智能，在自然语言处理 (NLP)、计算机视觉和多模式数据集成方面提供了无与伦比的性能。这些模型擅长通过注意力机制识别数据中的模式，使其成为复杂任务的理想选择。然而，由于传统结构的计算成本较高，变压器模型的快速扩展需要改进。随着这些模型的增长，它们需要大量的硬件资源和训练时间，并且随着模型大小的增加呈指数级增长。

The primary obstacle in scaling transformers lies in the fixed parameters within their linear projection layers. This static structure limits the model’s ability to expand without being entirely retrained, which becomes exponentially more expensive as model sizes increase. These traditional models typically demand comprehensive retraining when architectural modifications occur, such as increasing channel dimensions.

缩放变压器的主要障碍在于其线性投影层内的固定参数。这种静态结构限制了模型在不完全重新训练的情况下扩展的能力，随着模型大小的增加，这种能力会呈指数级增长。当发生架构修改（例如增加通道尺寸）时，这些传统模型通常需要全面的重新训练。

Consequently, the computational cost for these expansions grows impractically high, and the approach lacks flexibility. The inability to add new parameters dynamically stifles growth, rendering these models less adaptable to evolving AI applications and more costly in terms of time and resources.

因此，这些扩展的计算成本变得非常高，而且该方法缺乏灵活性。无法动态添加新参数会抑制增长，使这些模型不太适应不断发展的人工智能应用程序，并且在时间和资源方面成本更高。

Historically, approaches to managing model scalability included duplicating weights or restructuring models using methods like Net2Net, where duplicating neurons expand layers. However, these approaches often disrupt the balance of pre-trained models, resulting in slower convergence rates and additional training complexities.

从历史上看，管理模型可扩展性的方法包括使用 Net2Net 等方法复制权重或重构模型，其中复制神经元扩展层。然而，这些方法通常会破坏预训练模型的平衡，导致收敛速度变慢并增加训练复杂性。

While these methods have made incremental progress, they still face limitations in preserving model integrity during scaling. Transformers rely heavily on static linear projections, making parameter expansion expensive and inflexible. Traditional models like GPT and other large transformers often retrain from scratch, incurring high computational costs with each new scaling stage.

尽管这些方法取得了渐进的进展，但它们在扩展过程中保持模型完整性方面仍然面临限制。 Transformer 严重依赖静态线性投影，导致参数扩展成本高昂且不灵活。 GPT 和其他大型 Transformer 等传统模型通常从头开始重新训练，每个新的扩展阶段都会产生高昂的计算成本。

Now, researchers at the Max Planck Institute, Google, and Peking University have developed a new architecture called Tokenformer that fundamentally reimagines transformers by treating model parameters as tokens, allowing for dynamic interactions between tokens and parameters.

现在，马克斯·普朗克研究所、谷歌和北京大学的研究人员开发了一种名为 Tokenformer 的新架构，该架构通过将模型参数视为令牌来从根本上重新构想 Transformer，从而允许令牌和参数之间进行动态交互。

In this framework, Tokenformer introduces a novel component called the token-parameter attention (Pattention) layer, which facilitates incremental scaling. The model can add new parameter tokens without retraining, drastically reducing training costs.

在此框架中，Tokenformer 引入了一个称为令牌参数注意（Pattention）层的新颖组件，该组件有助于增量扩展。该模型可以添加新的参数标记而无需重新训练，从而大大降低了训练成本。

By representing input tokens and parameters within the same framework, Tokenformer allows for flexible scaling, providing researchers with a more efficient, resource-conscious model architecture that retains scalability and high performance.

通过在同一框架内表示输入令牌和参数，Tokenformer 可以灵活扩展，为研究人员提供更高效、资源意识更强的模型架构，同时保留可扩展性和高性能。

Tokenformer’s Pattention layer uses input tokens as queries, while model parameters serve as keys and values, which differs from the standard transformer approach, relying solely on linear projections.

Tokenformer 的 Pattention 层使用输入标记作为查询，而模型参数作为键和值，这与标准 Transformer 方法不同，仅依赖于线性投影。

The model’s scaling is achieved by adding new key-value parameter pairs, keeping input and output dimensions constant, and avoiding full retraining. Tokenformer’s architecture is designed to be modular, enabling researchers to expand the model seamlessly by incorporating additional tokens.

该模型的扩展是通过添加新的键值参数对、保持输入和输出维度恒定并避免完全重新训练来实现的。 Tokenformer 的架构设计为模块化，使研究人员能够通过合并额外的代币来无缝扩展模型。

This incremental scaling capability supports the efficient reuse of pre-trained weights while enabling rapid adaptation for new datasets or larger model sizes without disrupting learned information.

这种增量扩展功能支持预训练权重的有效重用，同时能够快速适应新数据集或更大的模型大小，而不会破坏学习到的信息。

The performance benefits of Tokenformer are notable, as the model significantly reduces computational costs while maintaining accuracy. For instance, Tokenformer scaled from 124 million to 1.4 billion parameters with only half the typical training costs traditional transformers require.

Tokenformer 的性能优势非常显着，因为该模型在保持准确性的同时显着降低了计算成本。例如，Tokenformer 将参数从 1.24 亿扩展到 14 亿，而训练成本仅为传统 Transformer 所需的一半。

In one experiment, the model achieved a test perplexity of 11.77 for a 1.4 billion parameter configuration, nearly matching the 11.63 perplexity of a similarly sized transformer trained from scratch.

在一项实验中，该模型针对 14 亿个参数配置实现了 11.77 的测试困惑度，几乎与从头开始训练的类似大小的 Transformer 的 11.63 困惑度相匹配。

This efficiency means Tokenformer can achieve high performance across multiple domains, including language and visual modeling tasks, at a fraction of the resource expenditure of traditional models.

这种效率意味着 Tokenformer 可以在多个领域实现高性能，包括语言和视觉建模任务，而资源支出只是传统模型的一小部分。

Tokenformer presents numerous key takeaways for advancing AI research and improving transformer-based models. These include:

Tokenformer 提出了许多关于推进人工智能研究和改进基于 Transformer 的模型的关键要点。这些包括：

Treating parameters as tokens enables incremental model scaling without retraining.

将参数视为标记可以实现增量模型扩展而无需重新训练。

The token-parameter attention layer facilitates efficient parameter expansion.

令牌参数注意层有助于有效的参数扩展。

Modular architecture supports seamless model growth by incorporating additional tokens.

模块化架构通过合并额外的代币来支持无缝模型增长。

The model achieves high performance across diverse domains with minimal resource expenditure.

该模型以最少的资源支出实现了跨不同领域的高性能。

In conclusion, Tokenformer offers a transformative approach to scaling transformer-based models. This model architecture achieves scalability and resource efficiency by treating parameters as tokens, reducing costs, and preserving model performance across tasks.

总之，Tokenformer 提供了一种变革性方法来扩展基于 Transformer 的模型。该模型架构通过将参数视为令牌、降低成本并保持跨任务的模型性能来实现可扩展性和资源效率。

This flexibility represents a breakthrough in transformer design, providing a model that can adapt to the demands of advancing AI applications without retraining. Tokenformer’s architecture holds promise for future AI research, offering a pathway to develop large-scale models sustainably and efficiently.

这种灵活性代表了变压器设计的突破，提供了一种无需重新训练即可适应先进人工智能应用需求的模型。 Tokenformer 的架构为未来的人工智能研究带来了希望，提供了可持续、高效地开发大规模模型的途径。

Check out the Paper, GitHub Page, and Models on HuggingFace.

查看论文、GitHub 页面和 HuggingFace 上的模型。

All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. If you like our work, you will love our newsletter. Don’t Forget to join our 55k+ ML SubReddit.

这项研究的所有功劳都归功于该项目的研究人员。另外，不要忘记在 Twitter 上关注我们并加入我们的 Telegram 频道和 LinkedIn 群组。如果您喜欢我们的工作，您一定会喜欢我们的时事通讯。不要忘记加入我们超过 55k 的 ML SubReddit。

[Sponsorship Opportunity with us] Promote Your Research/Product/Webinar with 1Million+ Monthly Readers and 500k+ Community Members

[我们的赞助机会] 向 100 万以上的每月读者和 50 万以上的社区成员推广您的研究/产品/网络研讨会