Tokenformer：透​​過將參數視為令牌來重新思考 Transformer

Transformers 改變了人工智慧，在 NLP、電腦視覺和多模式資料整合方面提供了無與倫比的性能。這些模型擅長透過注意力機制識別資料中的模式，使其成為複雜任務的理想選擇。然而，由於傳統結構的計算成本較高，變壓器模型的快速擴展需要改進。

Transformers have revolutionized artificial intelligence, offering unparalleled performance in natural language processing (NLP), computer vision, and multi-modal data integration. These models excel at identifying patterns within data through their attention mechanisms, making them ideal for complex tasks. However, the rapid scaling of transformer models needs to be improved because of the high computational cost associated with their traditional structure. As these models grow, they demand significant hardware resources and training time, which increases exponentially with the model size.

Transformers 徹底改變了人工智慧，在自然語言處理 (NLP)、電腦視覺和多模式資料整合方面提供了無與倫比的效能。這些模型擅長透過注意力機制識別資料中的模式，使其成為複雜任務的理想選擇。然而，由於傳統結構的計算成本較高，變壓器模型的快速擴展需要改進。隨著這些模型的增長，它們需要大量的硬體資源和訓練時間，並且隨著模型大小的增加而呈指數級增長。

The primary obstacle in scaling transformers lies in the fixed parameters within their linear projection layers. This static structure limits the model’s ability to expand without being entirely retrained, which becomes exponentially more expensive as model sizes increase. These traditional models typically demand comprehensive retraining when architectural modifications occur, such as increasing channel dimensions.

縮放變壓器的主要障礙在於其線性投影層內的固定參數。這種靜態結構限制了模型在不完全重新訓練的情況下擴展的能力，隨著模型大小的增加，這種能力會呈指數級增長。當發生架構修改（例如增加通道尺寸）時，這些傳統模型通常需要全面的重新訓練。

Consequently, the computational cost for these expansions grows impractically high, and the approach lacks flexibility. The inability to add new parameters dynamically stifles growth, rendering these models less adaptable to evolving AI applications and more costly in terms of time and resources.

因此，這些擴展的計算成本變得非常高，而且該方法缺乏靈活性。無法動態添加新參數會抑製成長，使這些模型不太適應不斷發展的人工智慧應用程序，並且在時間和資源方面成本更高。

Historically, approaches to managing model scalability included duplicating weights or restructuring models using methods like Net2Net, where duplicating neurons expand layers. However, these approaches often disrupt the balance of pre-trained models, resulting in slower convergence rates and additional training complexities.

從歷史上看，管理模型可擴展性的方法包括使用 Net2Net 等方法複製權重或重構模型，其中複製神經元擴展層。然而，這些方法通常會破壞預訓練模型的平衡，導致收斂速度變慢並增加訓練複雜性。

While these methods have made incremental progress, they still face limitations in preserving model integrity during scaling. Transformers rely heavily on static linear projections, making parameter expansion expensive and inflexible. Traditional models like GPT and other large transformers often retrain from scratch, incurring high computational costs with each new scaling stage.

儘管這些方法取得了漸進的進展，但它們在擴展過程中保持模型完整性方面仍然面臨限制。 Transformer 嚴重依賴靜態線性投影，導致參數擴展成本高且不靈活。 GPT 和其他大型 Transformer 等傳統模型通常從頭開始重新訓練，每個新的擴展階段都會產生高昂的計算成本。

Now, researchers at the Max Planck Institute, Google, and Peking University have developed a new architecture called Tokenformer that fundamentally reimagines transformers by treating model parameters as tokens, allowing for dynamic interactions between tokens and parameters.

現在，馬克斯普朗克研究所、谷歌和北京大學的研究人員開發了一種名為Tokenformer 的新架構，該架構透過將模型參數視為令牌來從根本上重新構想Transformer，從而允許令牌和參數之間進行動態交互作用。

In this framework, Tokenformer introduces a novel component called the token-parameter attention (Pattention) layer, which facilitates incremental scaling. The model can add new parameter tokens without retraining, drastically reducing training costs.

在此框架中，Tokenformer 引入了一個稱為令牌參數注意（Pattention）層的新穎元件，該元件有助於增量擴展。該模型可以添加新的參數標記而無需重新訓練，從而大大降低了訓練成本。

By representing input tokens and parameters within the same framework, Tokenformer allows for flexible scaling, providing researchers with a more efficient, resource-conscious model architecture that retains scalability and high performance.

透過在同一框架內表示輸入令牌和參數，Tokenformer 可以靈活擴展，為研究人員提供更有效率、資源意識更強的模型架構，同時保留可擴展性和高效能。

Tokenformer’s Pattention layer uses input tokens as queries, while model parameters serve as keys and values, which differs from the standard transformer approach, relying solely on linear projections.

Tokenformer 的 Pattention 層使用輸入標記作為查詢，而模型參數作為鍵和值，這與標準 Transformer 方法不同，僅依賴線性投影。

The model’s scaling is achieved by adding new key-value parameter pairs, keeping input and output dimensions constant, and avoiding full retraining. Tokenformer’s architecture is designed to be modular, enabling researchers to expand the model seamlessly by incorporating additional tokens.

此模型的擴展是透過添加新的鍵值參數對、保持輸入和輸出維度恆定並避免完全重新訓練來實現的。 Tokenformer 的架構設計為模組化，使研究人員能夠透過合併額外的代幣來無縫擴展模型。

This incremental scaling capability supports the efficient reuse of pre-trained weights while enabling rapid adaptation for new datasets or larger model sizes without disrupting learned information.

這種增量擴展功能支援預訓練權重的有效重複使用，同時能夠快速適應新資料集或更大的模型大小，而不會破壞學習到的資訊。

The performance benefits of Tokenformer are notable, as the model significantly reduces computational costs while maintaining accuracy. For instance, Tokenformer scaled from 124 million to 1.4 billion parameters with only half the typical training costs traditional transformers require.

Tokenformer 的效能優勢非常顯著，因為模型在保持準確性的同時顯著降低了計算成本。例如，Tokenformer 將參數從 1.24 億擴展到 14 億，而訓練成本僅為傳統 Transformer 所需的一半。

In one experiment, the model achieved a test perplexity of 11.77 for a 1.4 billion parameter configuration, nearly matching the 11.63 perplexity of a similarly sized transformer trained from scratch.

在一項實驗中，該模型針對 14 億個參數配置實現了 11.77 的測試困惑度，幾乎與從頭開始訓練的類似大小的 Transformer 的 11.63 困惑度相符。

This efficiency means Tokenformer can achieve high performance across multiple domains, including language and visual modeling tasks, at a fraction of the resource expenditure of traditional models.

這種效率意味著 Tokenformer 可以在多個領域實現高效能，包括語言和視覺建模任務，而資源支出只是傳統模型的一小部分。

Tokenformer presents numerous key takeaways for advancing AI research and improving transformer-based models. These include:

Tokenformer 提出了許多關於推進人工智慧研究和改進基於 Transformer 的模型的關鍵要點。這些包括：

Treating parameters as tokens enables incremental model scaling without retraining.

將參數視為標記可以實現增量模型擴展而無需重新訓練。

The token-parameter attention layer facilitates efficient parameter expansion.

令牌參數注意層有助於有效的參數擴展。

Modular architecture supports seamless model growth by incorporating additional tokens.

模組化架構透過合併額外的代幣來支援無縫模型成長。

The model achieves high performance across diverse domains with minimal resource expenditure.

該模型以最少的資源支出實現了跨不同領域的高效能。

In conclusion, Tokenformer offers a transformative approach to scaling transformer-based models. This model architecture achieves scalability and resource efficiency by treating parameters as tokens, reducing costs, and preserving model performance across tasks.

總之，Tokenformer 提供了一種變革性方法來擴展基於 Transformer 的模型。此模型架構透過將參數視為令牌、降低成本並保持跨任務的模型效能來實現可擴展性和資源效率。

This flexibility represents a breakthrough in transformer design, providing a model that can adapt to the demands of advancing AI applications without retraining. Tokenformer’s architecture holds promise for future AI research, offering a pathway to develop large-scale models sustainably and efficiently.

這種靈活性代表了變壓器設計的突破，提供了一種無需重新訓練即可適應先進人工智慧應用需求的模型。 Tokenformer 的架構為未來的人工智慧研究帶來了希望，提供了可持續、高效地開發大規模模型的途徑。

Check out the Paper, GitHub Page, and Models on HuggingFace.

查看論文、GitHub 頁面和 HuggingFace 上的模型。

All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. If you like our work, you will love our newsletter. Don’t Forget to join our 55k+ ML SubReddit.

這項研究的所有功勞都歸功於該計畫的研究人員。另外，不要忘記在 Twitter 上關注我們並加入我們的 Telegram 頻道和 LinkedIn 群組。如果您喜歡我們的工作，您一定會喜歡我們的時事通訊。不要忘記加入我們超過 55k 的 ML SubReddit。

[Sponsorship Opportunity with us] Promote Your Research/Product/Webinar with 1Million+ Monthly Readers and 500k+ Community Members

[我們的贊助機會] 向 100 萬以上的每月讀者和 50 萬以上的社區成員推廣您的研究/產品/網絡研討會