|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
預處理大語模型(LLM)的主要方法取決於下一步的預測,該預測已被證明有效地捕獲語言模式。但是,此方法具有明顯的局限性。

Large language models (LLMs) have become increasingly powerful thanks to pretraining on vast text corpora. But how can we improve their efficiency and interpretability? Meta AI introduces Continuous Concept Mixing (CoCoMix), a novel approach that combines token prediction with concept-based reasoning.
大型語言模型(LLMS)由於對龐大的文本語料庫的審核而變得越來越強大。但是,我們如何提高他們的效率和解釋性呢? Meta AI介紹了連續的概念混合(Cocomix),這是一種將令牌預測與基於概念的推理相結合的新方法。
Most LLMs are pretrained on a cloze-style next-token prediction task. While this method captures linguistic patterns, it also has limitations. Language tokens often convey only surface-level information, requiring models to process massive amounts of text to develop deeper reasoning capabilities. Furthermore, token-based learning has difficulty capturing long-term dependencies, making tasks that require planning and abstraction more challenging.
大多數LLM都按照緊縮風格的下一步預測任務進行了預定。儘管此方法捕獲了語言模式,但它也有局限性。語言令牌通常只傳達表面級信息,要求模型處理大量文本以發展更深的推理能力。此外,基於代幣的學習難以捕獲長期依賴性,使需要計劃和抽象更具挑戰性的任務。
To address these limitations, researchers have explored alternative strategies, such as knowledge distillation and structured input augmentation. However, these approaches do not fully resolve the limitations of token-based learning.
為了解決這些局限性,研究人員探索了替代策略,例如知識蒸餾和結構化輸入增強。但是,這些方法無法完全解決基於令牌學習的局限性。
This raises an important question: Can LLMs be trained in a way that combines token-level processing with conceptual understanding? Meta AI introduces Continuous Concept Mixing (CoCoMix) as a potential solution.
這提出了一個重要的問題:可以以將令牌級別的處理與概念理解相結合的方式進行培訓? Meta AI引入了連續的概念混合(Cocomix)作為潛在解決方案。
CoCoMix integrates token prediction with the modeling of continuous concepts derived from hidden states of a pretrained model. The method employs a Sparse Autoencoder (SAE) to extract high-level semantic representations, which are then incorporated into the training process by interleaving them with token embeddings.
Cocomix將令牌預測與源自預審計模型的隱藏狀態的連續概念的建模相結合。該方法採用稀疏的自動編碼器(SAE)提取高級語義表示,然後通過將它們與令牌嵌入方式交織在一起,然後將其納入訓練過程中。
This design allows the model to maintain the benefits of token-based learning while enhancing its ability to recognize and process broader conceptual structures. By enriching the token-based paradigm with concept-level information, CoCoMix aims to improve reasoning efficiency and model interpretability.
該設計使該模型能夠保持基於令牌學習的好處,同時增強其識別和處理更廣泛的概念結構的能力。通過通過概念級信息豐富基於令牌的範式,Cocomix旨在提高推理效率和模型的解釋性。
Meta AI introduces Continuous Concept Mixing (CoCoMix), a different approach to pretraining large language models (LLMs). This method combines token prediction with concept-based reasoning to enhance efficiency and interpretability.
Meta AI引入了連續的概念混合(Cocomix),這是一種預處理大語模型(LLM)的不同方法。該方法將令牌預測與基於概念的推理相結合,以提高效率和解釋性。
In a nutshell, CoCoMix operates through three main components:
簡而言之,Cocomix通過三個主要組成部分運行:
A token-based language model is trained on a cloze-style next-token prediction task. The model learns to predict the next token in a sequence given the preceding context. This component forms the basis of standard LLM pretraining.
基於令牌的語言模型接受了披風式的下一步預測任務的培訓。該模型學會在給定上下文的情況下以順序預測接下來的令牌。該組件構成了標準LLM預處理的基礎。
A Sparse Autoencoder (SAE) is trained on the hidden states of the language model. The SAE aims to extract high-level semantic representations from the language model's hidden states. This step is crucial for deriving continuous concepts.
在語言模型的隱藏狀態下對稀疏的自動編碼器(SAE)進行了訓練。 SAE旨在從語言模型的隱藏狀態中提取高級語義表示。此步驟對於得出連續概念至關重要。
The token embeddings from the language model are interleaved with the continuous concepts extracted by the SAE. This interleaving is performed at each training step. The resulting combined embeddings are then used to train the language model.
來自語言模型的令牌嵌入與SAE提取的連續概念交織在一起。這種交織在每個訓練步驟進行。然後,將結果組合的嵌入方式用於訓練語言模型。
This approach improves sample efficiency, enabling models to achieve comparable performance with fewer training tokens. Additionally, CoCoMix enhances interpretability by making it possible to inspect and adjust the extracted concepts, offering a clearer view of how the model processes information.
這種方法提高了樣本效率,從而使模型能夠通過更少的培訓令牌實現可比的性能。此外,Cocomix通過使您可以檢查和調整提取的概念,從而更清晰地了解模型如何處理信息,從而增強可解釋性。
Meta AI evaluated CoCoMix across multiple benchmarks, including OpenWebText, LAMBADA, WikiText-103, HellaSwag, PIQA, SIQA, Arc-Easy, and WinoGrande. The findings indicate:
Meta AI評估了跨多個基準測試的Cocomix,包括OpenWebText,Lambada,Wikitext-103,Hellaswag,Pics,Square,Arc-Easy和Winogrande。研究結果表明:
On OpenWebText, LAMBADA, and WikiText-103, models pretrained with CoCoMix achieve better performance in downstream tasks compared to models pretrained with standard token-based methods. This improvement is observed across a range of tasks, including masked language modeling, cloze-style next-token prediction, and question answering.
與用標準的基於標記的方法預測的模型相比,在OpenWebText,Lambada和Wikitext-103上,在下游任務中預讀的模型在下游任務中實現了更好的性能。在一系列任務中可以觀察到這種改進,包括蒙版語言建模,固定風格的下一步預測和問題回答。
Models pretrained with CoCoMix demonstrate improved sample efficiency. Specifically, they attain performance comparable to models pretrained with standard methods but using 4x fewer training tokens. This result highlights the efficiency gains achieved by incorporating continuous concepts into the pretraining process.
用Cocomix預測的模型表明樣品效率提高了。具體而言,它們的性能與用標準方法預處理但使用4倍訓練令牌的模型相當。該結果突出了通過將連續概念納入預訓練過程中所取得的效率提高。
CoCoMix enables the extraction of interpretable concepts that provide insights into the model's reasoning process. By adjusting the hyperparameters of the SAE, researchers can control the granularity and interpretability of the extracted concepts.
Cocomix可以提取可解釋的概念,這些概念為模型的推理過程提供了見解。通過調整SAE的超參數,研究人員可以控制提取概念的粒度和解釋性。
Overall, CoCoMix presents an alternative approach to LLM pretraining by combining token prediction with concept-based reasoning. This method enhances efficiency and interpretability without disrupting the underlying next-token prediction framework. Experimental results suggest that this technique provides a balanced way to improve language model training, particularly in areas requiring structured reasoning and transparent decision-making.
總體而言,Cocomix通過將令牌預測與基於概念的推理相結合,為LLM預處理提供了另一種方法。此方法可以提高效率和解釋性,而不會破壞基本的下一步預測框架。實驗結果表明,該技術提供了一種平衡的方法來改善語言模型培訓,尤其是在需要結構化推理和透明決策的領域。
Future research may focus on refining concept extraction methods and further integrating continuous representations into pretraining workflows.
未來的研究可能著重於精煉概念提取方法,並進一步將連續的表示形式納入訓練預處理。
免責聲明:info@kdj.com
所提供的資訊並非交易建議。 kDJ.com對任何基於本文提供的資訊進行的投資不承擔任何責任。加密貨幣波動性較大,建議您充分研究後謹慎投資!
如果您認為本網站使用的內容侵犯了您的版權,請立即聯絡我們(info@kdj.com),我們將及時刪除。
-
-
-
- Trust 錢包位址標籤增強加密安全:發送變得更簡單、更安全
- 2026-11-17 20:05:01
- Trust Wallet 的新地址標籤功能簡化了加密交易,減少了錯誤並增強了用戶安全地發送數位資產的信心。
-
-
-
-
- 由於現貨 ETF 流出和市場範圍內的拋售壓力,XRP 面臨價格走勢困境
- 2026-11-17 07:55:01
- XRP 價格大幅下跌,反映了 ETF 資金外流和宏觀經濟擔憂驅動的更廣泛的市場趨勢。
-
- 001k.bot 透過增強的數位資產管理徹底改變加密貨幣收益
- 2026-11-17 00:05:01
- 探索 001k.bot 的新加密貨幣賺取功能,加上強大的數位資產管理,如何改變加密貨幣持有者的遊戲規則。
-

































