|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
预处理大语模型(LLM)的主要方法取决于下一步的预测,该预测已被证明有效地捕获语言模式。但是,此方法具有明显的局限性。

Large language models (LLMs) have become increasingly powerful thanks to pretraining on vast text corpora. But how can we improve their efficiency and interpretability? Meta AI introduces Continuous Concept Mixing (CoCoMix), a novel approach that combines token prediction with concept-based reasoning.
大型语言模型(LLMS)由于对庞大的文本语料库的审核而变得越来越强大。但是,我们如何提高他们的效率和解释性呢? Meta AI介绍了连续的概念混合(Cocomix),这是一种将令牌预测与基于概念的推理相结合的新方法。
Most LLMs are pretrained on a cloze-style next-token prediction task. While this method captures linguistic patterns, it also has limitations. Language tokens often convey only surface-level information, requiring models to process massive amounts of text to develop deeper reasoning capabilities. Furthermore, token-based learning has difficulty capturing long-term dependencies, making tasks that require planning and abstraction more challenging.
大多数LLM都按照紧缩风格的下一步预测任务进行了预定。尽管此方法捕获了语言模式,但它也有局限性。语言令牌通常只传达表面级信息,要求模型处理大量文本以发展更深的推理能力。此外,基于代币的学习难以捕获长期依赖性,使需要计划和抽象更具挑战性的任务。
To address these limitations, researchers have explored alternative strategies, such as knowledge distillation and structured input augmentation. However, these approaches do not fully resolve the limitations of token-based learning.
为了解决这些局限性,研究人员探索了替代策略,例如知识蒸馏和结构化输入增强。但是,这些方法无法完全解决基于令牌学习的局限性。
This raises an important question: Can LLMs be trained in a way that combines token-level processing with conceptual understanding? Meta AI introduces Continuous Concept Mixing (CoCoMix) as a potential solution.
这提出了一个重要的问题:可以以将令牌级别的处理与概念理解相结合的方式进行培训? Meta AI引入了连续的概念混合(Cocomix)作为潜在解决方案。
CoCoMix integrates token prediction with the modeling of continuous concepts derived from hidden states of a pretrained model. The method employs a Sparse Autoencoder (SAE) to extract high-level semantic representations, which are then incorporated into the training process by interleaving them with token embeddings.
Cocomix将令牌预测与源自预审计模型的隐藏状态的连续概念的建模相结合。该方法采用稀疏的自动编码器(SAE)提取高级语义表示,然后通过将它们与令牌嵌入方式交织在一起,然后将其纳入训练过程中。
This design allows the model to maintain the benefits of token-based learning while enhancing its ability to recognize and process broader conceptual structures. By enriching the token-based paradigm with concept-level information, CoCoMix aims to improve reasoning efficiency and model interpretability.
该设计使该模型能够保持基于令牌学习的好处,同时增强其识别和处理更广泛的概念结构的能力。通过通过概念级信息丰富基于令牌的范式,Cocomix旨在提高推理效率和模型的解释性。
Meta AI introduces Continuous Concept Mixing (CoCoMix), a different approach to pretraining large language models (LLMs). This method combines token prediction with concept-based reasoning to enhance efficiency and interpretability.
Meta AI引入了连续的概念混合(Cocomix),这是一种预处理大语模型(LLM)的不同方法。该方法将令牌预测与基于概念的推理相结合,以提高效率和解释性。
In a nutshell, CoCoMix operates through three main components:
简而言之,Cocomix通过三个主要组成部分运行:
A token-based language model is trained on a cloze-style next-token prediction task. The model learns to predict the next token in a sequence given the preceding context. This component forms the basis of standard LLM pretraining.
基于令牌的语言模型接受了披风式的下一步预测任务的培训。该模型学会在给定上下文的情况下以顺序预测接下来的令牌。该组件构成了标准LLM预处理的基础。
A Sparse Autoencoder (SAE) is trained on the hidden states of the language model. The SAE aims to extract high-level semantic representations from the language model's hidden states. This step is crucial for deriving continuous concepts.
在语言模型的隐藏状态下对稀疏的自动编码器(SAE)进行了训练。 SAE旨在从语言模型的隐藏状态中提取高级语义表示。此步骤对于得出连续概念至关重要。
The token embeddings from the language model are interleaved with the continuous concepts extracted by the SAE. This interleaving is performed at each training step. The resulting combined embeddings are then used to train the language model.
来自语言模型的令牌嵌入与SAE提取的连续概念交织在一起。这种交织在每个训练步骤进行。然后,将结果组合的嵌入方式用于训练语言模型。
This approach improves sample efficiency, enabling models to achieve comparable performance with fewer training tokens. Additionally, CoCoMix enhances interpretability by making it possible to inspect and adjust the extracted concepts, offering a clearer view of how the model processes information.
这种方法提高了样本效率,从而使模型能够通过更少的培训令牌实现可比的性能。此外,Cocomix通过使您可以检查和调整提取的概念,从而更清晰地了解模型如何处理信息,从而增强可解释性。
Meta AI evaluated CoCoMix across multiple benchmarks, including OpenWebText, LAMBADA, WikiText-103, HellaSwag, PIQA, SIQA, Arc-Easy, and WinoGrande. The findings indicate:
Meta AI评估了跨多个基准测试的Cocomix,包括OpenWebText,Lambada,Wikitext-103,Hellaswag,Pics,Square,Arc-Easy和Winogrande。研究结果表明:
On OpenWebText, LAMBADA, and WikiText-103, models pretrained with CoCoMix achieve better performance in downstream tasks compared to models pretrained with standard token-based methods. This improvement is observed across a range of tasks, including masked language modeling, cloze-style next-token prediction, and question answering.
与用标准的基于标记的方法预测的模型相比,在OpenWebText,Lambada和Wikitext-103上,在下游任务中预读的模型在下游任务中实现了更好的性能。在一系列任务中可以观察到这种改进,包括蒙版语言建模,固定风格的下一步预测和问题回答。
Models pretrained with CoCoMix demonstrate improved sample efficiency. Specifically, they attain performance comparable to models pretrained with standard methods but using 4x fewer training tokens. This result highlights the efficiency gains achieved by incorporating continuous concepts into the pretraining process.
用Cocomix预测的模型表明样品效率提高了。具体而言,它们的性能与用标准方法预处理但使用4倍训练令牌的模型相当。该结果突出了通过将连续概念纳入预训练过程中所取得的效率提高。
CoCoMix enables the extraction of interpretable concepts that provide insights into the model's reasoning process. By adjusting the hyperparameters of the SAE, researchers can control the granularity and interpretability of the extracted concepts.
Cocomix可以提取可解释的概念,这些概念为模型的推理过程提供了见解。通过调整SAE的超参数,研究人员可以控制提取概念的粒度和解释性。
Overall, CoCoMix presents an alternative approach to LLM pretraining by combining token prediction with concept-based reasoning. This method enhances efficiency and interpretability without disrupting the underlying next-token prediction framework. Experimental results suggest that this technique provides a balanced way to improve language model training, particularly in areas requiring structured reasoning and transparent decision-making.
总体而言,Cocomix通过将令牌预测与基于概念的推理相结合,为LLM预处理提供了另一种方法。此方法可以提高效率和解释性,而不会破坏基本的下一步预测框架。实验结果表明,该技术提供了一种平衡的方法来改善语言模型培训,尤其是在需要结构化推理和透明决策的领域。
Future research may focus on refining concept extraction methods and further integrating continuous representations into pretraining workflows.
未来的研究可能着重于精炼概念提取方法,并进一步将连续的表示形式纳入训练预处理。
免责声明:info@kdj.com
所提供的信息并非交易建议。根据本文提供的信息进行的任何投资,kdj.com不承担任何责任。加密货币具有高波动性,强烈建议您深入研究后,谨慎投资!
如您认为本网站上使用的内容侵犯了您的版权,请立即联系我们(info@kdj.com),我们将及时删除。
-
-
-
- Trust 钱包地址标签增强加密安全:发送变得更简单、更安全
- 2026-11-17 20:05:01
- Trust Wallet 的新地址标签功能简化了加密交易,减少了错误并增强了用户安全发送数字资产的信心。
-
-
-
-
- 由于现货 ETF 流出和市场范围内的抛售压力,XRP 面临价格走势困境
- 2026-11-17 07:55:01
- XRP 价格大幅下跌,反映了 ETF 资金外流和宏观经济担忧驱动的更广泛的市场趋势。
-
- 001k.bot 通过增强的数字资产管理彻底改变加密货币收益
- 2026-11-17 00:05:01
- 探索 001k.bot 的新加密货币赚取功能,加上强大的数字资产管理,如何改变加密货币持有者的游戏规则。
-

































