Market Cap: $2.1973T 0.61%
Volume(24h): $72.7825B 5.15%
  • Market Cap: $2.1973T 0.61%
  • Volume(24h): $72.7825B 5.15%
  • Fear & Greed Index:
  • Market Cap: $2.1973T 0.61%
Cryptos
Topics
Cryptospedia
News
CryptosTopics
Videos
Top News
Cryptos
Topics
Cryptospedia
News
CryptosTopics
Videos
bitcoin
bitcoin

$64602.483777 USD

0.17%

ethereum
ethereum

$1828.427857 USD

1.11%

tether
tether

$0.999045 USD

0.00%

bnb
bnb

$592.614065 USD

0.38%

usd-coin
usd-coin

$0.999741 USD

-0.01%

xrp
xrp

$0.959759 USD

0.91%

solana
solana

$72.779464 USD

2.85%

tron
tron

$0.306254 USD

1.29%

hyperliquid
hyperliquid

$69.655101 USD

5.51%

zcash
zcash

$922.130514 USD

4.31%

monero
monero

$497.839010 USD

3.19%

dogecoin
dogecoin

$0.058918 USD

0.80%

unus-sed-leo
unus-sed-leo

$9.310662 USD

1.30%

chainlink
chainlink

$9.166951 USD

0.95%

cardano
cardano

$0.153248 USD

0.94%

Cryptocurrency News Articles

CoCoMix: A Different Approach to Pretraining Language Models

Feb 14, 2025 at 04:28 am

The dominant approach to pretraining large language models (LLMs) relies on next-token prediction, which has proven effective in capturing linguistic patterns. However, this method comes with notable limitations.

CoCoMix: A Different Approach to Pretraining Language Models

Large language models (LLMs) have become increasingly powerful thanks to pretraining on vast text corpora. But how can we improve their efficiency and interpretability? Meta AI introduces Continuous Concept Mixing (CoCoMix), a novel approach that combines token prediction with concept-based reasoning.

Most LLMs are pretrained on a cloze-style next-token prediction task. While this method captures linguistic patterns, it also has limitations. Language tokens often convey only surface-level information, requiring models to process massive amounts of text to develop deeper reasoning capabilities. Furthermore, token-based learning has difficulty capturing long-term dependencies, making tasks that require planning and abstraction more challenging.

To address these limitations, researchers have explored alternative strategies, such as knowledge distillation and structured input augmentation. However, these approaches do not fully resolve the limitations of token-based learning.

This raises an important question: Can LLMs be trained in a way that combines token-level processing with conceptual understanding? Meta AI introduces Continuous Concept Mixing (CoCoMix) as a potential solution.

CoCoMix integrates token prediction with the modeling of continuous concepts derived from hidden states of a pretrained model. The method employs a Sparse Autoencoder (SAE) to extract high-level semantic representations, which are then incorporated into the training process by interleaving them with token embeddings.

This design allows the model to maintain the benefits of token-based learning while enhancing its ability to recognize and process broader conceptual structures. By enriching the token-based paradigm with concept-level information, CoCoMix aims to improve reasoning efficiency and model interpretability.

Meta AI introduces Continuous Concept Mixing (CoCoMix), a different approach to pretraining large language models (LLMs). This method combines token prediction with concept-based reasoning to enhance efficiency and interpretability.

In a nutshell, CoCoMix operates through three main components:

A token-based language model is trained on a cloze-style next-token prediction task. The model learns to predict the next token in a sequence given the preceding context. This component forms the basis of standard LLM pretraining.

A Sparse Autoencoder (SAE) is trained on the hidden states of the language model. The SAE aims to extract high-level semantic representations from the language model's hidden states. This step is crucial for deriving continuous concepts.

The token embeddings from the language model are interleaved with the continuous concepts extracted by the SAE. This interleaving is performed at each training step. The resulting combined embeddings are then used to train the language model.

This approach improves sample efficiency, enabling models to achieve comparable performance with fewer training tokens. Additionally, CoCoMix enhances interpretability by making it possible to inspect and adjust the extracted concepts, offering a clearer view of how the model processes information.

Meta AI evaluated CoCoMix across multiple benchmarks, including OpenWebText, LAMBADA, WikiText-103, HellaSwag, PIQA, SIQA, Arc-Easy, and WinoGrande. The findings indicate:

On OpenWebText, LAMBADA, and WikiText-103, models pretrained with CoCoMix achieve better performance in downstream tasks compared to models pretrained with standard token-based methods. This improvement is observed across a range of tasks, including masked language modeling, cloze-style next-token prediction, and question answering.

Models pretrained with CoCoMix demonstrate improved sample efficiency. Specifically, they attain performance comparable to models pretrained with standard methods but using 4x fewer training tokens. This result highlights the efficiency gains achieved by incorporating continuous concepts into the pretraining process.

CoCoMix enables the extraction of interpretable concepts that provide insights into the model's reasoning process. By adjusting the hyperparameters of the SAE, researchers can control the granularity and interpretability of the extracted concepts.

Overall, CoCoMix presents an alternative approach to LLM pretraining by combining token prediction with concept-based reasoning. This method enhances efficiency and interpretability without disrupting the underlying next-token prediction framework. Experimental results suggest that this technique provides a balanced way to improve language model training, particularly in areas requiring structured reasoning and transparent decision-making.

Future research may focus on refining concept extraction methods and further integrating continuous representations into pretraining workflows.

Original source:marktechpost

Disclaimer:info@kdj.com

The information provided is not trading advice. kdj.com does not assume any responsibility for any investments made based on the information provided in this article. Cryptocurrencies are highly volatile and it is highly recommended that you invest with caution after thorough research!

If you believe that the content used on this website infringes your copyright, please contact us immediately (info@kdj.com) and we will delete it promptly.

Other articles published on Nov 18, 2026