LLM 弱点暴露：“故障代币”损害模型准确性

标记化是训练大型语言模型 (LLM) 的重要步骤，它可能会引入因标记生成器和模型训练数据集之间的不一致而导致的“故障标记”。为了解决这个问题，研究人员提出了一种使用嵌入权重分析来检测训练不足的令牌的自动化方法。通过分析嵌入矩阵差异，该方法识别需要进一步训练的标记，显着提高自然语言处理应用中的 LLM 准确性和鲁棒性。

Under-Trained Tokens: A Hidden Vulnerability in Large Language Models

训练不足的令牌：大型语言模型中的隐藏漏洞

Introduction

介绍

Tokenization, the process of decomposing text into manageable units known as tokens, serves as the foundation for training and operating large language models (LLMs). While effective tokenization yields significant performance enhancements, it becomes problematic when tokens within the model's vocabulary are underrepresented or completely absent in the training data, giving rise to the phenomenon of 'glitch tokens.' These tokens, upon encountering new input data, can destabilize the model, leading to unpredictable and erroneous outputs.

标记化是将文本分解为可管理单元（称为标记）的过程，是训练和操作大型语言模型 (LLM) 的基础。虽然有效的标记化可以显着提高性能，但当模型词汇表中的标记在训练数据中代表性不足或完全不存在时，就会出现问题，从而导致“故障标记”现象。这些令牌在遇到新的输入数据时可能会破坏模型的稳定性，从而导致不可预测和错误的输出。

Discrepancies in Tokenization Training

代币化培训中的差异

A fundamental issue with LLMs lies in the disparity between tokenizer training and model training procedures. Tokenizers are often trained separately using distinct datasets, which may differ substantially from the data utilized for training the model. This misalignment can result in some vocabulary tokens being under-trained, rendering them susceptible to the glitch token issue. The infamous "_SolidGoldMagikarp" token epitomizes this problem, as it can trigger unwanted model behaviors, such as hallucinations and nonsensical outputs.

法学硕士的一个基本问题在于分词器训练和模型训练程序之间的差异。分词器通常使用不同的数据集进行单独训练，这些数据集可能与用于训练模型的数据有很大不同。这种不一致可能会导致一些词汇标记训练不足，从而使它们容易受到故障标记问题的影响。臭名昭著的“_SolidGoldMagikarp”令牌集中体现了这个问题，因为它可以触发不需要的模型行为，例如幻觉和无意义的输出。

Conventional Glitch Token Identification Methods

传统的故障令牌识别方法

Traditionally, identifying under-trained tokens has involved manual inspection of the tokenizer's behavior, assessing how tokens are encoded and decoded, or meticulously analyzing their frequency within the training data. However, these methods lack scalability, proving ineffective for the increasingly vast and complex LLMs being developed today.

传统上，识别训练不足的令牌需要手动检查令牌生成器的行为，评估令牌的编码和解码方式，或者仔细分析它们在训练数据中的频率。然而，这些方法缺乏可扩展性，对于当今正在开发的日益庞大和复杂的法学硕士来说证明是无效的。

A Novel Approach: Leveraging Embedding Weights

一种新颖的方法：利用嵌入权重

Researchers from Cohere have devised a groundbreaking approach that leverages the model's embedding weights to automate and scale the detection of under-trained tokens. By meticulously analyzing these weights, the researchers have developed a method to pinpoint anomalies indicative of insufficient training. This method scrutinizes the embedding matrix of a model, identifying tokens whose embedding weights deviate significantly from those of well-represented tokens. It provides a systematic approach to detecting glitch tokens by calculating the variance and distribution of embedding weights and comparing them against a normative model of adequately trained tokens.

Cohere 的研究人员设计了一种突破性的方法，利用模型的嵌入权重来自动化和扩展对训练不足的标记的检测。通过仔细分析这些权重，研究人员开发了一种方法来查明表明训练不足的异常情况。该方法仔细检查模型的嵌入矩阵，识别嵌入权重与具有良好代表性的令牌的嵌入权重显着偏差的令牌。它提供了一种系统方法，通过计算嵌入权重的方差和分布并将其与经过充分训练的令牌的规范模型进行比较来检测故障令牌。

Experimental Validation

实验验证

The study's efficacy was demonstrated by applying the method to several prominent models, including variants of Google's BERT and OpenAI's GPT series. The analysis revealed a substantial percentage of the tokenizer's vocabulary, reaching up to 10% in some instances, as under-trained. These tokens were frequently specialized or infrequently used words, which exhibited the most significant discrepancies in embedding weight patterns.

该研究的有效性通过将该方法应用于几个著名模型（包括 Google 的 BERT 和 OpenAI 的 GPT 系列的变体）得到证明。分析显示，分词器的词汇量中很大一部分是由于训练不足而造成的，在某些情况下高达 10%。这些标记经常是专门的或不常用的单词，它们在嵌入权重模式中表现出最显着的差异。

Implications for LLM Development and Maintenance

对法学硕士开发和维护的影响

This research holds pivotal implications for the development and maintenance of LLMs. Automating the detection and remediation of under-trained tokens empowers developers to enhance the accuracy and robustness of language models. This advancement is critical as LLMs find increasing application in a wide range of tasks, from automated writing assistants to sophisticated conversational agents.

这项研究对法学硕士的发展和维护具有关键意义。自动检测和修复未经训练的令牌使开发人员能够提高语言模型的准确性和稳健性。这一进步至关重要，因为法学硕士在从自动写作助理到复杂的对话代理等各种任务中的应用越来越多。

Conclusion

结论

This research uncovers a critical vulnerability in LLM training and presents a scalable solution to combat this issue. Adopting automated methods for detecting under-trained tokens enables more robust training processes, ensuring that all tokens within a model's vocabulary are adequately prepared for real-world scenarios. By rectifying this vulnerability, this research significantly improves the efficacy and reliability of language models, paving the way for more trustworthy and effective natural language processing tools.

这项研究发现了法学硕士培训中的一个关键漏洞，并提出了一个可扩展的解决方案来解决这个问题。采用自动化方法来检测训练不足的标记可以实现更强大的训练过程，确保模型词汇表中的所有标记都为现实场景做好充分准备。通过纠正这一漏洞，这项研究显着提高了语言模型的有效性和可靠性，为更值得信赖和有效的自然语言处理工具铺平了道路。