Zyda：用于语言建模的突破性 1.3 万亿代币开放数据集

Zyphra 宣布发布 Zyda，这是一个用于语言建模的突破性的包含 1.3 万亿代币的开放数据集。这一创新数据集将重新定义语言模型训练和研究的标准

Zyphra, a pioneer in the field of natural language processing (NLP), has announced the release of Zyda, a groundbreaking 1.3 trillion-token open dataset for language modeling. This massive and meticulously curated dataset is poised to redefine the standards of language model training and research, offering an unparalleled combination of size, quality, and accessibility.

自然语言处理 (NLP) 领域的先驱 Zyphra 宣布发布 Zyda，这是一个突破性的、包含 1.3 万亿代币的语言建模开放数据集。这个庞大且精心策划的数据集将重新定义语言模型训练和研究的标准，提供无与伦比的规模、质量和可访问性组合。

To create Zyda, several high-quality open datasets were combined and refined through a rigorous filtering and deduplication process. The resulting dataset boasts an impressive token count while maintaining the highest data quality standards. Zyda is primarily designed to facilitate advanced language modeling experiments and training at a scale that was previously unattainable with open datasets.

为了创建 Zyda，我们通过严格的过滤和重复数据删除流程组合并完善了多个高质量的开放数据集。生成的数据集拥有令人印象深刻的令牌数量，同时保持最高的数据质量标准。 Zyda 的主要目的是促进高级语言建模实验和训练，其规模是以前使用开放数据集无法实现的。

In comprehensive ablation studies, Zyda has consistently outperformed existing datasets, including Dolma, Fineweb, Pile, RefinedWeb, and SlimPajama. This makes Zyda a crucial resource for researchers and developers looking to contribute to the field of language modeling.

在综合消融研究中，Zyda 始终优于现有数据集，包括 Dolma、Fineweb、Pile、RefinedWeb 和 SlimPajama。这使得 Zyda 成为希望为语言建模领域做出贡献的研究人员和开发人员的重要资源。

Key Features of Zyda

Zyda 的主要特点

Zyda was meticulously crafted by merging seven well-respected open language modeling datasets: RefinedWeb, Starcoder, C4, Pile, Slimpajama, pe2so, and arXiv. Each dataset underwent a uniform post-processing pipeline designed to enhance quality and coherence.

Zyda 是通过合并七个备受推崇的开放语言建模数据集精心打造的：RefinedWeb、Starcoder、C4、Pile、Slimpajama、pe2so 和 arXiv。每个数据集都经过统一的后处理流程，旨在提高质量和一致性。

The creation process involved thorough syntactic filtering to eliminate low-quality documents, followed by an aggressive cross-deduplication pass. Many datasets contained significant overlaps due to common data sources like Common Crawl, making cross-deduplication particularly important. This extensive cleaning process reduced the initial 2 trillion tokens to a more refined and manageable 1.3 trillion.

创建过程涉及彻底的语法过滤以消除低质量文档，然后进行积极的交叉重复数据删除。由于 Common Crawl 等常见数据源，许多数据集包含大量重叠，因此交叉重复数据删除尤为重要。这种广泛的清理过程将最初的 2 万亿代币减少到更加精细和易于管理的 1.3 万亿。

The effectiveness of Zyda is evident in the performance of Zamba, a language model trained on Zyda. When compared to models trained on competing datasets, Zamba demonstrates significant strength on a per-token basis. This serves as a testament to Zyda’s superior quality and potential to drive language modeling advancements.

Zyda 的有效性在 Zamba（一种在 Zyda 上训练的语言模型）的性能中得到了体现。与在竞争数据集上训练的模型相比，Zamba 在每个代币的基础上表现出了显着的优势。这证明了 Zyda 的卓越品质和推动语言建模进步的潜力。

In conclusion, Zyda represents a monumental leap forward in the field of language modeling. By providing a massive, high-quality, open dataset, Zyphra is paving the way for the next generation of NLP research and applications. The release of Zyda not only underscores Zyphra’s leadership in the field but also sets a new benchmark for what is possible with open datasets.

总之，Zyda 代表了语言建模领域的巨大飞跃。通过提供海量、高质量、开放的数据集，Zyphra 正在为下一代 NLP 研究和应用铺平道路。 Zyda 的发布不仅凸显了 Zyphra 在该领域的领导地位，还为开放数据集的可能性树立了新的基准。