英特尔和 Ampere 证明 CPU 的可行性，CPU 在人工智能竞赛中取得进展

CPU 硬件和软件优化的进步使 CPU 更适合运行传统上由 GPU 主导的生成式 AI 聊天机器人和服务。英特尔和 Ampere 已经通过较小的语言模型展示了令人鼓舞的性能，实现了适合实际使用的令牌延迟。虽然与专用加速器相比，CPU 面临内存带宽限制，但即将推出的 MCR DIMM 和 4 位操作等功能旨在解决这些瓶颈。这表明 CPU 可能能够处理中等大小的人工智能模型，将重点转移到优化以实现广泛采用。

CPUs Gain Ground in Generative AI Race as Intel and Ampere Push Limits

随着英特尔和 Ampere 突破极限，CPU 在生成式 AI 竞赛中取得进展

Introduction

介绍

The realm of generative artificial intelligence (AI) has been largely dominated by graphics processing units (GPUs) and specialized accelerators due to their unparalleled computational power. However, as smaller and more widely deployable AI models emerge within enterprises, CPU manufacturers Intel and Ampere are asserting that their products can effectively handle these tasks. Recent advancements in software optimizations and mitigation of hardware bottlenecks have paved the way for CPUs to become viable options in the AI landscape.

生成式人工智能 (AI) 领域主要由图形处理单元 (GPU) 和专用加速器主导，因为它们具有无与伦比的计算能力。然而，随着企业内部出现更小且可广泛部署的人工智能模型，CPU 制造商英特尔和 Ampere 声称他们的产品可以有效地处理这些任务。软件优化和硬件瓶颈缓解方面的最新进展为 CPU 成为人工智能领域的可行选择铺平了道路。

Intel's Progress with Xeon Processors

英特尔至强处理器的进展

At Intel's Vision event in April, CEO Pat Gelsinger showcased the company's achievements in adapting larger language models (LLMs) for execution on its Xeon platform. A live demonstration featuring the forthcoming Granite Rapids Xeon 6 processor revealed Meta's Llama2-70B model operating at 4-bit precision with an impressive second token latency of 82 milliseconds (ms).

在 4 月份的英特尔愿景活动中，首席执行官帕特·基辛格 (Pat Gelsinger) 展示了该公司在采用更大的语言模型 (LLM) 以在其至强平台上执行方面取得的成就。采用即将推出的 Granite Rapids Xeon 6 处理器的现场演示揭示了 Meta 的 Llama2-70B 模型以 4 位精度运行，具有令人印象深刻的 82 毫秒 (ms) 秒令牌延迟。

Second token latency gauges the time required for an AI model to analyze a query and provide its first response. A lower latency translates to perceived performance enhancements. In terms of performance metrics, the observed 82ms latency corresponds to approximately 12 tokens per second.

第二个令牌延迟衡量人工智能模型分析查询并提供第一个响应所需的时间。较低的延迟意味着可感知的性能增强。就性能指标而言，观察到的 82 毫秒延迟相当于每秒大约 12 个令牌。

This result represents a significant improvement over Intel's 5th-generation Xeon processors released in December, which exhibited a second token latency of 151ms.

这一结果比英特尔去年 12 月发布的第五代至强处理器有了显着改进，后者的第二个令牌延迟为 151 毫秒。

Oracle's Results with Ampere's CPUs

Oracle 使用 Ampere CPU 得出的结果

Oracle has also published test data related to executing the Llama2-7B model on Ampere's Altra central processing units (CPUs). Utilizing a 64-core OCI A1 instance paired with a 4-bit quantized version of the model, Oracle achieved throughput rates ranging from 33 to 119 tokens per second for batch sizes of 1 and 16, respectively.

Oracle 还发布了在 Ampere 的 Altra 中央处理单元 (CPU) 上执行 Llama2-7B 模型的相关测试数据。利用 64 核 OCI A1 实例与模型的 4 位量化版本配对，Oracle 在批量大小为 1 和 16 的情况下分别实现了每秒 33 到 119 个令牌的吞吐率。

In the context of conversational chatbots, a larger batch size equates to a higher capacity to concurrently handle multiple queries. Oracle's testing revealed a direct correlation between batch size and throughput; however, the larger the batch size, the slower the model generated text. For instance, at a batch size of 16, Oracle attained its highest throughput performance, but the output rate was approximately 7.5 tokens per second per query. This delay would be noticeable from the end-user's perspective.

在对话式聊天机器人的上下文中，较大的批量大小相当于同时处理多个查询的更高容量。 Oracle 的测试揭示了批量大小和吞吐量之间的直接相关性；但是，批量大小越大，模型生成文本的速度就越慢。例如，在批量大小为 16 时，Oracle 获得了最高的吞吐量性能，但每个查询的输出率约为每秒 7.5 个令牌。从最终用户的角度来看，这种延迟是显而易见的。

Oracle shared results across multiple batch sizes, while Intel's data is limited to batch size one. Intel has been contacted for further details on performance at higher batch sizes.

Oracle 共享多个批次大小的结果，而英特尔的数据仅限于一个批次大小。已联系英特尔以获取有关更高批量大小的性能的更多详细信息。

Causes of Improved Performance

提高绩效的原因

According to Jeff Wittich, Ampere's chief product officer, these performance gains were largely attributed to custom software libraries and optimizations to Llama.cpp, developed in collaboration with Oracle. Both Oracle and Intel have since released performance metrics for Meta's newly launched Llama3 models, demonstrating similar performance characteristics.

Ampere 首席产品官 Jeff Wittich 表示，这些性能提升很大程度上归功于与 Oracle 合作开发的定制软件库和对 Llama.cpp 的优化。此后，甲骨文和英特尔都发布了 Meta 新推出的 Llama3 模型的性能指标，展示了类似的性能特征。

Pending the accuracy of these performance claims – given the test parameters and our experience running 4-bit quantized models on CPUs – CPUs appear to be a viable option for executing small-scale models. In the near future, they may also be capable of handling moderately sized models, particularly at relatively small batch sizes.

在这些性能声明的准确性之前——考虑到测试参数和我们在 CPU 上运行 4 位量化模型的经验——CPU 似乎是执行小规模模型的可行选择。在不久的将来，它们也可能能够处理中等大小的模型，特别是相对较小的批量大小。

Limitations and Ongoing Challenges

局限性和持续的挑战

While Intel and Ampere have successfully demonstrated LLMs running on their respective CPU platforms, it is crucial to recognize that various compute and memory limitations prevent CPUs from completely replacing GPUs or dedicated accelerators for larger-scale models.

虽然英特尔和 Ampere 已成功演示了在各自 CPU 平台上运行的法学硕士，但重要的是要认识到，各种计算和内存限制阻止 CPU 完全取代 GPU 或大型模型的专用加速器。

For models pushing the boundaries of generative AI, Ronak Shah, director of Xeon AI product management at Intel, emphasized that upcoming products like the Gaudi accelerator are specifically engineered for such tasks.

对于突破生成式人工智能边界的模型，英特尔至强人工智能产品管理总监 Ronak Shah 强调，即将推出的产品（例如 Gaudi 加速器）是专门为此类任务而设计的。

Overcoming Bottlenecks

克服瓶颈

Historically, conversations surrounding the execution of LLMs on CPUs have been subdued because, despite increasing core counts, conventional processors still fall short in terms of parallelism compared to modern GPUs and accelerators designed for AI workloads.

从历史上看，围绕在 CPU 上执行 LLM 的讨论一直很低调，因为尽管核心数量不断增加，但与专为 AI 工作负载设计的现代 GPU 和加速器相比，传统处理器在并行性方面仍然存在不足。

However, CPUs are undergoing significant enhancements. Modern units dedicate a substantial portion of their die space to features such as vector extensions or even specialized matrix math accelerators.

然而，CPU 正在经历显着的增强。现代单元将其芯片空间的很大一部分用于矢量扩展甚至专用矩阵数学加速器等功能。

Intel incorporated the latter feature in its Sapphire Rapids Xeon Scalable processors released early last year. Each core is also equipped with Advanced Matrix Extensions (AMX), although not all stock-keeping units (SKUs) support AMX due to the flexibility of software-defined silicon.

英特尔在去年初发布的 Sapphire Rapids Xeon 可扩展处理器中整合了后一个功能。每个核心还配备了高级矩阵扩展 (AMX)，但由于软件定义芯片的灵活性，并非所有库存单元 (SKU) 都支持 AMX。

As the name suggests, AMX extensions are tailored to accelerate matrix math calculations prevalent in deep learning workloads. Since its initial implementation, Intel has continuously refined its AMX engines for improved performance on larger models. This advancement is likely reflected in the upcoming Intel Xeon 6 processors slated for release later this year.

顾名思义，AMX 扩展专为加速深度学习工作负载中普遍存在的矩阵数学计算而定制。自最初实施以来，英特尔不断改进其 AMX 引擎，以提高大型型号的性能。这一进步可能会反映在即将于今年晚些时候发布的英特尔至强 6 处理器中。

While Intel heavily relies on matrix acceleration, Ampere's Wittich explained that acceptable performance can be achieved using the two 128-bit vector units embedded in each of its AmpereOne and Altra cores. These vector units support FP16, BF16, INT8, and INT16 precision levels.

虽然英特尔严重依赖矩阵加速，但 Ampere 的 Wittich 解释说，使用每个 AmpereOne 和 Altra 内核中嵌入的两个 128 位矢量单元可以获得可接受的性能。这些矢量单元支持 FP16、BF16、INT8 和 INT16 精度级别。

Memory Bottlenecks and MCR DIMMs

内存瓶颈和 MCR DIMM

Despite their inferior performance in executing OPS or FLOPS compared to GPUs, CPUs possess a significant advantage: their independence from expensive and capacity-constrained high-bandwidth memory (HBM) modules.

尽管与 GPU 相比，CPU 在执行 OPS 或 FLOPS 方面的性能较差，但它具有显着的优势：它们独立于昂贵且容量受限的高带宽内存 (HBM) 模块。

As previously discussed, operating a model at FP8/INT8 requires approximately 1 gigabyte (GB) of memory for each billion parameters. Consequently, executing a model like OpenAI's 1.7 trillion parameter GPT-4 model at FP8 would necessitate over 1.7 terabytes (TB) of memory, which would be approximately halved when quantized to 4-bits. This memory requirement exceeds the capacity of any single GPU but falls within the capabilities of modern CPUs.

如前所述，在 FP8/INT8 下运行模型时，每十亿个参数需要大约 1 GB 的内存。因此，在 FP8 上执行 OpenAI 的 1.7 万亿参数 GPT-4 模型等模型将需要超过 1.7 TB 的内存，当量化为 4 位时，内存将大约减半。此内存要求超出了任何单个 GPU 的容量，但在现代 CPU 的能力范围内。

However, the drawback lies in the sluggish speed of large DRAM modules used by CPUs compared to HBM.

但缺点是CPU使用的大型DRAM模块的速度比HBM慢。

With only eight memory channels currently supported on Intel's 5th-generation Xeon and Ampere's One processors, these chips are limited to roughly 350 gigabytes per second (GB/sec) of memory bandwidth when running 5600MT/sec DIMMs. While Wittich mentioned plans for a 12-channel version of Ampere's chip with a targeted release later this year – featuring a purported 256 cores – it is not yet available.

由于英特尔第 5 代 Xeon 和 Ampere's One 处理器目前仅支持 8 个内存通道，因此这些芯片在运行 5600MT/秒 DIMM 时，内存带宽限制为大约 350 GB/秒 (GB/秒)。虽然 Wittich 提到计划于今年晚些时候发布 Ampere 芯片的 12 通道版本（据称具有 256 个内核），但目前该版本尚未上市。

Nevertheless, all of Oracle's testing has been conducted on Ampere's Altra generation, which utilizes even slower DDR4 memory and operates at a maximum bandwidth of approximately 200GB/sec. This suggests the potential for substantial performance gains by upgrading to the newer AmpereOne cores.

尽管如此，Oracle 的所有测试都是在 Ampere 的 Altra 一代上进行的，该一代使用更慢的 DDR4 内存，并以大约 200GB/秒的最大带宽运行。这表明升级到较新的 AmpereOne 内核有可能大幅提升性能。

These bandwidth speeds may seem impressive – certainly faster than an SSD – but the eight HBM modules found on AMD's MI300X or Nvidia's upcoming Blackwell GPUs deliver speeds of 5.3 TB/sec and 8TB/sec, respectively. However, HBM modules are constrained by a maximum capacity of 192GB.

这些带宽速度可能看起来令人印象深刻（当然比 SSD 更快），但 AMD MI300X 或 Nvidia 即将推出的 Blackwell GPU 上的 8 个 HBM 模块可分别提供 5.3 TB/秒和 8TB/秒的速度。然而，HBM 模块的最大容量限制为 192GB。

To illustrate this concept, consider memory capacity as a fuel tank, memory bandwidth as a fuel line, and compute as an internal combustion engine. Regardless of the size of the fuel tank or the power of the engine, if the fuel line is too narrow to supply sufficient fuel for optimal engine performance, the system will be hindered.

为了说明这个概念，请将内存容量视为油箱，将内存带宽视为燃油管路，并将计算视为内燃机。无论油箱尺寸或发动机功率有多大，如果燃油管路太窄而无法提供足够的燃油以实现最佳发动机性能，系统就会受到阻碍。

This limitation explains why previous attempts to execute LLMs on CPUs were largely confined to smaller models.

这一限制解释了为什么之前在 CPU 上执行 LLM 的尝试主要局限于较小的模型。

Clearing the Bottlenecks: Intel's Granite Rapids Xeon 6

清除瓶颈：英特尔 Granite Rapids Xeon 6

Despite these obstacles, Intel's forthcoming Granite Rapids Xeon 6 platform provides clues on how CPUs could potentially handle larger models in the near future.

尽管存在这些障碍，英特尔即将推出的 Granite Rapids Xeon 6 平台为 CPU 如何在不久的将来处理更大的模型提供了线索。

Intel's recent demonstration showcased a single Xeon 6 processor effortlessly running Llama2-70B with a reasonable second token latency of 82ms. Crucially, many details regarding the test rig remain unknown, including the number and clock speed of the cores. These details will likely be revealed later this year – potentially in December.

英特尔最近的演示展示了单个 Xeon 6 处理器轻松运行 Llama2-70B，第二个令牌延迟合理为 82 毫秒。至关重要的是，有关测试装置的许多细节仍然未知，包括内核的数量和时钟速度。这些细节可能会在今年晚些时候（可能是 12 月）公布。

"The substantial advancement from 5th-generation Xeon to Xeon 6 lies in the introduction of MCR DIMMs, which effectively clears many of the bottlenecks associated with memory-bound workloads," explained Shah.

“从第 5 代 Xeon 到 Xeon 6 的重大进步在于 MC​​R DIMM 的引入，它有效地消除了与内存限制工作负载相关的许多瓶颈，”Shah 解释道。

Multiplexer combined rank (MCR) DIMMs facilitate much faster memory access compared to standard DRAM. Intel has already demonstrated this technology running at 8,800MT/sec. With 12 memory channels equipped with MCR DIMMs, a single Granite Rapids socket would access approximately 825GB/sec of bandwidth – a significant leap from previous generations.

与标准 DRAM 相比，多路复用器组合列 (MCR) DIMM 可以实现更快的内存访问速度。英特尔已经展示了该技术的运行速度为 8,800MT/秒。凭借配备 MCR DIMM 的 12 个内存通道，单个 Granite Rapids 插槽可访问约 825GB/秒的带宽，与前几代相比实现了显着飞跃。

Wittich noted that Ampere is also exploring the implementation of MCR DIMMs but did not provide a timeline for their inclusion in Ampere silicon.

Wittich 指出，Ampere 也在探索 MCR DIMM 的实施，但没有提供将其纳入 Ampere 芯片的时间表。

However, faster memory technology is not the sole innovation offered by Granite Rapids. Intel's AMX engine has gained support for 4-bit operations via the introduction of the MXFP4 data type, which theoretically has the potential to double effective performance.

然而，更快的内存技术并不是 Granite Rapids 提供的唯一创新。英特尔的 AMX 引擎通过引入 MXFP4 数据类型获得了对 4 位操作的支持，理论上有可能使有效性能翻倍。

Moreover, lower precision reduces the model footprint and subsequently lowers memory capacity and bandwidth requirements. Quantization techniques employed to compress models trained at higher precisions can also achieve similar reductions in footprint and bandwidth usage. Therefore, the practical benefit of supporting 4-bit mathematics in hardware primarily manifests as performance enhancements.

此外，较低的精度会减少模型占用空间，从而降低内存容量和带宽要求。用于压缩以更高精度训练的模型的量化技术也可以实现类似的占地面积和带宽使用量的减少。因此，在硬件中支持4位数学的实际好处主要表现为性能的增强。

Balancing Act: Optimizing CPU Design for AI

平衡法：针对 AI 优化 CPU 设计

For CPU designers, striking the right balance of AI capabilities presents a challenge. Excessive allocation of die area to features like AMX risks transforming the chip into an AI accelerator rather than a general-purpose processor.

对于 CPU 设计人员来说，在人工智能功能之间取得适当的平衡是一项挑战。为 AMX 等功能过多分配芯片面积可能会导致芯片转变为 AI 加速器而不是通用处理器。

As a result, instead of aiming for CPUs capable of handling the largest and most demanding LLMs, vendors are focusing on the distribution of AI models to identify the most widely adopted models and optimizing their products to cater to these workloads.

因此，供应商不再将目标放在能够处理最大、要求最高的 LLM 的 CPU 上，而是专注于 AI 模型的分发，以确定最广泛采用的模型，并优化其产品以满足这些工作负载。

"From a customer perspective, the sweet spot right now revolves around models with 7–13 billion parameters. That's where most of our attention is directed today," stated Wittich.

“从客户的角度来看，目前的最佳点围绕着具有 7-130 亿个参数的模型。这就是我们今天大部分注意力所关注的地方，”Wittich 说道。

Intel's Shah has observed a similar

英特尔的 Shah 也观察到了类似的情况