CPUs Gain Ground in AI Race as Intel and Ampere Assert CPU Viability

Advances in CPU hardware and software optimizations are making CPUs more viable for running generative AI chatbots and services, traditionally dominated by GPUs. Intel and Ampere have demonstrated promising performance with smaller language models, achieving token latencies suitable for practical use. While CPUs face memory bandwidth limitations compared to specialized accelerators, forthcoming features like MCR DIMMs and 4-bit operations aim to address these bottlenecks. This suggests that CPUs may become capable of handling modestly sized AI models, shifting the focus to optimizing for widespread adoption.

CPUs Gain Ground in Generative AI Race as Intel and Ampere Push Limits

Introduction

The realm of generative artificial intelligence (AI) has been largely dominated by graphics processing units (GPUs) and specialized accelerators due to their unparalleled computational power. However, as smaller and more widely deployable AI models emerge within enterprises, CPU manufacturers Intel and Ampere are asserting that their products can effectively handle these tasks. Recent advancements in software optimizations and mitigation of hardware bottlenecks have paved the way for CPUs to become viable options in the AI landscape.

Intel's Progress with Xeon Processors

At Intel's Vision event in April, CEO Pat Gelsinger showcased the company's achievements in adapting larger language models (LLMs) for execution on its Xeon platform. A live demonstration featuring the forthcoming Granite Rapids Xeon 6 processor revealed Meta's Llama2-70B model operating at 4-bit precision with an impressive second token latency of 82 milliseconds (ms).

Second token latency gauges the time required for an AI model to analyze a query and provide its first response. A lower latency translates to perceived performance enhancements. In terms of performance metrics, the observed 82ms latency corresponds to approximately 12 tokens per second.

This result represents a significant improvement over Intel's 5th-generation Xeon processors released in December, which exhibited a second token latency of 151ms.

Oracle's Results with Ampere's CPUs

Oracle has also published test data related to executing the Llama2-7B model on Ampere's Altra central processing units (CPUs). Utilizing a 64-core OCI A1 instance paired with a 4-bit quantized version of the model, Oracle achieved throughput rates ranging from 33 to 119 tokens per second for batch sizes of 1 and 16, respectively.

In the context of conversational chatbots, a larger batch size equates to a higher capacity to concurrently handle multiple queries. Oracle's testing revealed a direct correlation between batch size and throughput; however, the larger the batch size, the slower the model generated text. For instance, at a batch size of 16, Oracle attained its highest throughput performance, but the output rate was approximately 7.5 tokens per second per query. This delay would be noticeable from the end-user's perspective.

Oracle shared results across multiple batch sizes, while Intel's data is limited to batch size one. Intel has been contacted for further details on performance at higher batch sizes.

Causes of Improved Performance

According to Jeff Wittich, Ampere's chief product officer, these performance gains were largely attributed to custom software libraries and optimizations to Llama.cpp, developed in collaboration with Oracle. Both Oracle and Intel have since released performance metrics for Meta's newly launched Llama3 models, demonstrating similar performance characteristics.

Pending the accuracy of these performance claims – given the test parameters and our experience running 4-bit quantized models on CPUs – CPUs appear to be a viable option for executing small-scale models. In the near future, they may also be capable of handling moderately sized models, particularly at relatively small batch sizes.

Limitations and Ongoing Challenges

While Intel and Ampere have successfully demonstrated LLMs running on their respective CPU platforms, it is crucial to recognize that various compute and memory limitations prevent CPUs from completely replacing GPUs or dedicated accelerators for larger-scale models.

For models pushing the boundaries of generative AI, Ronak Shah, director of Xeon AI product management at Intel, emphasized that upcoming products like the Gaudi accelerator are specifically engineered for such tasks.

Overcoming Bottlenecks

Historically, conversations surrounding the execution of LLMs on CPUs have been subdued because, despite increasing core counts, conventional processors still fall short in terms of parallelism compared to modern GPUs and accelerators designed for AI workloads.

However, CPUs are undergoing significant enhancements. Modern units dedicate a substantial portion of their die space to features such as vector extensions or even specialized matrix math accelerators.

Intel incorporated the latter feature in its Sapphire Rapids Xeon Scalable processors released early last year. Each core is also equipped with Advanced Matrix Extensions (AMX), although not all stock-keeping units (SKUs) support AMX due to the flexibility of software-defined silicon.

As the name suggests, AMX extensions are tailored to accelerate matrix math calculations prevalent in deep learning workloads. Since its initial implementation, Intel has continuously refined its AMX engines for improved performance on larger models. This advancement is likely reflected in the upcoming Intel Xeon 6 processors slated for release later this year.

While Intel heavily relies on matrix acceleration, Ampere's Wittich explained that acceptable performance can be achieved using the two 128-bit vector units embedded in each of its AmpereOne and Altra cores. These vector units support FP16, BF16, INT8, and INT16 precision levels.

Memory Bottlenecks and MCR DIMMs

Despite their inferior performance in executing OPS or FLOPS compared to GPUs, CPUs possess a significant advantage: their independence from expensive and capacity-constrained high-bandwidth memory (HBM) modules.

As previously discussed, operating a model at FP8/INT8 requires approximately 1 gigabyte (GB) of memory for each billion parameters. Consequently, executing a model like OpenAI's 1.7 trillion parameter GPT-4 model at FP8 would necessitate over 1.7 terabytes (TB) of memory, which would be approximately halved when quantized to 4-bits. This memory requirement exceeds the capacity of any single GPU but falls within the capabilities of modern CPUs.

However, the drawback lies in the sluggish speed of large DRAM modules used by CPUs compared to HBM.

With only eight memory channels currently supported on Intel's 5th-generation Xeon and Ampere's One processors, these chips are limited to roughly 350 gigabytes per second (GB/sec) of memory bandwidth when running 5600MT/sec DIMMs. While Wittich mentioned plans for a 12-channel version of Ampere's chip with a targeted release later this year – featuring a purported 256 cores – it is not yet available.

Nevertheless, all of Oracle's testing has been conducted on Ampere's Altra generation, which utilizes even slower DDR4 memory and operates at a maximum bandwidth of approximately 200GB/sec. This suggests the potential for substantial performance gains by upgrading to the newer AmpereOne cores.

These bandwidth speeds may seem impressive – certainly faster than an SSD – but the eight HBM modules found on AMD's MI300X or Nvidia's upcoming Blackwell GPUs deliver speeds of 5.3 TB/sec and 8TB/sec, respectively. However, HBM modules are constrained by a maximum capacity of 192GB.

To illustrate this concept, consider memory capacity as a fuel tank, memory bandwidth as a fuel line, and compute as an internal combustion engine. Regardless of the size of the fuel tank or the power of the engine, if the fuel line is too narrow to supply sufficient fuel for optimal engine performance, the system will be hindered.

This limitation explains why previous attempts to execute LLMs on CPUs were largely confined to smaller models.

Clearing the Bottlenecks: Intel's Granite Rapids Xeon 6

Despite these obstacles, Intel's forthcoming Granite Rapids Xeon 6 platform provides clues on how CPUs could potentially handle larger models in the near future.

Intel's recent demonstration showcased a single Xeon 6 processor effortlessly running Llama2-70B with a reasonable second token latency of 82ms. Crucially, many details regarding the test rig remain unknown, including the number and clock speed of the cores. These details will likely be revealed later this year – potentially in December.

"The substantial advancement from 5th-generation Xeon to Xeon 6 lies in the introduction of MCR DIMMs, which effectively clears many of the bottlenecks associated with memory-bound workloads," explained Shah.

Multiplexer combined rank (MCR) DIMMs facilitate much faster memory access compared to standard DRAM. Intel has already demonstrated this technology running at 8,800MT/sec. With 12 memory channels equipped with MCR DIMMs, a single Granite Rapids socket would access approximately 825GB/sec of bandwidth – a significant leap from previous generations.

Wittich noted that Ampere is also exploring the implementation of MCR DIMMs but did not provide a timeline for their inclusion in Ampere silicon.

However, faster memory technology is not the sole innovation offered by Granite Rapids. Intel's AMX engine has gained support for 4-bit operations via the introduction of the MXFP4 data type, which theoretically has the potential to double effective performance.

Moreover, lower precision reduces the model footprint and subsequently lowers memory capacity and bandwidth requirements. Quantization techniques employed to compress models trained at higher precisions can also achieve similar reductions in footprint and bandwidth usage. Therefore, the practical benefit of supporting 4-bit mathematics in hardware primarily manifests as performance enhancements.

Balancing Act: Optimizing CPU Design for AI

For CPU designers, striking the right balance of AI capabilities presents a challenge. Excessive allocation of die area to features like AMX risks transforming the chip into an AI accelerator rather than a general-purpose processor.

As a result, instead of aiming for CPUs capable of handling the largest and most demanding LLMs, vendors are focusing on the distribution of AI models to identify the most widely adopted models and optimizing their products to cater to these workloads.

"From a customer perspective, the sweet spot right now revolves around models with 7–13 billion parameters. That's where most of our attention is directed today," stated Wittich.

Intel's Shah has observed a similar