NVIDIA 的 HGX H200 AI 加速器凭借 NVIDIA 独家解码算法“Medusa”，在 Llama 3.1 推理方面获得了巨大提升。

性能取决于组合 GPU 作为“一个强大的 GPU”处理请求的能力，具有超快的 GPU 到 GPU 通信和先进的软件能力

NVIDIA's HGX H200 AI accelerators are getting a big boost in Llama 3.1 inferencing, thanks to a new NVIDIA-exclusive decoding algorithm called "Medusa."

得益于名为“Medusa”的全新 NVIDIA 独家解码算法，NVIDIA 的 HGX H200 AI 加速器在 Llama 3.1 推理方面获得了巨大提升。

As large language models (LLMs) continue to grow in size and complexity, multi-GPU compute is a must-have to deliver the low latency and high throughput that real-time generative AI applications demand.

随着大型语言模型 (LLM) 的规模和复杂性不断增长，多 GPU 计算是提供实时生成 AI 应用程序所需的低延迟和高吞吐量的必备条件。

Performance depends on the combined GPUs' ability to work together as “one mighty GPU” with ultra-fast GPU-to-GPU communication and advanced software able to take full advantage of the multiple GPUs. By splitting the calculations of each model layer across the available GPUs using a technique called tensor parallelism in tandem with advanced algorithms like speculative decoding, token generation latency can be reduced, delivering an interactive user experience.

性能取决于组合 GPU 作为“一个强大的 GPU”协同工作的能力，具有超快的 GPU 到 GPU 通信以及能够充分利用多个 GPU 的高级软件。通过使用张量并行技术与推测解码等高级算法将每个模型层的计算拆分到可用的 GPU 上，可以减少令牌生成延迟，从而提供交互式用户体验。

For very low latency Llama 3.1 serving, cloud services can use a full NVIDIA HGX H200 server, each incorporating eight H200 Tensor Core GPUs and four all-to-all NVLink Switch chips. Each GPU within the server can communicate at the full 900 GB/s bandwidth to any other GPU via NVLink Switch. High GPU-to-GPU fabric bandwidth is required to keep multi-GPU communication from becoming the bottleneck in interactive use cases.

对于极低延迟的 Llama 3.1 服务，云服务可以使用完整的 NVIDIA HGX H200 服务器，每个服务器包含八个 H200 Tensor Core GPU 和四个全对全 NVLink Switch 芯片。服务器内的每个 GPU 都可以通过 NVLink Switch 以 900 GB/s 的带宽与任何其他 GPU 进行通信。为了防止多 GPU 通信成为交互式用例中的瓶颈，需要高 GPU 到 GPU 结构带宽。

To efficiently implement optimization algorithms on NVIDIA H200 HGX systems, NVIDIA TensorRT-LLM is used. TensorRT-LLM is an open-source TensorRT library that delivers state-of-the-art inference performance on the latest LLMs using a variety of techniques, including tensor parallelism and speculative decoding.

为了在 NVIDIA H200 HGX 系统上高效实施优化算法，使用了 NVIDIA TensorRT-LLM。 TensorRT-LLM 是一个开源 TensorRT 库，它使用各种技术（包括张量并行性和推测解码）在最新的 LLM 上提供最先进的推理性能。

Upcoming TensorRT-LLM optimizations, including the improvement of a speculative decoding algorithm called Medusa, provide outstanding low latency performance on Llama 3.1 70B and Llama 3.1 405B of 268 tokens/second/user and 108 tokens/second/user, respectively on HGX H200.

即将推出的 TensorRT-LLM 优化（包括名为 Medusa 的推测性解码算法的改进）可在 HGX H200 上的 Llama 3.1 70B 和 Llama 3.1 405B 上提供出色的低延迟性能，分别为 268 个令牌/秒/用户和 108 个令牌/秒/用户。

Medusa boosts token generation by up to 1.9x on NVIDIA HGX H200

Medusa 在 NVIDIA HGX H200 上将代币生成量提高了 1.9 倍

Transformer-based LLMs are auto-regressive, meaning that tokens need to be generated sequentially, limiting throughput per generation step to just one token. Typically, during LLM inference, the rate at which a single token is generated depends on how quickly model weights are loaded into memory. This means that the workload can leave the substantial Tensor Core capabilities of H200 GPUs underutilized.

基于 Transformer 的 LLM 是自回归的，这意味着需要按顺序生成令牌，将每个生成步骤的吞吐量限制为一个令牌。通常，在 LLM 推理期间，生成单个标记的速率取决于模型权重加载到内存中的速度。这意味着工作负载可能导致 H200 GPU 的大量 Tensor Core 功能未得到充分利用。

Speculative decoding is a technique that increases token generation throughput per token generation step by using a “draft model” to try to predict multiple subsequent tokens beyond the next token. The target LLM then “batches” the prediction candidates and validates them in parallel with the next token, making more effective use of available parallel GPU compute resources. If the original LLM accepts any candidate sequence, multiple tokens are generated in the generation step and therefore accelerate token generation.

推测性解码是一种通过使用“草稿模型”尝试预测下一个令牌之外的多个后续令牌来增加每个令牌生成步骤的令牌生成吞吐量的技术。然后，目标 LLM 对预测候选进行“批量”处理，并与下一个标记并行验证它们，从而更有效地利用可用的并行 GPU 计算资源。如果原始LLM接受任何候选序列，则在生成步骤中生成多个令牌，从而加速令牌生成。

Medusa, described in this paper, is a speculative decoding algorithm that uses the original model as the draft model, avoiding the system complexity and distribution discrepancy of using a separate draft model. This technique employs additional decoding “heads”, called Medusa heads, to predict candidate tokens beyond the next token. Each Medusa head generates a distribution of tokens beyond the previous.

本文描述的 Medusa 是一种推测性解码算法，使用原始模型作为草稿模型，避免了使用单独的草稿模型带来的系统复杂性和分布差异。该技术采用额外的解码“头”（称为 Medusa 头）来预测下一个标记之外的候选标记。每个美杜莎头都会生成超出前一个头的代币分布。

With Medusa, an HGX H200 can produce 268 tokens per second per user for Llama 3.1 70B and 108 for Llama 3.1 405B. This is over 1.5x faster on Llama 3.1 70B and over 1.9x faster on Llama 3.1 405B than without Medusa. Although there is variability in the Medusa acceptance rate between tasks depending on how the heads are fine-tuned, its overall performance is generalized across a wide range of tasks.

借助 Medusa，HGX H200 每秒可为 Llama 3.1 70B 生成 268 个代币，为 Llama 3.1 405B 生成 108 个代币。与没有 Medusa 的情况相比，Llama 3.1 70B 上的速度快了 1.5 倍以上，Llama 3.1 405B 上的速度快了 1.9 倍以上。尽管不同任务之间的美杜莎接受率存在差异，具体取决于头部的微调方式，但其整体性能在广泛的任务中是通用的。

Medusa heads for both Llama 3.1 70B and Llama 3.1 405B were trained using the NVIDIA TensorRT Model Optimizer integration with the NVIDIA NeMo framework. The Medusa head training used a frozen backbone, ensuring that the use of Medusa yields identical accuracy to the base model.

Llama 3.1 70B 和 Llama 3.1 405B 的 Medusa 头均使用 NVIDIA TensorRT 模型优化器与 NVIDIA NeMo 框架集成进行训练。美杜莎头部训练使用了冷冻骨干，确保使用美杜莎产生与基础模型相同的精度。

NVIDIA full-stack innovation never stops

NVIDIA全栈创新永不停歇

NVIDIA HGX H200 with NVLink Switch and TensorRT-LLM already deliver excellent real-time inference performance on popular and demanding community models. To continue improving user experiences and reduce inference costs, we relentlessly innovate across every layer of the technology stack – chips, systems, software libraries, algorithms, and more.

配备 NVLink Switch 和 TensorRT-LLM 的 NVIDIA HGX H200 已经在流行且要求严格的社区模型上提供了出色的实时推理性能。为了持续改善用户体验并降低推理成本，我们在技术堆栈的每一层（芯片、系统、软件库、算法等）不断创新。

We look forward to sharing future updates on our low latency inference performance as both our platform and the LLM ecosystem advances.

随着我们的平台和 LLM 生态系统的进步，我们期待分享我们的低延迟推理性能的未来更新。