NVIDIA 的 HGX H200 AI 加速器憑藉 NVIDIA 獨家解碼演算法“Medusa”，在 Llama 3.1 推理方面獲得了巨大提升。

效能取決於組合 GPU 作為「一個強大的 GPU」處理請求的能力，具有超快的 GPU 到 GPU 通訊和先進的軟體能力

NVIDIA's HGX H200 AI accelerators are getting a big boost in Llama 3.1 inferencing, thanks to a new NVIDIA-exclusive decoding algorithm called "Medusa."

由於名為「Medusa」的全新 NVIDIA 獨家解碼演算法，NVIDIA 的 HGX H200 AI 加速器在 Llama 3.1 推理方面獲得了巨大提升。

As large language models (LLMs) continue to grow in size and complexity, multi-GPU compute is a must-have to deliver the low latency and high throughput that real-time generative AI applications demand.

隨著大型語言模型 (LLM) 的規模和複雜性不斷增長，多 GPU 運算是提供即時產生 AI 應用程式所需的低延遲和高吞吐量的必備條件。

Performance depends on the combined GPUs' ability to work together as “one mighty GPU” with ultra-fast GPU-to-GPU communication and advanced software able to take full advantage of the multiple GPUs. By splitting the calculations of each model layer across the available GPUs using a technique called tensor parallelism in tandem with advanced algorithms like speculative decoding, token generation latency can be reduced, delivering an interactive user experience.

效能取決於組合 GPU 作為「一個強大的 GPU」協同工作的能力，具有超快的 GPU 到 GPU 通訊以及能夠充分利用多個 GPU 的高級軟體。透過使用張量平行技術與推測解碼等進階演算法將每個模型層的運算拆分到可用的 GPU 上，可以減少令牌產生延遲，從而提供互動式使用者體驗。

For very low latency Llama 3.1 serving, cloud services can use a full NVIDIA HGX H200 server, each incorporating eight H200 Tensor Core GPUs and four all-to-all NVLink Switch chips. Each GPU within the server can communicate at the full 900 GB/s bandwidth to any other GPU via NVLink Switch. High GPU-to-GPU fabric bandwidth is required to keep multi-GPU communication from becoming the bottleneck in interactive use cases.

對於極低延遲的 Llama 3.1 服務，雲端服務可以使用完整的 NVIDIA HGX H200 伺服器，每個伺服器包含八個 H200 Tensor Core GPU 和四個全對全 NVLink Switch 晶片。伺服器內的每個 GPU 都可以透過 NVLink Switch 以 900 GB/s 的頻寬與任何其他 GPU 進行通訊。為了防止多 GPU 通訊成為互動式用例中的瓶頸，需要高 GPU 到 GPU 結構頻寬。

To efficiently implement optimization algorithms on NVIDIA H200 HGX systems, NVIDIA TensorRT-LLM is used. TensorRT-LLM is an open-source TensorRT library that delivers state-of-the-art inference performance on the latest LLMs using a variety of techniques, including tensor parallelism and speculative decoding.

為了在 NVIDIA H200 HGX 系統上有效實施最佳化演算法，使用了 NVIDIA TensorRT-LLM。 TensorRT-LLM 是一個開源 TensorRT 庫，它使用各種技術（包括張量並行性和推測解碼）在最新的 LLM 上提供最先進的推理性能。

Upcoming TensorRT-LLM optimizations, including the improvement of a speculative decoding algorithm called Medusa, provide outstanding low latency performance on Llama 3.1 70B and Llama 3.1 405B of 268 tokens/second/user and 108 tokens/second/user, respectively on HGX H200.

即將推出的TensorRT-LLM 優化（包括名為Medusa 的推測性解碼演算法的改進）可在HGX H200 上的Llama 3.1 70B 和Llama 3.1 405B 上提供出色的低延遲性能，分別為268 個令牌/秒/使用者和108 個令牌/秒/使用者。

Medusa boosts token generation by up to 1.9x on NVIDIA HGX H200

Medusa 在 NVIDIA HGX H200 上將代幣生成量提高了 1.9 倍

Transformer-based LLMs are auto-regressive, meaning that tokens need to be generated sequentially, limiting throughput per generation step to just one token. Typically, during LLM inference, the rate at which a single token is generated depends on how quickly model weights are loaded into memory. This means that the workload can leave the substantial Tensor Core capabilities of H200 GPUs underutilized.

基於 Transformer 的 LLM 是自回歸的，這意味著需要按順序產生令牌，將每個生成步驟的吞吐量限制為一個令牌。通常，在 LLM 推理期間，產生單一標記的速率取決於模型權重載入到記憶體中的速度。這意味著工作負載可能導致 H200 GPU 的大量 Tensor Core 功能未充分利用。

Speculative decoding is a technique that increases token generation throughput per token generation step by using a “draft model” to try to predict multiple subsequent tokens beyond the next token. The target LLM then “batches” the prediction candidates and validates them in parallel with the next token, making more effective use of available parallel GPU compute resources. If the original LLM accepts any candidate sequence, multiple tokens are generated in the generation step and therefore accelerate token generation.

推測性解碼是一種透過使用「草稿模型」嘗試預測下一個令牌之外的多個後續令牌來增加每個令牌產生步驟的令牌產生吞吐量的技術。然後，目標 LLM 對預測候選進行「批次」處理，並與下一個標記並行驗證它們，從而更有效地利用可用的平行 GPU 運算資源。如果原始LLM接受任何候選序列，則在生成步驟中產生多個令牌，從而加速令牌生成。

Medusa, described in this paper, is a speculative decoding algorithm that uses the original model as the draft model, avoiding the system complexity and distribution discrepancy of using a separate draft model. This technique employs additional decoding “heads”, called Medusa heads, to predict candidate tokens beyond the next token. Each Medusa head generates a distribution of tokens beyond the previous.

本文所描述的 Medusa 是一種推測性解碼演算法，使用原始模型作為草稿模型，避免了使用單獨的草稿模型所帶來的系統複雜性和分佈差異。該技術採用額外的解碼「頭」（稱為 Medusa 頭）來預測下一個標記之外的候選標記。每個美杜莎頭都會產生超出前一個頭的代幣分佈。

With Medusa, an HGX H200 can produce 268 tokens per second per user for Llama 3.1 70B and 108 for Llama 3.1 405B. This is over 1.5x faster on Llama 3.1 70B and over 1.9x faster on Llama 3.1 405B than without Medusa. Although there is variability in the Medusa acceptance rate between tasks depending on how the heads are fine-tuned, its overall performance is generalized across a wide range of tasks.

透過 Medusa，HGX H200 每秒可為 Llama 3.1 70B 產生 268 個代幣，為 Llama 3.1 405B 產生 108 個代幣。與沒有 Medusa 的情況相比，Llama 3.1 70B 上的速度快了 1.5 倍以上，Llama 3.1 405B 上的速度快了 1.9 倍以上。儘管不同任務之間的美杜莎接受率存在差異，具體取決於頭部的微調方式，但其整體性能在廣泛的任務中是通用的。

Medusa heads for both Llama 3.1 70B and Llama 3.1 405B were trained using the NVIDIA TensorRT Model Optimizer integration with the NVIDIA NeMo framework. The Medusa head training used a frozen backbone, ensuring that the use of Medusa yields identical accuracy to the base model.

Llama 3.1 70B 和 Llama 3.1 405B 的 Medusa 頭均使用 NVIDIA TensorRT 模型優化器與 NVIDIA NeMo 框架整合進行訓練。美杜莎頭部訓練使用了冷凍骨幹，確保使用美杜莎產生與基礎模型相同的精確度。

NVIDIA full-stack innovation never stops

NVIDIA全端創新永不停歇

NVIDIA HGX H200 with NVLink Switch and TensorRT-LLM already deliver excellent real-time inference performance on popular and demanding community models. To continue improving user experiences and reduce inference costs, we relentlessly innovate across every layer of the technology stack – chips, systems, software libraries, algorithms, and more.

配備 NVLink Switch 和 TensorRT-LLM 的 NVIDIA HGX H200 已經在流行且要求嚴格的社區模型上提供了出色的即時推理性能。為了持續改善使用者體驗並降低推理成本，我們在技術堆疊的每一層（晶片、系統、軟體庫、演算法等）不斷創新。

We look forward to sharing future updates on our low latency inference performance as both our platform and the LLM ecosystem advances.

隨著我們的平台和 LLM 生態系統的進步，我們期待分享我們的低延遲推理性能的未來更新。