市值: $2.1169T -2.34%
成交额(24h): $82.6304B 15.78%
  • 市值: $2.1169T -2.34%
  • 成交额(24h): $82.6304B 15.78%
  • 恐惧与贪婪指数:
  • 市值: $2.1169T -2.34%
加密货币
话题
百科
资讯
加密话题
视频
热门新闻
加密货币
话题
百科
资讯
加密话题
视频
bitcoin
bitcoin

$61605.656648 USD

-3.04%

ethereum
ethereum

$1741.712974 USD

-3.34%

tether
tether

$0.999452 USD

0.06%

bnb
bnb

$569.840001 USD

-3.10%

usd-coin
usd-coin

$1.000291 USD

0.05%

xrp
xrp

$0.962638 USD

-1.69%

solana
solana

$72.392737 USD

-3.10%

tron
tron

$0.309467 USD

-1.00%

hyperliquid
hyperliquid

$65.780663 USD

-3.44%

zcash
zcash

$850.819478 USD

-9.16%

monero
monero

$479.362672 USD

-4.57%

dogecoin
dogecoin

$0.057347 USD

-3.31%

unus-sed-leo
unus-sed-leo

$9.228306 USD

-1.21%

chainlink
chainlink

$8.839656 USD

-3.37%

cardano
cardano

$0.149404 USD

-2.68%

加密货币新闻

NVIDIA GH200 NVL32:彻底改变实时 AI 应用的首次令牌时间性能

2024/09/27 18:00

NVIDIA 最新的 GH200 NVL32 系统展示了首次令牌时间 (TTFT) 性能的显着飞跃,满足了 Llama 3.1 和 3.2 等大型语言模型 (LLM) 不断增长的需求。

NVIDIA GH200 NVL32:彻底改变实时 AI 应用的首次令牌时间性能

NVIDIA's latest GH200 NVL32 system demonstrates a remarkable leap in time-to-first-token (TTFT) performance, addressing the growing needs of large language models (LLMs) such as Llama 3.1 and 3.2. According to the NVIDIA Technical Blog, this system is set to significantly impact real-time applications like interactive speech bots and coding assistants.

NVIDIA 最新的 GH200 NVL32 系统展示了首次令牌时间 (TTFT) 性能的显着飞跃,满足了 Llama 3.1 和 3.2 等大型语言模型 (LLM) 不断增长的需求。据 NVIDIA 技术博客称,该系统将对交互式语音机器人和编码助手等实时应用产生重大影响。

TTFT is the time it takes for an LLM to process a user prompt and begin generating a response. As LLMs grow in complexity, with models like Llama 3.1 now featuring hundreds of billions of parameters, the need for faster TTFT becomes critical. This is particularly true for applications requiring immediate responses, such as AI-driven customer support and digital assistants.

TTFT 是法学硕士处理用户提示并开始生成响应所需的时间。随着 LLM 的复杂性不断增加,Llama 3.1 等模型现在具有数千亿个参数,对更快 TTFT 的需求变得至关重要。对于需要立即响应的应用程序尤其如此,例如人工智能驱动的客户支持和数字助理。

NVIDIA's GH200 NVL32 system, powered by 32 NVIDIA GH200 Grace Hopper Superchips and connected via the NVLink Switch system, is designed to meet these demands. The system leverages TensorRT-LLM improvements to deliver outstanding TTFT for long-context inference, making it ideal for the latest Llama 3.1 models.

NVIDIA 的 GH200 NVL32 系统由 32 个 NVIDIA GH200 Grace Hopper 超级芯片提供支持,并通过 NVLink 交换机系统连接,旨在满足这些需求。该系统利用 TensorRT-LLM 改进为长上下文推理提供出色的 TTFT,使其成为最新 Llama 3.1 模型的理想选择。

Applications like AI speech bots and digital assistants require TTFT in the range of a few hundred milliseconds to simulate natural, human-like conversations. For instance, a TTFT of half a second is significantly more user-friendly than a TTFT of five seconds. Fast TTFT is particularly crucial for services that rely on up-to-date information, such as agentic workflows that use Retrieval-Augmented Generation (RAG) to enhance LLM prompts with relevant data.

人工智能语音机器人和数字助理等应用需要几百毫秒范围内的 TTFT 来模拟自然的、类人的对话。例如,半秒的 TTFT 比 5 秒的 TTFT 更加用户友好。快速 TTFT 对于依赖最新信息的服务尤其重要,例如使用检索增强生成 (RAG) 来通过相关数据增强 LLM 提示的代理工作流程。

The NVIDIA GH200 NVL32 system achieves the fastest published TTFT for Llama 3.1 models, even with extensive context lengths. This performance is essential for real-time applications that demand quick and accurate responses.

NVIDIA GH200 NVL32 系统为 Llama 3.1 模型实现了最快的已发布 TTFT,即使上下文长度很长。这种性能对于需要快速、准确响应的实时应用程序至关重要。

The GH200 NVL32 system connects 32 NVIDIA GH200 Grace Hopper Superchips, each combining an NVIDIA Grace CPU and an NVIDIA Hopper GPU via NVLink-C2C. This setup allows for high-bandwidth, low-latency communication, essential for minimizing synchronization time and maximizing compute performance. The system delivers up to 127 petaFLOPs of peak FP8 AI compute, significantly reducing TTFT for demanding models with long contexts.

GH200 NVL32 系统连接 32 个 NVIDIA GH200 Grace Hopper 超级芯片,每个芯片通过 NVLink-C2C 结合了 NVIDIA Grace CPU 和 NVIDIA Hopper GPU。此设置可实现高带宽、低延迟通信,这对于最小化同步时间和最大化计算性能至关重要。该系统可提供高达 127 petaFLOPs 的峰值 FP8 AI 计算,显着降低具有长上下文的高要求模型的 TTFT。

For example, the system can achieve a TTFT of just 472 milliseconds for Llama 3.1 70B with an input sequence length of 32,768 tokens. Even for more complex models like Llama 3.1 405B, the system provides a TTFT of about 1.6 seconds using a 32,768-token input.

例如,对于输入序列长度为 32,768 个令牌的 Llama 3.1 70B,系统可以实现仅为 472 毫秒的 TTFT。即使对于 Llama 3.1 405B 等更复杂的模型,系统也可以使用 32,768 个令牌输入提供约 1.6 秒的 TTFT。

Inference continues to be a hotbed of innovation, with advancements in serving techniques, runtime optimizations, and more. Techniques like in-flight batching, speculative decoding, and FlashAttention are enabling more efficient and cost-effective deployments of powerful AI models.

随着服务技术、运行时优化等方面的进步,推理仍然是创新的温床。飞行中批处理、推测性解码和 FlashAttention 等技术可实现更高效、更具成本效益的强大 AI 模型部署。

NVIDIA's accelerated computing platform, supported by a vast ecosystem of developers and a broad installed base of GPUs, is at the forefront of these innovations. The platform's compatibility with the CUDA programming model and deep engagement with the developer community ensure rapid advancements in AI capabilities.

NVIDIA 的加速计算平台得到了庞大的开发者生态系统和广泛的 GPU 安装基础的支持,处于这些创新的前沿。该平台与CUDA编程模型的兼容性以及与开发者社区的深入参与确保了AI能力的快速进步。

Looking ahead, the NVIDIA Blackwell GB200 NVL72 platform promises even greater advancements. With second-generation Transformer Engine and fifth-generation Tensor Cores, Blackwell delivers up to 20 petaFLOPs of FP4 AI compute, significantly enhancing performance. The platform's fifth-generation NVLink provides 1,800 GB/s of GPU-to-GPU bandwidth, expanding the NVLink domain to 72 GPUs.

展望未来,NVIDIA Blackwell GB200 NVL72 平台有望取得更大的进步。凭借第二代 Transformer Engine 和第五代 Tensor Core,Blackwell 可提供高达 20 petaFLOPs 的 FP4 AI 计算,显着提高性能。该平台的第五代 NVLink 提供 1,800 GB/s 的 GPU 到 GPU 带宽,将 NVLink 域扩展到 72 个 GPU。

As AI models continue to grow and agentic workflows become more prevalent, the need for high-performance, low-latency computing solutions like the GH200 NVL32 and Blackwell GB200 NVL72 will only increase. NVIDIA's ongoing innovations ensure that the company remains at the forefront of AI and accelerated computing.

随着人工智能模型的不断发展和代理工作流程变得更加普遍,对 GH200 NVL32 和 Blackwell GB200 NVL72 等高性能、低延迟计算解决方案的需求只会增加。 NVIDIA 的持续创新确保该公司始终处于人工智能和加速计算领域的前沿。

原文来源:blockchain

免责声明:info@kdj.com

所提供的信息并非交易建议。根据本文提供的信息进行的任何投资,kdj.com不承担任何责任。加密货币具有高波动性,强烈建议您深入研究后,谨慎投资!

如您认为本网站上使用的内容侵犯了您的版权,请立即联系我们(info@kdj.com),我们将及时删除。

2026年11月20日 发表的其他文章