A modern AI chip can do about a thousand trillion calculations a second. Ask it one question, and the answer still arrives a fragment of a word at a time, while the part of the chip that does the arithmetic spends most of its time waiting. Waiting for what? We follow one question, "Why is the sky blue?", through the chip. The text becomes tokens and then whole numbers; each number becomes a list of thousands of numbers; and every layer of the model is mostly matrix multiplication, cut into tiles and shared across the chip's tensor cores. The question is read all at once. The answer can only be written one token at a time, and every new token needs nearly every one of the model's weights read from memory again: for a model with eight billion 16-bit weights, about 15 GB per token. At the 3.35 terabytes a second of NVIDIA's H100, that takes about 4.5 thousandths of a second, while the arithmetic itself would take about 15 millionths. Add the KV cache, the keys and values kept for every earlier token, and you see why AI chips are built around memory, and why data centres batch many conversations into one read, store weights in fewer bits and split big models across several chips. The token ids and the next-token probabilities shown are real: they come from a small open-weight model asked the same question for this film. The H100 figures are NVIDIA's published peak specifications. The chip on screen is a generic, stylised accelerator drawn for this film, not any real chip's layout; the timings are an idealised count (peak rates, no overheads), so real systems are slower; and the 8-billion-weight model has the published shape of Meta's Llama 3 8B. Chapters 0:00 One question, as numbers 1:15 Lists of numbers 2:15 Arithmetic in tiles 3:34 The memory wall 5:06 One token, one layer, one choice 6:31 More people, fewer bits, more chips 8:01 The answer, one piece at a time Sources - NVIDIA Technical Blog, "NVIDIA Hopper Architecture In-Depth" (2022): https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/ - NVIDIA, H100 Tensor Core GPU specifications: https://www.nvidia.com/en-us/data-center/h100/ - NVIDIA Technical Blog, "NVIDIA Ampere Architecture In-Depth" (2020): https://developer.nvidia.com/blog/nvidia-ampere-architecture-in-depth/ - NVIDIA Technical Blog, "Accelerated Inference for Large Transformer Models Using NVIDIA Triton Inference Server and FasterTransformer": https://developer.nvidia.com/blog/accelerated-inference-for-large-transformer-models-using-nvidia-fastertransformer-and-nvidia-triton-inference-server/ - NVIDIA Technical Blog, "Mastering LLM Techniques: Inference Optimization": https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/ - NVIDIA Deep Learning Performance Guide, "Matrix Multiplication Background": https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html - A. Vaswani et al., "Attention Is All You Need", NeurIPS 2017: https://arxiv.org/abs/1706.03762 - R. Sennrich, B. Haddow, A. Birch, "Neural Machine Translation of Rare Words with Subword Units", ACL 2016: https://arxiv.org/abs/1508.07909 - J. Kaplan et al., "Scaling Laws for Neural Language Models" (2020): https://arxiv.org/abs/2001.08361 - R. Pope et al., "Efficiently Scaling Transformer Inference", MLSys 2023: https://arxiv.org/abs/2211.05102 - W. Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention", SOSP 2023: https://arxiv.org/abs/2309.06180 - J. Ainslie et al., "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", EMNLP 2023: https://arxiv.org/abs/2305.13245 - Llama Team, AI @ Meta, "The Llama 3 Herd of Models" (2024): https://arxiv.org/abs/2407.21783 - T. Dettmers et al., "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale", NeurIPS 2022: https://arxiv.org/abs/2208.07339 - E. Frantar et al., "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers", ICLR 2023: https://arxiv.org/abs/2210.17323 - J. Lin et al., "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration", MLSys 2024: https://arxiv.org/abs/2306.00978 - SK hynix Newsroom, "SK hynix Develops Industry's First 12-Layer HBM3": https://news.skhynix.com/sk-hynix-develops-industrys-first-12-layer-hbm3/ Music and sound effects: original, made for this film. Fonts: Source Sans 3 (Adobe) and Figtree (Erik Kennedy), SIL Open Font License 1.1. #AI #Technology #Science #Engineering #Computing
The information provided is not trading advice. kdj.com does not assume any responsibility for any investments made based on the information provided in this article. Cryptocurrencies are highly volatile and it is highly recommended that you invest with caution after thorough research!
If you believe that the content used on this website infringes your copyright, please contact us immediately (info@kdj.com) and we will delete it promptly.