Every Line of This AI Model's Config File Is a Paper

#qwen #deepseek #flashattention Nobody can keep up with AI research, and the reading lists people hand you are ranked by how interesting a paper is rather than by what it costs you. Open the config file of a frontier open model instead and the ranking writes itself: every line in it is a paper somebody had to publish first, and only a few of them decide whether your serving bill is payable. This video reads one config file line by line and ranks the eleven papers behind it by what they actually buy you. It starts at a 2.4 trillion parameter file where 512 experts sit beside 10 experts per token, so 95 billion parameters switch on and the rest sleep, and traces that line back to Shazeer's 2017 sparsely gated mixture of experts and its claimed 1000x capacity gain at 137 billion parameters. Then FlashAttention, where the insight was IO awareness rather than better maths: 3x on GPT-2 in the first paper, 35 to 75 percent utilisation on Hopper in version 3, 1,613 teraflops per second and 71 percent of a B200 in version 4, plus four releases in four weeks against a four year old paper. Rotary embeddings arrive as a Chinese blog post from Jianlin Su four weeks before RoFormer, and turn up as rope theta of 10 million in Qwen, 8 million in GLM and 10,000 with a scaling block in DeepSeek. The cache lineage runs from multi query attention in 2019 through grouped query attention at 64 query heads to 4 key value heads, paged attention and vLLM's 2 to 4x throughput, to DeepSeek's latent attention cutting 93.3 percent of the cache, a 1,048,576 token context computed from 65,536 by a factor of 16, and cached input at a thirty-first of the cold price. The other seven get read too: chain of thought and the University of Texas audit that found gains of 14.2 points on symbolic tasks and 12.3 on maths but very little elsewhere, ReAct and its three known failure modes, GraphRAG and the cheaper method the same authors shipped seven months later, MCP and A2A, CLIP and BLIP, and a twelfth entry on learned sparse indexing hiding in the file as index topk. The ranking is written for someone who pays for inference, and it is worth less if you do not. Treat the model as a settled commodity with your hard problem in the space between the calls and the order flips, with ReAct and the retrieval work first and the efficiency four able to wait a month. The FlashAttention 4 numbers also come from beta tags with no stable release behind them yet. ⏱️ TIMESTAMPS: 00:00 - Opening the config file 00:21 - 512 experts, 10 awake 00:37 - The 2017 sparsity paper 01:45 - Flash attention is IO aware 03:22 - Rotary embeddings from a blog post 04:43 - Where the serving money goes 05:42 - Grouped query attention 06:04 - Paged attention and vLLM 06:31 - Latent attention cuts 93 percent 07:26 - Chain of thought, audited 08:28 - ReAct and the agent loop 09:12 - GraphRAG and its cheaper sequel 10:54 - The ranking, unhedged 12:20 - Paper twelve and a bet 🔗 The four that pay rent: Outrageously Large Neural Networks (sparsely gated mixture of experts, 2017) https://arxiv.org/abs/1701.06538 FlashAttention (IO aware exact attention) https://arxiv.org/abs/2205.14135 flash-attention (the repository you pin, not cite) https://github.com/Dao-AILab/flash-attention RoFormer (rotary position embedding) https://arxiv.org/abs/2104.09864 Jianlin Su's blog, where rotary appeared first https://kexue.fm Fast Transformer Decoding (multi query attention, 2019) https://arxiv.org/abs/1911.02150 GQA (grouped query attention) https://arxiv.org/abs/2305.13245 PagedAttention and vLLM https://arxiv.org/abs/2309.06180 vLLM (the serving engine) https://github.com/vllm-project/vllm DeepSeek-V2 (multi head latent attention) https://arxiv.org/abs/2405.04434 ⚡ The other seven, and the twelfth: • Chain-of-Thought Prompting https://arxiv.org/abs/2201.11903 • To CoT or not to CoT (the University of Texas meta-analysis) • ReAct (reason and act) https://arxiv.org/abs/2210.03629 • GraphRAG (Microsoft Research) https://arxiv.org/abs/2404.16130 • LazyGraphRAG (the cheaper follow-up from the same authors) • Model Context Protocol https://modelcontextprotocol.io • A2A Protocol (Linux Foundation) • CLIP (learning transferable visual models) https://arxiv.org/abs/2103.00020 • BLIP (caption, filter, bootstrap) https://arxiv.org/abs/2201.12086 #qwen #deepseek #flashattention #mixtureofexperts #kvcache #vllm #aipapers #llm #machinelearning 👇 SUBSCRIBE & WATCH NEXT Subscribe for a new deep-dive every week: https://www.youtube.com/@CloudAICode User Queries : what papers are in an llm config file qwen config file explained mixture of experts explained how flash attention works what is kv cache in llms grouped query attention vs multi query attention rope theta explained how does paged attention work is chain of thought still useful best ai papers to read