bitcoin
bitcoin

$64602.483777 USD

0.17%

ethereum
ethereum

$1828.427857 USD

1.11%

tether
tether

$0.999045 USD

0.00%

bnb
bnb

$592.614065 USD

0.38%

usd-coin
usd-coin

$0.999741 USD

-0.01%

xrp
xrp

$0.959759 USD

0.91%

solana
solana

$72.779464 USD

2.85%

tron
tron

$0.306254 USD

1.29%

hyperliquid
hyperliquid

$69.655101 USD

5.51%

zcash
zcash

$922.130514 USD

4.31%

monero
monero

$497.839010 USD

3.19%

dogecoin
dogecoin

$0.058918 USD

0.80%

unus-sed-leo
unus-sed-leo

$9.310662 USD

1.30%

chainlink
chainlink

$9.166951 USD

0.95%

cardano
cardano

$0.153248 USD

0.94%

Cryptocurrency News Video

Engram: DeepSeek's n-gram lookup memory inside an LLM

Nov 02, 2026 at 12:22 am DATUREX GmbH

DeepSeek's Engram gives a transformer a table of short token sequences that it reads by lookup instead of re-deriving by computation. The video builds it from classic n-gram counts: normalize tokens, hash n-grams into tables, gate the result into the hidden state. It then covers the reported trade-off between experts and memory, why the table can sit in CPU memory, and what is still open: the results are DeepSeek's own, from one run per model on base models, with no outside repeat at 27B so far. 00:00 A lookup instead of layers 00:26 Counting what comes next 01:02 What a transformer does instead 01:26 Step one: normalize 01:52 Step two: hash into tables 02:34 Step three: gate and add 03:09 Two kinds of sparsity 03:54 What DeepSeek reports at 27 billion 04:22 Why the table can live off the GPU 04:53 What we know, and what we don't 05:29 Recap The numbers in the video N-gram counts (Berkeley Restaurant Project corpus; Jurafsky & Martin, Fig. 3.1-3.2, Eq. 3.11): "to" follows "want" 608 of 927 times, so P(to | want) = 608 / 927 = 0.66. Shakespeare, V = 29,066: about 844 million possible bigrams, most of which never occur (SLP3 section 3.5). Layer ladder (illustration from Ghandeharioun et al. 2024, reproduced in the Engram paper, Table 3): for "Diana, Princess of Wales", early layers read a country in the UK, layer 4 the title, layer 6 the person. Not an Engram measurement. Normalization: mapping token ids that differ only in case or Unicode form to one id shrinks the effective vocabulary of DeepSeek's 128k tokenizer by 23% (Engram paper section 2.2, Appendix C). The ids shown are toy ids. Hash and gate examples use TOY numbers, not real tokenizer values (hash: two multipliers combined with exclusive-or, table sizes 11 and 13; gate: d = 4). Several hash heads make a clash in every table unlikely. The gate formula is from Engram paper section 2.3, Eq. (3)-(5). Allocation sweep (DeepSeek's own pre-training runs at two sizes, about 5.7B and 9.9B total parameters, fixed sparsity): at the 6e20 FLOPs budget, validation loss is 1.7248 with all spare parameters in experts and 1.7109 near 80% experts; the best mix gave Engram 20-25% of the spare budget; with experts cut to about 40% the model only matched the all-expert baseline. The curve between the marked points is a schematic shape, not values read from the figure (Engram paper section 3.1, Fig. 3). 27B comparison (DeepSeek's own results, pre-trained base models, one reported run per model, evaluated by the authors, one scale): MoE-27B vs Engram-27B, both 26.7B total and 3.8B active parameters, trained on 262B tokens; 72 routed experts vs 55 experts plus 5.7B parameters of Engram memory. MMLU 57.4 to 60.4, BBH 50.9 to 55.9, HumanEval 37.8 to 40.8; Engram-27B is higher on all 27 benchmarks in Table 1. The abstract says MMLU +3.4; the table, which the video uses, says +3.0. Offload: a 100B-parameter Engram table held entirely in host memory cost 2.8% throughput on an 8B dense model (6,315.52 to 6,140.02 tokens per second; 1.9% on a 4B dense model); NVIDIA H800, one workload, dense backbones only, not measured with an MoE backbone or Engram-27B itself (Engram paper section 6.4, Table 4). Outside check, much smaller scale (Memory Grafting preprint, Tsinghua University and Microsoft Research Asia, nine-benchmark average against a similar MoE): 2.8B trainable parameters, 51.95 for the MoE vs 52.43 for Engram, +0.48; 0.92B, 45.62 vs 45.03, -0.59, and that drop comes from one test (BoolQ). Different data, benchmarks and scale than DeepSeek's; it does not test DeepSeek's numbers. We found no outside repeat of the 27B result as of 2026-10-31. DeepSeek's own later use: the DeepSeek-V4.1-Flash model card lists Engram conditional memory of 196B parameters. That is DeepSeek's choice, not outside evidence. Each result holds for the models and settings named in its source. Sources Cheng et al. (Peking University, DeepSeek-AI): Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models (2026): https://arxiv.org/abs/2601.07372 deepseek-ai/Engram, official repository: https://github.com/deepseek-ai/Engram Jurafsky & Martin: Speech and Language Processing, 3rd ed. draft, Chapter 3 "N-gram Language Models": https://web.stanford.edu/~jurafsky/slp3/3.pdf Svenstrup, Hansen, Winther: Hash Embeddings for Efficient Word Representations (2017): https://arxiv.org/abs/1709.03933 Shazeer et al.: Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017): https://arxiv.org/abs/1701.06538 Cheng et al. (Tsinghua University, Microsoft Research Asia): Memory Grafting (preprint, 2026): https://arxiv.org/abs/2605.20948 DeepSeek-V4.1-Flash model card: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash Part of the series LLM internals. Deutsche Fassung: ‹link after upload› Script, narration and music are AI-generated and were checked against the cited sources. Music: Stable Audio 3 – Powered by Stability AI.
Video source:Youtube

Disclaimer:info@kdj.com

The information provided is not trading advice. kdj.com does not assume any responsibility for any investments made based on the information provided in this article. Cryptocurrencies are highly volatile and it is highly recommended that you invest with caution after thorough research!

If you believe that the content used on this website infringes your copyright, please contact us immediately (info@kdj.com) and we will delete it promptly.

Other videos published on Nov 18, 2026