How AI Actually Works (It's Just Guessing the Next Word)

How does AI actually work? Every answer from a chatbot like ChatGPT, Claude or Gemini is written one token at a time, by a large language model guessing the next word. This visual explainer follows one prompt through the whole machine. It starts in 1948, when Claude Shannon built a sentence by flipping through a book and guessing word after word. It almost worked, then fell apart. Today the goal is the same, but the machine can look back at everything it has read. We follow "The first person to walk on the Moon was…" from tokens to vectors, through attention and 96 layers, to a list of probabilities and the one word that gets picked. Then we look at training, why models sound confident when they're wrong, and how far prediction can go. In this video: • What a large language model (LLM) is, and what "parameters" are • Tokens: how text is cut into pieces and turned into numbers • Vectors and word embeddings (Bengio, 2003) • Attention: queries, keys and values, explained visually (Vaswani et al., 2017) • Softmax, probabilities and temperature: how the next token is chosen • Training, fine-tuning and human feedback • Why AI hallucinates, context windows and knowledge cutoffs CHAPTERS How next-word prediction models actually generate text. See the logic behind probabilistic language models and attention. Most people assume AI models think like humans, but they are actually statistical engines. This film breaks down the mechanics of how next-word prediction models calculate the probability of the next token based on training data. We explore the original experiments by Claude Shannon and how they evolved into the architecture powering modern systems. Beyond simple probability, we examine word vectors and attention mechanisms to show how machines process context. You will learn why these systems sometimes produce nonsensical outputs and how they map language into mathematical space. Understanding these core principles is essential for anyone building with or researching large language models. Stack Zero. Complex Systems. Clear Explanations. Real Impact. Subscribe for weekly breakdowns of the technology behind modern software. 0:00 A 1948 experiment in guessing words 0:40 Today's chatbots: still guessing the next word 1:41 What a large language model is 2:02 Tokens and the prediction loop 2:41 Why Shannon's sentence fell apart 2:56 Shannon's human guessing game (1951) 3:32 Counting words: n-grams 3:57 Word vectors (2003) 4:19 Attention arrives (2014) 4:41 Transformers: "Attention Is All You Need" (2017) 5:07 Inside the model: one prompt, step by step 5:15 Step 1: Tokens 5:38 Step 2: Vectors 6:01 Step 3: Attention (query, key, value) 6:49 Attention heads 7:04 96 layers and feed-forward networks 7:33 Step 4: Probabilities and softmax 7:48 Step 5: Choosing the next token (greedy, sampling, temperature) 8:14 The loop: one token at a time 8:33 Training: Shannon's game at scale 9:16 From predictor to assistant 9:34 One loop, many products 9:45 Why AI hallucinates 10:18 Context windows, cutoffs and sources 10:37 Recap: the whole machine 10:59 How far can prediction go? Numbers on screen are illustrative unless a source is shown. SOURCES • Shannon (1948), A Mathematical Theory of Communication: https://people.csail.mit.edu/dmoshkov/courses/codes/shannon1948.pdf • Shannon (1951), Prediction and Entropy of Printed English: https://sites.socsci.uci.edu/~rfutrell/teaching/itl-davis/readings/shannon1951prediction.pdf • Bengio et al. (2003), A Neural Probabilistic Language Model: https://www.jmlr.org/papers/v3/bengio03a.html • Bahdanau, Cho & Bengio (2014), Neural Machine Translation by Jointly Learning to Align and Translate: https://arxiv.org/abs/1409.0473 • Vaswani et al. (2017), Attention Is All You Need: https://arxiv.org/abs/1706.03762 • Brown et al. (2020), Language Models are Few-Shot Learners (GPT-3): https://arxiv.org/abs/2005.14165 • Meta (2024), Introducing Meta Llama 3: https://ai.meta.com/blog/meta-llama-3/ • OpenAI Help, What are tokens and how to count them: https://help.openai.com/en/articles/4936856 • Geva et al. (2021), Transformer Feed-Forward Layers Are Key-Value Memories: https://arxiv.org/abs/2012.14913 • Ouyang et al. (2022), Training language models to follow instructions with human feedback: https://arxiv.org/abs/2203.02155 • OpenAI (2025), Why language models hallucinate: https://openai.com/index/why-language-models-hallucinate/ • Olsson et al. (2022), In-context Learning and Induction Heads: https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html Stack Zero explains the systems behind everyday technology, from the ground up. #AI #LLM #HowAIWorks