Looking beyond the next token (Apr 2025)

Title: Looking beyond the next token (Apr 2025) Link: http://arxiv.org/abs/2504.11336v1 Date: April 2025 Summary: Introduces TRELAWNEY, a data-centric method for improving language models by rearranging training data to include 'lookahead' tokens representing future information, enhancing performance on planning, reasoning, and story generation tasks without architectural changes. Key Topics: - Language Models - Next Token Prediction - Data Augmentation - Teacher Forcing - Planning - Reasoning - Story Generation - Lookahead Tokens - TRELAWNEY Chapters: 00:00 - The Problem with Next Token Prediction 00:17 - Goal-Oriented Thinking 00:43 - Introducing TRELAWNEY 01:17 - A Data-Centric Solution 01:56 - Deep Dive Overview 02:21 - Human Planning vs. NTP 02:57 - How TRELAWNEY Works 03:34 - Benefits of TRELAWNEY 03:51 - Teacher Forcing Limitations 04:38 - The Clever Hans Cheat 05:45 - The Indecipherable Token Problem 06:26 - Exposure Bias 07:11 - The Nonlinear Nature of Information Flow 07:38 - TRELAWNEY: Data Augmentation 08:03 - Key Choices for Augmentation 08:26 - Importance of Choosing the Right Chunk 08:58 - The Distance Between Decision Point and Sequence 09:33 - Positional Information 10:12 - Leveraging Preexisting Knowledge 10:41 - Training with Augmented Sequences 11:24 - Loss Function Tweak 11:47 - Masking the T Token 12:09 - Inference with TRELAWNEY 12:31 - T Generation 12:58 - Explicit Control 13:23 - Experiment Domains 13:58 - StarCraft Task 14:23 - NTP Struggles 14:44 - Excluding V1 15:11 - Results on Standard Autoregressive Generation 16:00 - Algorithmic Reasoning 16:39 - Rule Based vs Random Selection 17:16 - Natural Language Planning 17:50 - Evaluate Story Quality 18:35 - Perplexity 19:06 - Big Picture Takeaway 19:52 - Goal Orientation 20:11 - Further Thoughts