Blick über das nächste Token (APR 2025) überblicken

Title: Looking beyond the next token (Apr 2025) Link: http://arxiv.org/abs/2504.11336v1 Date: April 2025 Summary: Introduces TRELAWNEY, a data-centric method for improving language models by rearranging training data to include 'lookahead' tokens representing future information, enhancing performance on planning, reasoning, and story generation tasks without architectural changes. Key Topics: - Language Models - Next Token Prediction - Data Augmentation - Teacher Forcing - Planning - Reasoning - Story Generation - Lookahead Tokens - TRELAWNEY Chapters: 00:00 - The Problem with Next Token Prediction 00:17 - Goal-Oriented Thinking 00:43 - Introducing TRELAWNEY 01:17 - A Data-Centric Solution 01:56 - Deep Dive Overview 02:21 - Human Planning vs. ntp 02:57 - Wie Trelawney funktioniert 03:34 - Vorteile von Trelawney 03:51 - Lehrer erzwingen Einschränkungen 04:38 - Der clevere Hans -Cheat 05:45 - Der unanständige Token -Problem 06:26 - Expositionsverzerrung 07:11 - der nichtlineare Naturflow 07:38 - TRELAWNY: Daten Augmentation 07 - 07: 07: 38 - TRELAWNY: Daten Augmentation 03: 03: 03: 03: 07: 07: 07:38 - TRELAWNEY: Daten Augmentation 03: 03: 03: 07: 07: 07:38 - TRELAWNEY: Daten Augmentation 03 - 03: 03. Augmentation 08:26 - Importance of Choosing the Right Chunk 08:58 - The Distance Between Decision Point and Sequence 09:33 - Positional Information 10:12 - Leveraging Preexisting Knowledge 10:41 - Training with Augmented Sequences 11:24 - Loss Function Tweak 11:47 - Masking the T Token 12:09 - Inference with TRELAWNEY 12:31 - T Generation 12:58 - Explicit Control 13:23 - Experiment -Domänen 13:58 - StarCraft -Aufgabe 14:23 - NTP -Kämpfe 14:44 - ohne V1 15:11 - Ergebnisse zur autoregressiven Generation von Standards 16:00 - Algorithmisches Denken 16:39 - Regelbasierte gegen zufällige Auswahl 17:16 - natürliche Sprachplanung 17:50 - Evaluieren Sie die Story -Qualität - Perlexität 195 - Perlexität 19:06 - Perlexität 19:06 - Perlexität 19:06 - Perlexität 19:5. Orientierung 20:11 - Weitere Gedanken