Do Sparse Autoencoders Capture Concept Manifolds? (Apr 2026)

Title: Do Sparse Autoencoders Capture Concept Manifolds? (Apr 2026) Link: http://arxiv.org/abs/2604.28119v1 Date: April 2026 Summary: This paper investigates whether Sparse Autoencoders (SAEs) can effectively capture concept manifolds in neural network representations, moving beyond the traditional Linear Representation Hypothesis. It develops a theoretical framework showing that SAEs capture manifolds either globally via subspace recovery or locally through tiling. The authors find that current SAEs often result in a fragmented 'dilution' regime and propose a new unsupervised discovery method using Ising models to identify groups of atoms that represent these manifolds. Key Topics: - Sparse Autoencoders - Concept Manifolds - Interpretability - Linear Representation Hypothesis - Ising Model - Manifold Learning - Representation Learning Chapters: 00:00 - Intro: The Map Analogy 01:00 - Explaining Linear Representation Hypothesis 02:37 - Exploring Neural Network Geometry 03:55 - Analyzing Llama 3 Activations 05:08 - Testing Steering and Bias 06:55 - Local vs Global Utility 08:08 - Comparing Capture vs Tiling 10:13 - Testing Synthetic Manifolds 11:27 - Identifying Dilution Regime Failures 13:13 - Debunking Reconstruction Error Metrics 14:15 - Borrowing Ising Physics Models 15:52 - Mapping Epistemic Uncertainty 17:37 - Building Better Interpretability Tools 19:22 - Synthesizing Alien Machine Cognition Stock video credits: - Pressmaster - https://www.pexels.com/@pressmaster - Google DeepMind - https://www.pexels.com/@googledeepmind - olia danilevich - https://www.pexels.com/@olia-danilevich - José Alfredo Munguía Lira - https://www.pexels.com/@rectorretro - Soumya - https://www.pexels.com/@soumya-1446957 - Trippy Lagoon - https://www.pexels.com/@trippy-lagoon-511515544 - Colors Motion Graphics - https://www.pexels.com/@colors-motion-graphics-183847699 - tunnel motions - https://www.pexels.com/@tunnelmotions - StefWithAnF - https://www.pexels.com/@stefwithanf-1955763 - Ron Lach - https://www.pexels.com/@ron-lach - Engin Akyurt - https://www.pexels.com/@enginakyurt - fauxels - https://www.pexels.com/@fauxels - Bedrijfsfilmspecialist.nl - https://www.pexels.com/@bedrijfsfilmspecialist-nl-1284006 - Yaroslav Shuraev - https://www.pexels.com/@yaroslav-shuraev - Nino Souza - https://www.pexels.com/@ninosouza - Tiger Lily - https://www.pexels.com/@tiger-lily - Max Fischer - https://www.pexels.com/@max-fischer - Pachon in Motion - https://www.pexels.com/@pachon-in-motion-426015731 - Silviu Din - https://www.pexels.com/@silviu-din-1620549 - Dan Cristian Pădureț - https://www.pexels.com/@paduret - Oleg Gamulinskii - https://www.pexels.com/@oleg-gamulinskii-755060 - Colin Jones - https://www.pexels.com/@larchmedia - Anthony 🙂 - https://www.pexels.com/@inspiredimages - Glenn Langhorst - https://www.pexels.com/@glenn-langhorst-120487 - Kelly - https://www.pexels.com/@kelly - Caleb Oquendo - https://www.pexels.com/@caleboquendo - crazy motions - https://www.pexels.com/@crazy-motions-80195021 - @svetjekolem - https://www.pexels.com/@svetjekolem - Pixabay - https://www.pexels.com/@pixabay - Mikhail Nilov - https://www.pexels.com/@mikhail-nilov - Claudiu Ciobanu - https://www.pexels.com/@claudiuciobanu - Tom Fisk - https://www.pexels.com/@tomfisk - Anete Lusina - https://www.pexels.com/@anete-lusina - Pavel Danilyuk - https://www.pexels.com/@pavel-danilyuk - Tima Miroshnichenko - https://www.pexels.com/@tima-miroshnichenko - Danil Shostak - https://www.pexels.com/@danil-shostak-1324124 - cottonbro studio - https://www.pexels.com/@cottonbro - Thirdman - https://www.pexels.com/@thirdman - Adis Resic - https://www.pexels.com/@adis-resic-297996969 - Charlie Mounsey - https://www.pexels.com/@charlie-mounsey-1653902