Superposition & toy models

The puzzle: too many concepts, too few neurons

Suppose a network layer has exactly 2 neurons. Its internal state at that layer contains 2 numbers. How many distinct properties of the input can it track?

Two orthogonal directions fit in a 2D space, so the layer can store 2 features without overlap. A third feature must share space with the first two and create interference.

Real networks represent far more features than they have dimensions. A layer with a few hundred neurons can track thousands of distinct properties. This week you train a minimal example: 5 features squeezed into 2 dimensions. You generate the data, so every true feature stays visible while the model learns to pack them. This packing scheme is Superposition , the problem that sparse autoencoders in week 8 try to undo.

What is a feature?

A feature is a human-meaningful property represented in an activation: "this token is inside a Python comment", "the sentence is in French", or "the next word should be a name". Interpretability research has also found features for hedging language, characters inside quotations, and years in the 1900s. A feature names a property the model tracks. A neuron names one concrete slot in an activation vector. Mechanistic interpretability asks how those abstract properties map onto concrete neurons and weights. Superposition makes that mapping indirect.

Features are sparse when only a few activate for any one input. This sentence may activate features for English prose, technical writing, and the word "feature" while thousands of unrelated properties stay silent. That Sparsity gives superposition room to work.

Five independently active features would exceed the clean capacity of a two-dimensional hidden space. When only one or two usually activate, the model can overlap their directions and use ReLU to clip small accidental leakage.

We give the toy model 5 abstract features. Each feature turns on independently with a chosen probability and receives a random strength when active. A tiny network compresses and reconstructs those values. Since we manufactured the ground truth, we can compare every learned direction with the true feature that produced it.

The trick: overlap a little, lean on sparsity

The model represents each feature as a direction in its hidden space, a vector pointing out from the origin at an angle the model chooses. Two directions at 90° are orthogonal, so their dot product is zero and they create no interference. Five directions exceed the number of mutually orthogonal directions in 2D. The model can spread them evenly into a pentagon, 72° apart. When one feature fires, each non-orthogonal readout picks up a phantom signal.

One matrix product measures the leakage. If the rows of W are feature directions, then W @ W.T contains every pairwise overlap. The diagonal compares each feature with itself and equals 1.0. Every off-diagonal entry measures interference between two features:

# no-run
# 01: interference is a matrix of dot products
import torch
import torch.nn.functional as F

# rows are three feature directions in a two-dimensional hidden space
W = F.normalize(torch.tensor([[1.0, 0.0], [0.3, 0.95], [-0.8, 0.6]]), dim=1)
leakage = W @ W.T
print(leakage.round(decimals=2))
# tensor([[ 1.0000,  0.3000, -0.8000],
#         [ 0.3000,  1.0000,  0.3300],
#         [-0.8000,  0.3300,  1.0000]])
# off-diagonal values show features interfering with each other

The off-diagonal entries are interference. Superposition is the model deciding that small interference is cheaper than dedicating full dimensions to rare features.

The pentagon shows how the model controls that interference. When only feature 0 fires, decoding the compressed 2D state produces five readouts with small positive and negative phantoms. A ReLU (max(0, x)) erases every negative value:

# 02: the pentagon's leakage and the ReLU rescue
import numpy as np

angles = np.deg2rad(90 + 72 * np.arange(5))     # five directions, 72° apart
W = np.stack([np.cos(angles), np.sin(angles)])  # (2, 5): columns are features

x = np.array([1.0, 0, 0, 0, 0])   # only feature 0 fires
readoff = W.T @ (W @ x)           # squeeze to 2 dims, then read all 5 back
print(readoff.round(2))
# [ 1.    0.31 -0.81 -0.81  0.31]
print(np.maximum(readoff, 0).round(2))
# [1.   0.31 0.   0.   0.31]

Feature 0 survives at full strength. ReLU clips two phantoms to zero, while two positive phantoms of 0.31 remain. During training, the model learns negative biases that push this small positive leakage below zero before ReLU. Sparsity handles co-occurrences: strong interference requires two overlapping features to fire simultaneously, an infrequent event in sparse data. Paying a small reconstruction cost on those cases costs less than dropping 3 of the 5 features. Superposition accepts bounded interference so the model can represent every feature.

The toy model: 5 → 2 → 5

A tiny autoencoder demonstrates the full mechanism. An autoencoder learns to reproduce its input. Here, a bottleneck squeezes 5 input features through a hidden layer of 2 numbers before reconstructing all 5. The encoder and decoder share a weight matrix W with shape (2, 5); the decoder uses its transpose, a design called tied weights. A bias and final ReLU complete the model. Each column of W gives one feature's direction in the 2D hidden plane, so an arrow plot of those columns exposes the learned geometry directly.

The synthetic data exposes two controls. Each feature activates independently with probability 1 − S, where S is sparsity. Active features take a random value, and inactive features equal 0. At S = 0, every feature is dense and always active. At S = 0.99, each feature activates on about 1% of examples. Each feature also receives an importance weight, a geometrically decaying multiplier on its contribution to the loss. When capacity runs short, the loss tells the model which reconstruction errors cost most.

The main experiment trains fresh model copies across a sweep from dense to sparse and plots the columns of W. With dense inputs, the model drops three features and stores the two most important features orthogonally. With sparse inputs, all five arrows appear in an evenly spaced pentagon. Intermediate settings produce other geometries, including antipodal feature pairs. Sparsity controls when shared dimensions become economical.

Why neurons become polysemantic

Superposition makes neuron-by-neuron reading unreliable. If features occupy pentagon directions, one neuron, or axis of the hidden space, contains projections from several features. A Polysemantic neuron therefore responds to multiple unrelated features. The neuron basis and the feature basis point in different directions.

When that neuron lights up, its scalar value leaves the bridge feature, the code feature, and a mixture of both indistinguishable. Week 6 found attention heads with clean roles. Many model components lack that alignment. A better microscope changes coordinates and searches for directions that correspond to single features. Week 8's sparse autoencoders automate that search.

What ARENA adds beyond the headline plot

The notebook goes well past the pentagon, and its extra sections all circle one question: when is packing features together worth it?

• Importance and sparsity sweeps: Training model grids across both controls produces a phase diagram with regions where a feature owns a full dimension, shares one, or gets dropped. Superposition switches on and off at sharp boundaries.
• Privileged and non-privileged bases (section 2): A purely linear layer has no preferred neuron axes. You could rotate the hidden space without changing the model, which makes "what does neuron 3 mean?" a question about an arbitrary coordinate. An elementwise ReLU breaks this rotational symmetry because it acts on each neuron separately. That privileged basis lets individual neurons become candidates for stable meanings.
• Feature geometry (section 3): Different feature counts and sparsities produce digons, triangles, pentagons, tetrahedra, and other uniform polytopes. Each feature receives a fraction of a dimension: 1 in an orthogonal arrangement, 1/2 in an antipodal pair, and 2/5 at a pentagon vertex. Models jump between these rational values in discrete steps.
• Double descent (section 4, stretch): The same toy setup can make test loss rise and then fall as model size grows. This section links superposition with the tradeoff between memorization and generalization. Save it for a second pass if time runs short.

First contact with SAEs

The notebook's final section introduces a tool for unmixing features hidden in overlapping directions. A Sparse autoencoder (SAE) learns a wider latent representation and penalizes simultaneous use of many latents.

The SAE maps the toy model's 2 compressed dimensions into a wider layer. Its sparsity penalty rewards explanations that use few slots, giving each slot pressure to specialize in one original feature.

# no-run
# 03: the conceptual core of an SAE
import torch
import torch.nn.functional as F

x = torch.randn(16, 2)
W_enc = torch.randn(2, 8)
W_dec = torch.randn(8, 2)
acts = F.relu(x @ W_enc)
recon = acts @ W_dec
loss = ((recon - x) ** 2).mean() + 0.01 * acts.abs().mean()
print(loss.item())
# a positive scalar; the exact value varies because the tensors are random

The reconstruction term rewards preservation of the original activation. The sparsity term rewards explanations with few active latents. Their balance makes dedicated slots economical: each true feature can activate one latent without recruiting a dense mixture. In the notebook, the SAE decoder directions align with the five known pentagon directions. Week 8 applies the same method to real LLMs, whose true feature dictionaries remain unknown.

Why this matters

Week 7 changes the unit of analysis from neurons to directions. Gradient descent can pack sparse features into overlapping directions because that geometry reconstructs more useful information with limited width. SAEs in week 8 try to unpack those directions. Probes in week 10 search for a named direction without trusting individual neurons. Steering in week 11 writes along feature directions. The small toy model gives each later method a concrete mechanical purpose.

Pair-session guide

Core work is ARENA 1.5.4 sections 1 and 2 plus section 5 (the toy SAE section): train toy models, see how a ReLU makes the basis privileged, and watch an SAE recover known features. ARENA's own intro agrees: if time is short, it recommends sections 1 and 5 with a brief look at 2. Stretch work is sections 3 and 4, feature geometry and double descent. Keep asking which dimension is batch, which is feature, and which is parallel model instance.

What you should see

You should see feature directions spread into a pentagon-like plot as sparsity increases. In the SAE section, decoder directions should align with the true feature directions. Explain the result through feature geometry: several concept directions can cross each neuron axis.

Where to go next

The notebook's own reading list is unusually good this week; these are the three ARENA recommends:

• Anthropic's Toy Models of Superposition is the paper this week reimplements. Read up to and including "Summary: A Hierarchy of Feature Properties"; the "Key Results" and "Definitions and Motivation" sections matter most.
• Exploring Polysemanticity and Superposition, from Neel Nanda's "200 Concrete Open Problems" series, is a 15-minute plain-language tour of the same ideas plus the open questions they raise.
• Neel Nanda's glossary notes on superposition are short enough to skim now and keep open as a reference during the exercises.