every term, defined once

Glossary

Each term is introduced in exactly one week, then linked back from later weeks.

week 01 · Orientation and a first digit classifier

Tensor: An n-dimensional array. In this course, tensors are the basic data structure flowing through every model.
Shape: The size of each tensor axis, read like a type signature for the data.
Broadcasting: PyTorch automatically expands compatible smaller tensors across missing dimensions before an operation.
einops: A small library for readable tensor rearrangement, reduction, and repetition.
einsum: Index-notation syntax for dot products, matrix multiplies, reductions, and many tensor contractions.
Weight: A learnable number inside a model, usually stored in a tensor.
Bias: A learnable offset added after a weighted sum.
ReLU: The function max(0, x). It lets stacked layers represent more than one linear operation.
Logits: Raw model scores before softmax converts them into probabilities.
Residual connection: A direct branch that adds an earlier activation to the output of other layers: out = x + f(x).
Batch normalization: A layer that normalizes each channel during training, then applies a learned scale and offset.
Softmax: A function that turns a vector of scores into a probability distribution by exponentiating and normalising.
Cross-entropy: The standard classification loss. It measures how little probability the model put on the correct label.
Convolution: An operation that applies the same small kernel at each position in an image to find local patterns.
Gradient: For each parameter, the gradient describes how a small increase would change the loss.
SGD: Stochastic gradient descent: step parameters opposite the gradient.
Learning rate: The step size an optimizer uses when updating parameters.
Adam: A popular optimizer that adapts per-parameter step sizes using running gradient statistics.
Weight decay: A regularizer that nudges weights toward zero during optimization, discouraging solutions that require large parameter values.
Chain rule: The rule that lets backprop multiply local derivatives through a composed computation.
Computational graph: A graph of tensor operations whose reverse traversal computes gradients.
Autograd: Automatic differentiation: software that builds and backpropagates through a computation graph.

week 02 · Transformers from scratch

Autoregressive: An autoregressive model generates a sequence one item at a time. Each new item depends on the items before it.
Token: A token is a text chunk represented by an integer.
Vocabulary: A vocabulary is the fixed list of tokens that a model can read or produce.
Byte pair encoding: Byte pair encoding is the method GPT-2 uses to split text into tokens. It starts from single bytes and repeatedly joins the most common neighboring pair into a new token, so frequent chunks of text become single tokens while rare text still breaks into smaller pieces.
Embedding: An embedding is a learned vector that represents a token inside the model.
Residual stream: The residual stream has one vector at each token position and carries information between transformer layers.
Attention: Attention combines information from the token positions that a position is allowed to read.
Query, key, and value: Attention computes a query, key, and value for each position. Queries and keys determine the attention weights. Values carry the information those weights combine.
Causal attention: Causal attention prevents each position from reading later positions.
Attention pattern: An attention pattern is a matrix of weights with one row per query position and one column per key position.
Positional encoding: Positional encoding records a token's position in the sequence.
MLP: An MLP applies two linear layers and a nonlinear function to each token position independently.
LayerNorm: LayerNorm normalizes each token vector, then applies a learned scale and offset.
Unembedding: Unembedding maps each final residual stream vector to one logit for every token in the vocabulary.

week 03 · Introduction to mechanistic interpretability

Mechanistic interpretability: The project of explaining model behavior by identifying the internal algorithms and components that cause it.
TransformerLens: A library that exposes transformer activations and hooks for mechanistic interpretability work.
Activation cache: A collection of named intermediate activations recorded during a model forward pass.
K-composition: A circuit in which one attention head builds its keys from information written by an earlier head.
Induction head: An attention head that implements a "copy what followed this token last time" pattern.
Induction score: The average attention weight on the expected offset stripe for destinations in the second copy of a repeated sequence.
Hook: A function called at a named activation during a forward pass to read or replace that activation.

week 04 · Probes, steering vectors, and SAEs

Feature: A property of the data or the computation that researchers may want to track. Features exist in the data distribution whether or not an SAE finds them.
Sparse autoencoder (SAE): A model trained to reconstruct activations using a sparse, wider latent representation.
SAE latent: One learned coordinate of an SAE: an encoder direction that decides when it fires, and a decoder direction it writes back. A candidate representation of one or more features, not a feature by definition.
L0: The average number of SAE latents active on a token. The standard sparsity measure for an SAE.
sae_lens: An open-source library for loading, training, and evaluating sparse autoencoders on model activations.
Neuronpedia: A public browser for inspecting model neurons and SAE features.
Feature dashboard: A report showing examples, stats, and effects for one SAE latent or neuron.
Linear probe: A simple linear classifier trained on activations to test whether information is linearly readable.
Mass-mean probe: A linear probe whose direction is the difference between the mean activation for each of two labeled classes, computed in closed form rather than fitted by optimization. It scores raw activations with no intercept.
Logistic regression: A linear classifier that fits a weight vector by optimization, then maps its score to a class probability with the logistic function. The probes fitted this week standardize activations first and fit no intercept.
PCA: Principal component analysis: a way to project high-dimensional activations onto high-variance directions.
Truth direction: A direction in activation space associated with true versus false statements.
Confound: A variable correlated with the label that can create an apparent effect without representing the property the experiment aims to measure.
Steering vector: A vector added to an activation to push model behavior in a chosen direction.
Function vector: An activation direction that appears to encode a task such as antonym generation.
Contrastive pair: Two inputs matched except for a target property; subtracting their activations estimates a direction associated with that property.
Activation addition: An intervention that adds a chosen vector to a model activation during a forward pass to steer subsequent behavior.
nnsight: A tracing and intervention library for reading and editing model internals, including remotely hosted models.

week 05 · Finding circuits in language models

Ablation: Removing or replacing a model component to test whether a behavior depends on it.
IOI: Indirect Object Identification, a benchmark sentence task used to study a GPT-2 circuit.
Activation patching: Swapping activations between model runs to test which internals causally affect behavior.
Noising vs. denoising: Complementary patching experiments: noising corrupts a clean run to test necessity, while denoising restores part of a corrupted run to test sufficiency.
Logit difference: A metric comparing model scores for the correct and incorrect answer tokens.
Direct logit attribution: Measuring how much each component’s direct residual-stream write changes the logit for a chosen token. It shows association, not cause.
Name-mover head: An attention head in the IOI circuit that attends to the correct name and copies it into the output logits.
S-inhibition head: An attention head in the IOI circuit that moves the "this name is duplicated" signal to the final position, reducing later heads' attention to the repeated subject name.
Path patching: A refinement of activation patching that isolates a specific sender-to-receiver path while holding the rest of the activation fixed.

week 06 · Understanding models trained on small tasks

Superposition: Representing more features than dimensions by packing sparse features into overlapping directions.
Sparsity: The property that most possible features are inactive for any given input.
Polysemantic neuron: A neuron that responds to multiple unrelated features.
Grokking: A delayed transition where a model moves from memorization to real generalization long after fitting the training set.
Progress measure: A quantity computed during training that tracks the gradual formation of a mechanism, even while headline loss or accuracy is flat.
Fourier basis: A way to represent periodic patterns as sums of sine and cosine waves.
FFT: Fast Fourier transform: an efficient algorithm for computing a signal's discrete Fourier transform and revealing its frequency components.
Restricted ablation: An ablation that removes everything except a hypothesized subspace or mechanism.

week 07 · Reinforcement learning foundations

Markov decision process: A model of sequential decisions built from states, actions, transition probabilities, rewards, and a discount factor.
Policy: A rule or probability distribution that chooses an action from a state.
Trajectory: One sequence of states, actions, and rewards produced as an agent interacts with an environment.
Discounted return: The sum of future rewards after multiplying later rewards by progressively smaller powers of the discount factor.
Discount factor: A number between zero and one that controls how strongly an agent values later rewards.
Value function: The expected discounted return from a state under a policy.
Q-value: The expected discounted return after taking a chosen action in a state and following a policy afterward.
Bellman equation: A recursive equation that writes a value as immediate reward plus the discounted value of what follows.
Tabular method: A reinforcement learning method that stores one value for every state or state-action pair.
Q-learning: An off-policy method that updates a Q-value toward the reward plus the best estimated value at the next state.
SARSA: An on-policy method that updates a Q-value using the next action selected by the current policy.
TD error: The difference between a current value estimate and its one-step reward-plus-next-value target.
Epsilon-greedy: A policy that usually takes the highest-value action and chooses a random action with probability epsilon.
DQN: Deep Q-Network: a neural network trained to estimate Q-values for each action.
Replay buffer: A store of past transitions sampled to train a reinforcement learning agent.
Policy gradient: A method that changes policy parameters in the direction that raises expected return.
Gymnasium: A Python library that provides standard reinforcement learning environments and interfaces.

week 10 · Running evaluations, building agents, and AI control

Inspect: A library from the UK AI Security Institute for defining and running language model evaluations.
Solver: A step in an Inspect evaluation that transforms model state, such as by prompting a model or handling a tool call.
Scorer: A function that compares a model response with a target or rubric and returns an evaluation result.
Elicitation: The effort to make a model demonstrate the capability an evaluation aims to measure.
Scaffolding: The prompts, tools, memory, and control loop wrapped around a model to help it complete a task.
Tool calling: A model requesting a structured external action, such as a search or code execution, then using the returned result.
AI control: Methods that use monitoring and restrictions to keep an AI system safe even when it may try to cause harm.

week 11 · Studying misaligned behavior

Emergent misalignment: Broadly misaligned behavior that appears after fine-tuning a model on one narrow harmful task.
Model organism: A model deliberately trained to display a behavior so researchers can study that behavior under controlled conditions.
LoRA: Low-rank adaptation: fine-tuning that freezes the original weights and trains small low-rank updates.
Autorater: A language model prompted to score other model responses against a rubric.
Compliance gap: A difference in model compliance rates between two conditions, such as monitored and unmonitored conversations.
Alignment faking: A model strategically complying during training to avoid having its existing preferences changed.