Neural Interpretability In Practice

Work through every problem on paper before expanding the step-by-step solutions.

Part 1: Abstract Mechanics

Work through these 5 drill problems testing latent transformations, linear separability, feature ablation, Transformer MLP dimensions, and token projection math.

Problem 1: Latent Space Coordinate Transformation Mapping

A 2-input, 2-neuron hidden layer has the following configuration:

Input vector: $x = \begin{bmatrix} 2.0 \\ -1.0 \end{bmatrix}$
Weight matrix: $W^{(1)} = \begin{bmatrix} 1.5 & -1.0 \\ -0.5 & 2.0 \end{bmatrix}$
Bias vector: $b^{(1)} = \begin{bmatrix} -1.0 \\ 1.0 \end{bmatrix}$
Activation function: $\text{ReLU}(z) = \max(0, z)$

Compute:

The pre-activation vector $z^{(1)} = W^{(1)} x + b^{(1)}$ .
The hidden latent activation vector $a^{(1)} = \text{ReLU}(z^{(1)})$ .

Reveal Solution

Pre-Activation Vector $z^{(1)}$ :
$z_1^{(1)} = (1.5)(2.0) + (-1.0)(-1.0) - 1.0 = 3.0 + 1.0 - 1.0 = \mathbf{+3.0}$ $z_2^{(1)} = (-0.5)(2.0) + (2.0)(-1.0) + 1.0 = -1.0 - 2.0 + 1.0 = \mathbf{-2.0}$ $z^{(1)} = \begin{bmatrix} +3.0 \\ -2.0 \end{bmatrix}$
Hidden Activation Vector $a^{(1)}$ :
$a^{(1)} = \text{ReLU}\left(\begin{bmatrix} +3.0 \\ -2.0 \end{bmatrix}\right) = \begin{bmatrix} \max(0, 3.0) \\ \max(0, -2.0) \end{bmatrix} = \begin{bmatrix} \mathbf{3.0} \\ \mathbf{0.0} \end{bmatrix}$

Problem 2: Linear Separability Classification Check

Consider 4 points in a 2D latent representation space $\mathbb{R}^2$ :

Class A ( $y = +1$ ): $p_1 = \begin{bmatrix} 3.0 \\ 1.0 \end{bmatrix}, \quad p_2 = \begin{bmatrix} 2.0 \\ 3.0 \end{bmatrix}$
Class B ( $y = -1$ ): $p_3 = \begin{bmatrix} 0.0 \\ 1.0 \end{bmatrix}, \quad p_4 = \begin{bmatrix} 1.0 \\ 0.0 \end{bmatrix}$

A candidate linear separating hyperplane has parameter vector $w_{\text{sep}} = \begin{bmatrix} 1.0 \\ 1.0 \end{bmatrix}$ and threshold bias $b_{\text{sep}} = -3.0$ , defining the decision function:

s(p) = w_{\text{sep}} \cdot p + b_{\text{sep}}

Where a point is classified as Class A if $s(p) > 0$ , and Class B if $s(p) < 0$ .

Compute the decision score $s(p_i)$ for each of the 4 points and verify whether this hyperplane cleanly separates Class A from Class B.

Reveal Solution

Score for Point $p_1 = [3.0, 1.0]^T$ (Class A):
$s(p_1) = (1.0)(3.0) + (1.0)(1.0) - 3.0 = 3.0 + 1.0 - 3.0 = \mathbf{+1.0} > 0 \implies \mathbf{\text{Class A (Correct)}}$
Score for Point $p_2 = [2.0, 3.0]^T$ (Class A):
$s(p_2) = (1.0)(2.0) + (1.0)(3.0) - 3.0 = 2.0 + 3.0 - 3.0 = \mathbf{+2.0} > 0 \implies \mathbf{\text{Class A (Correct)}}$
Score for Point $p_3 = [0.0, 1.0]^T$ (Class B):
$s(p_3) = (1.0)(0.0) + (1.0)(1.0) - 3.0 = 0.0 + 1.0 - 3.0 = \mathbf{-2.0} < 0 \implies \mathbf{\text{Class B (Correct)}}$
Score for Point $p_4 = [1.0, 0.0]^T$ (Class B):
$s(p_4) = (1.0)(1.0) + (1.0)(0.0) - 3.0 = 1.0 + 0.0 - 3.0 = \mathbf{-2.0} < 0 \implies \mathbf{\text{Class B (Correct)}}$

Conclusion: Since $s(p_1) > 0, s(p_2) > 0$ and $s(p_3) < 0, s(p_4) < 0$ , the 4 points are strictly linearly separable by the hyperplane $w_{\text{sep}} \cdot p + b_{\text{sep}} = 0$ .

Problem 3: Hidden Neuron Feature Ablation Calculation

A 2-layer network with 2 hidden neurons and a linear output neuron has parameters:

Hidden activation vector: $a^{(1)} = \begin{bmatrix} 2.0 \\ 4.0 \end{bmatrix}$
Output weight matrix: $W^{(2)} = \begin{bmatrix} 1.5 & -0.5 \end{bmatrix}$
Output bias: $b^{(2)} = 0.20$
Output prediction: $\hat{y} = W^{(2)} a^{(1)} + b^{(2)}$

Perform an ablation experiment:

Compute the intact output prediction $\hat{y}_{\text{intact}}$ .
Ablate hidden neuron 2 (set $a_2^{(1)} = 0.0$ ) and compute the ablated output prediction $\hat{y}_{\text{ablated}}$ .
Compute the exact causal output shift $\Delta \hat{y} = \hat{y}_{\text{ablated}} - \hat{y}_{\text{intact}}$ .

Reveal Solution

Intact Output Prediction:
$\hat{y}_{\text{intact}} = W^{(2)} a^{(1)} + b^{(2)} = (1.5)(2.0) + (-0.5)(4.0) + 0.20 = 3.0 - 2.0 + 0.20 = \mathbf{1.20}$
Ablated Output Prediction ( $a_2^{(1)} = 0.0$ ):
$a^{(1)}_{\text{ablated}} = \begin{bmatrix} 2.0 \\ 0.0 \end{bmatrix}$ $\hat{y}_{\text{ablated}} = (1.5)(2.0) + (-0.5)(0.0) + 0.20 = 3.0 + 0.0 + 0.20 = \mathbf{3.20}$
Causal Output Shift $\Delta \hat{y}$ :
$\Delta \hat{y} = \hat{y}_{\text{ablated}} - \hat{y}_{\text{intact}} = 3.20 - 1.20 = \mathbf{+2.00}$

(Interpretation: Hidden neuron 2 was exerting a strong inhibitory effect of $-0.5 \times 4.0 = -2.00$ on the output. Ablating it removes that inhibition, increasing the final prediction by $+2.00$ ).

Problem 4: Transformer MLP Sublayer Dimension Trace

A miniaturized Transformer model has token embedding dimension $d_{\text{model}} = 4$ . Its Feed-Forward MLP sublayer expands the token vector by a factor of 4 to intermediate dimension $d_{\text{ff}} = 4 d_{\text{model}} = 16$ , then projects it back to $d_{\text{model}} = 4$ .

The sublayer equations are:

z_{\text{ff}} = W_{\text{up}} x + b_{\text{up}}, \qquad h_{\text{ff}} = \text{ReLU}(z_{\text{ff}}), \qquad y_{\text{mlp}} = W_{\text{down}} h_{\text{ff}} + b_{\text{down}}

For an input token vector $x \in \mathbb{R}^{4 \times 1}$ :

State the exact matrix dimensions of $W_{\text{up}}, b_{\text{up}}, W_{\text{down}}$ , and $b_{\text{down}}$ .
State the vector dimensions of $z_{\text{ff}}, h_{\text{ff}}$ , and $y_{\text{mlp}}$ .
Calculate the total number of trainable parameters (weights + biases) in this MLP sublayer.

Reveal Solution

Parameter Dimensions ( $d_{\text{out}} \times d_{\text{in}}$ ):
- $W_{\text{up}} \in \mathbf{\mathbb{R}^{16 \times 4}}$
- $b_{\text{up}} \in \mathbf{\mathbb{R}^{16 \times 1}}$
- $W_{\text{down}} \in \mathbf{\mathbb{R}^{4 \times 16}}$
- $b_{\text{down}} \in \mathbf{\mathbb{R}^{4 \times 1}}$
Activation Vector Dimensions:
- $z_{\text{ff}} \in \mathbf{\mathbb{R}^{16 \times 1}}$
- $h_{\text{ff}} \in \mathbf{\mathbb{R}^{16 \times 1}}$
- $y_{\text{mlp}} \in \mathbf{\mathbb{R}^{4 \times 1}}$
Total Trainable Parameter Count:
- Weights in $W_{\text{up}}$ : $16 \times 4 = 64$
- Biases in $b_{\text{up}}$ : $16$
- Weights in $W_{\text{down}}$ : $4 \times 16 = 64$
- Biases in $b_{\text{down}}$ : $4$
- Total Parameters: $64 + 16 + 64 + 4 = \mathbf{148 \text{ parameters}}$ (or $128$ weights without biases).

Problem 5: Token Embedding Linear Projection

A language model has a vocabulary of 3 words: ["invest", "market", "risk"] ( $|V| = 3$ ). The model represents each word in a 2D embedding space ( $d_{\text{model}} = 2$ ) using embedding matrix:

W_E = \begin{bmatrix} 0.80 & -0.40 & 0.10 \\ 0.20 & 0.90 & -0.70 \end{bmatrix} \in \mathbb{R}^{2 \times 3}

The word "market" corresponds to vocabulary index 2 (the second column) and is represented as one-hot vector:

x_{\text{onehot}} = \begin{bmatrix} 0.0 \\ 1.0 \\ 0.0 \end{bmatrix} \in \mathbb{R}^{3 \times 1}

Compute the projected embedding coordinate vector $e = W_E x_{\text{onehot}}$ .
Verify that matrix multiplication by a one-hot vector is algebraically identical to selecting the corresponding column vector of $W_E$ .

Reveal Solution

Matrix Multiplication Calculation:
$e = W_E x_{\text{onehot}} = \begin{bmatrix} 0.80 & -0.40 & 0.10 \\ 0.20 & 0.90 & -0.70 \end{bmatrix} \begin{bmatrix} 0.0 \\ 1.0 \\ 0.0 \end{bmatrix}$ $e_1 = (0.80)(0.0) + (-0.40)(1.0) + (0.10)(0.0) = \mathbf{-0.40}$ $e_2 = (0.20)(0.0) + (0.90)(1.0) + (-0.70)(0.0) = \mathbf{+0.90}$ $e = \begin{bmatrix} \mathbf{-0.40} \\ \mathbf{+0.90} \end{bmatrix}$
Column Extraction Identity: Column 2 of $W_E$ is $\begin{bmatrix} -0.40 \\ 0.90 \end{bmatrix}$ . Multiplying $W_E$ by the one-hot basis vector $e_2 = [0, 1, 0]^T$ acts as a linear selector that extracts column 2 exactly:
$W_E x_{\text{onehot}} = W_{E, *2} = \begin{bmatrix} -0.40 \\ 0.90 \end{bmatrix}$

Part 2: Applied Scenario: The VC Decision Circuit & Latent Representation Scenario

A venture capital firm uses a miniaturized Transformer MLP sublayer to process early-stage startup decision embeddings.

[ Startup Feature Vector x ]          [ Expanded Hidden Circuit h_ff ]          [ Updated Decision Embedding y_mlp ]
  (d_model = 2)                        (d_ff = 4 Latent Features)                 (d_model = 2)
  x_1: Team Traction                   h_1: Team Scalability                     y_1: Investment Conviction
  x_2: Market Expansion                h_2: High-Burn Risk                       y_2: Follow-On Likelihood
                                       h_3: Capital Efficiency
                                       h_4: Pure Market Velocity

System Configuration:

Input Decision Vector ( $x \in \mathbb{R}^{2 \times 1}$ ):
$x = \begin{bmatrix} 2.0 \\ 1.0 \end{bmatrix} \begin{matrix} \text{(Team Traction Measurement)} \\ \text{(Market Expansion Measurement)} \end{matrix}$
Up-Projection Parameters ( $W_{\text{up}} \in \mathbb{R}^{4 \times 2}, b_{\text{up}} \in \mathbb{R}^{4 \times 1}$ ):
$W_{\text{up}} = \begin{bmatrix} 1.0 & 0.5 \\ -0.5 & 1.5 \\ 0.5 & -1.0 \\ 0.0 & 2.0 \end{bmatrix}, \qquad b_{\text{up}} = \begin{bmatrix} -0.5 \\ -1.0 \\ 0.5 \\ -1.5 \end{bmatrix}$
Activation Function: $\text{ReLU}(z) = \max(0, z)$ .
Down-Projection Parameters ( $W_{\text{down}} \in \mathbb{R}^{2 \times 4}, b_{\text{down}} \in \mathbb{R}^{2 \times 1}$ ):
$W_{\text{down}} = \begin{bmatrix} 0.5 & 1.0 & -0.5 & 0.5 \\ 1.0 & -0.5 & 0.5 & 0.0 \end{bmatrix}, \qquad b_{\text{down}} = \begin{bmatrix} 0.0 \\ 0.5 \end{bmatrix}$

Problem 6: The VC Decision Circuit & Latent Representation Scenario

Execute the step-by-step forward trace and mechanistic ablation on paper:

Part A (Up-Projection & Pre-Activations): Compute the expanded pre-activation vector $z_{\text{ff}} = W_{\text{up}} x + b_{\text{up}} \in \mathbb{R}^{4 \times 1}$ .
Part B (Non-Linear Latent Activations): Compute intermediate activation vector $h_{\text{ff}} = \text{ReLU}(z_{\text{ff}})$ .
Part C (Down-Projection to Updated Embedding): Compute final updated embedding vector $y_{\text{mlp}} = W_{\text{down}} h_{\text{ff}} + b_{\text{down}} \in \mathbb{R}^{2 \times 1}$ .
Part D (Circuit Ablation Experiment): Ablate latent neuron 4 ( $h_{\text{ff}, 4} = 0.0$ ), compute ablated output $y_{\text{mlp},\text{ablated}}$ , and calculate the exact causal shift vector $\Delta y_{\text{mlp}} = y_{\text{mlp},\text{ablated}} - y_{\text{mlp}}$ .

Reveal Solution

Part A: Up-Projection & Pre-Activations ( $z_{\text{ff}}$ )

z_{\text{ff}} = W_{\text{up}} x + b_{\text{up}} = \begin{bmatrix} 1.0 & 0.5 \\ -0.5 & 1.5 \\ 0.5 & -1.0 \\ 0.0 & 2.0 \end{bmatrix} \begin{bmatrix} 2.0 \\ 1.0 \end{bmatrix} + \begin{bmatrix} -0.5 \\ -1.0 \\ 0.5 \\ -1.5 \end{bmatrix}

Compute each coordinate:

$z_{\text{ff}, 1} = (1.0)(2.0) + (0.5)(1.0) - 0.5 = 2.0 + 0.5 - 0.5 = \mathbf{+2.00}$
$z_{\text{ff}, 2} = (-0.5)(2.0) + (1.5)(1.0) - 1.0 = -1.0 + 1.5 - 1.0 = \mathbf{-0.50}$
$z_{\text{ff}, 3} = (0.5)(2.0) + (-1.0)(1.0) + 0.5 = 1.0 - 1.0 + 0.5 = \mathbf{+0.50}$
$z_{\text{ff}, 4} = (0.0)(2.0) + (2.0)(1.0) - 1.5 = 0.0 + 2.0 - 1.5 = \mathbf{+0.50}$

z_{\text{ff}} = \begin{bmatrix} +2.00 \\ -0.50 \\ +0.50 \\ +0.50 \end{bmatrix}

Part B: Non-Linear Latent Activations ( $h_{\text{ff}}$ )

Apply $\text{ReLU}(z) = \max(0, z)$ element-wise:

h_{\text{ff}} = \begin{bmatrix} \max(0, 2.00) \\ \max(0, -0.50) \\ \max(0, 0.50) \\ \max(0, 0.50) \end{bmatrix} = \begin{bmatrix} \mathbf{2.00} \\ \mathbf{0.00} \\ \mathbf{0.50} \\ \mathbf{0.50} \end{bmatrix}

(Note: Neuron 2 receives negative pre-activation $-0.50$ , so ReLU gates it off to $0.00$ ).

Part C: Down-Projection to Updated Embedding ( $y_{\text{mlp}}$ )

y_{\text{mlp}} = W_{\text{down}} h_{\text{ff}} + b_{\text{down}} = \begin{bmatrix} 0.5 & 1.0 & -0.5 & 0.5 \\ 1.0 & -0.5 & 0.5 & 0.0 \end{bmatrix} \begin{bmatrix} 2.00 \\ 0.00 \\ 0.50 \\ 0.50 \end{bmatrix} + \begin{bmatrix} 0.00 \\ 0.50 \end{bmatrix}

Compute each output dimension:

Dimension 1 (Investment Conviction):
$y_{\text{mlp}, 1} = (0.5)(2.00) + (1.0)(0.00) + (-0.5)(0.50) + (0.5)(0.50) + 0.00$ $y_{\text{mlp}, 1} = 1.00 + 0.00 - 0.25 + 0.25 + 0.00 = \mathbf{1.00}$
Dimension 2 (Follow-On Likelihood):
$y_{\text{mlp}, 2} = (1.0)(2.00) + (-0.5)(0.00) + (0.5)(0.50) + (0.0)(0.50) + 0.50$ $y_{\text{mlp}, 2} = 2.00 + 0.00 + 0.25 + 0.00 + 0.50 = \mathbf{2.75}$

y_{\text{mlp}} = \begin{bmatrix} \mathbf{1.00} \\ \mathbf{2.75} \end{bmatrix}

Part D: Circuit Ablation Experiment

We ablate latent neuron 4 (the dedicated Market Expansion detector where $W_{\text{up}, 4} = [0.0, 2.0]$ ) by forcing $h_{\text{ff}, 4} = 0.00$ :

h_{\text{ff},\text{ablated}} = \begin{bmatrix} 2.00 \\ 0.00 \\ 0.50 \\ \mathbf{0.00} \end{bmatrix}

Recompute down-projection:

Dimension 1 (Ablated):
$y_{\text{mlp},\text{ablated}, 1} = (0.5)(2.00) + (1.0)(0.00) + (-0.5)(0.50) + (0.5)(\mathbf{0.00}) + 0.00$ $y_{\text{mlp},\text{ablated}, 1} = 1.00 + 0.00 - 0.25 + 0.00 + 0.00 = \mathbf{0.75}$
Dimension 2 (Ablated):
$y_{\text{mlp},\text{ablated}, 2} = (1.0)(2.00) + (-0.5)(0.00) + (0.5)(0.50) + (0.0)(\mathbf{0.00}) + 0.50$ $y_{\text{mlp},\text{ablated}, 2} = 2.00 + 0.00 + 0.25 + 0.00 + 0.50 = \mathbf{2.75}$

y_{\text{mlp},\text{ablated}} = \begin{bmatrix} \mathbf{0.75} \\ \mathbf{2.75} \end{bmatrix}

Causal Shift Vector $\Delta y_{\text{mlp}}$ :

\Delta y_{\text{mlp}} = y_{\text{mlp},\text{ablated}} - y_{\text{mlp}} = \begin{bmatrix} 0.75 - 1.00 \\ 2.75 - 2.75 \end{bmatrix} = \begin{bmatrix} \mathbf{-0.25} \\ \mathbf{0.00} \end{bmatrix}

Mechanistic Finding: Neuron 4 is a specialized sub-circuit that contributes exactly $+0.25$ to Investment Conviction ( $y_1$ ) through weight $W_{\text{down}, 14} = 0.5$ , while having zero direct causal connection ( $W_{\text{down}, 24} = 0.0$ ) to Follow-On Likelihood ( $y_2$ ).

Neural Interpretability In Practice

Part 1: Abstract Mechanics

Problem 1: Latent Space Coordinate Transformation Mapping

Problem 2: Linear Separability Classification Check

Problem 3: Hidden Neuron Feature Ablation Calculation

Problem 4: Transformer MLP Sublayer Dimension Trace

Problem 5: Token Embedding Linear Projection

Part 2: Applied Scenario: The VC Decision Circuit & Latent Representation Scenario

System Configuration:

Problem 6: The VC Decision Circuit & Latent Representation Scenario

Part A: Up-Projection & Pre-Activations (zffz_{\text{ff}}zff​)

Part B: Non-Linear Latent Activations (hffh_{\text{ff}}hff​)

Part C: Down-Projection to Updated Embedding (ymlpy_{\text{mlp}}ymlp​)

Part D: Circuit Ablation Experiment

Part A: Up-Projection & Pre-Activations ( $z_{\text{ff}}$ )

Part B: Non-Linear Latent Activations ( $h_{\text{ff}}$ )

Part C: Down-Projection to Updated Embedding ( $y_{\text{mlp}}$ )