Hidden Layer Error Backpropagation

We now have the output layer error delta $\delta^{(2)}$. But how do we determine how much each hidden weight $W^{(1)}_{ij}$ contributed to the error?

---

The Hidden Layer Credit Assignment Problem

Hidden neurons have no external ground-truth labels. The world does not tell the network what the Summer Popcorn Flick neuron ($h_1$) or the Rom-Com neuron ($h_2$) should have produced.

To solve this credit assignment problem, we propagate the downstream error delta $\delta^{(2)}$ backward through the network's weights.

[ Forward Pass ]:  Inputs x ──► Hidden Layer (z^(1), a^(1)) ──► Output Layer (z^(2), a^(2)) ──► Loss L
                                                                                                  │
[ Backward Pass ]: ∂L/∂W^(1) ◄── Hidden Delta δ^(1) ◄── (W^(2))^T ◄── Output Delta δ^(2) ◄────────┘

---

Backpropagating Error Through Transposed Weights

To evaluate how sensitive the final loss is to a hidden activation $a_j^{(1)}$, apply the multivariable chain rule across all output pre-activations $z_k^{(2)}$ that depend on $a_j^{(1)}$:

$$\frac{\partial L}{\partial a_j^{(1)}} = \sum_{k=1}^{d_2} \frac{\partial L}{\partial z_k^{(2)}} \frac{\partial z_k^{(2)}}{\partial a_j^{(1)}} = \sum_{k=1}^{d_2} \delta_k^{(2)} W^{(2)}_{kj}$$

In matrix-vector notation, this sum is computed by multiplying the downstream delta vector by the transposed weight matrix:

$$\frac{\partial L}{\partial a^{(1)}} = (W^{(2)})^T \delta^{(2)}$$

Why Do We Transpose the Weight Matrix?

• In the forward pass, $W^{(2)} \in \mathbb{R}^{d_2 \times d_1}$ maps hidden activations $a^{(1)} \in \mathbb{R}^{d_1 \times 1}$ forward to output pre-activations $z^{(2)} \in \mathbb{R}^{d_2 \times 1}$.
• In the backward pass, we must map output error deltas $\delta^{(2)} \in \mathbb{R}^{d_2 \times 1}$ backward to the hidden layer dimension $\mathbb{R}^{d_1 \times 1}$.
• Transposing flips the matrix shape to $(W^{(2)})^T \in \mathbb{R}^{d_1 \times d_2}$, making the matrix-vector multiplication $(W^{(2)})^T \delta^{(2)}$ dimensionally consistent: $\mathbb{R}^{d_1 \times d_2} \times \mathbb{R}^{d_2 \times 1} = \mathbb{R}^{d_1 \times 1}$

---

The Hidden Layer Error Delta ($\delta^{(1)}$)

To convert the activation sensitivity $\frac{\partial L}{\partial a^{(1)}}$ into the pre-activation error delta $\delta^{(1)} \equiv \frac{\partial L}{\partial z^{(1)}}$, we apply the chain rule across the hidden activation function $f$:

$$\delta_j^{(1)} = \frac{\partial L}{\partial a_j^{(1)}} \cdot \frac{\partial a_j^{(1)}}{\partial z_j^{(1)}} = \left( \sum_k \delta_k^{(2)} W^{(2)}_{kj} \right) \cdot f'(z_j^{(1)})$$

In compact vector notation:

$$\mathbf{\delta^{(1)} = \left( (W^{(2)})^T \delta^{(2)} \right) \odot f'(z^{(1)})}$$

Where:

• For ReLU activation ($f(z) = \max(0, z)$): $$f'(z) = \begin{cases} 1.0 & \text{if } z > 0 \\ 0.0 & \text{if } z \le 0 \end{cases}$$
• For Sigmoid activation ($f(z) = \sigma(z)$): $$f'(z) = \sigma(z)(1 - \sigma(z)) = a^{(1)} \odot (1 - a^{(1)})$$

---

Hidden Layer Parameter Gradients

Once the hidden delta vector $\delta^{(1)} \in \mathbb{R}^{d_1 \times 1}$ is established, we compute the gradients for the first-layer weights $W^{(1)} \in \mathbb{R}^{d_1 \times d_0}$ and biases $b^{(1)} \in \mathbb{R}^{d_1 \times 1}$ using the outer product with the input feature vector $x \in \mathbb{R}^{d_0 \times 1}$:

$$\mathbf{\frac{\partial L}{\partial W^{(1)}} = \delta^{(1)} x^T}$$ $$\mathbf{\frac{\partial L}{\partial b^{(1)}} = \delta^{(1)}}$$

---

Dimensional Consistency Audit for Layer 1

Mathematical Object	Variable	Dimension ($4 \to 2 \to 1$)	Dimension ($d_0 \to d_1 \to d_2$)
Input Feature Vector	$x$	$\mathbb{R}^{4 \times 1}$	$\mathbb{R}^{d_0 \times 1}$
Transposed Input Vector	$x^T$	$\mathbb{R}^{1 \times 4}$	$\mathbb{R}^{1 \times d_0}$
Transposed Output Weight Matrix	$(W^{(2)})^T$	$\mathbb{R}^{2 \times 1}$	$\mathbb{R}^{d_1 \times d_2}$
Backpropagated Error Signal	$(W^{(2)})^T \delta^{(2)}$	$\mathbb{R}^{2 \times 1}$	$\mathbb{R}^{d_1 \times 1}$
Hidden Activation Slope Vector	$f'(z^{(1)})$	$\mathbb{R}^{2 \times 1}$	$\mathbb{R}^{d_1 \times 1}$
Hidden Error Delta Vector	$\delta^{(1)}$	$\mathbb{R}^{2 \times 1}$	$\mathbb{R}^{d_1 \times 1}$
Hidden Weight Gradient Matrix	$\frac{\partial L}{\partial W^{(1)}} = \delta^{(1)} x^T$	$\mathbb{R}^{2 \times 1} \times \mathbb{R}^{1 \times 4} = \mathbf{\mathbb{R}^{2 \times 4}}$	$\mathbb{R}^{d_1 \times 1} \times \mathbb{R}^{1 \times d_0} = \mathbf{\mathbb{R}^{d_1 \times d_0}}$
Hidden Bias Gradient Vector	$\frac{\partial L}{\partial b^{(1)}} = \delta^{(1)}$	$\mathbf{\mathbb{R}^{2 \times 1}}$	$\mathbf{\mathbb{R}^{d_1 \times 1}}$

---

Step-by-Step Numerical Walkthrough: 'Die Hard in Space'

Let us continue our backward pass on 'Die Hard in Space' to compute the hidden layer gradients.

• Input feature vector: $x = \begin{bmatrix} 1.0 \\ 0.0 \\ 0.5 \\ 1.0 \end{bmatrix} \begin{matrix} \text{(Action Measurement)} \\ \text{(Romance Measurement)} \\ \text{(Comedy Measurement)} \\ \text{(Sci-Fi Measurement)} \end{matrix}$
• Layer 1 pre-activations: $z^{(1)} = \begin{bmatrix} +4.0 \\ -3.5 \end{bmatrix}$
• Layer 2 weights: $W^{(2)} = \begin{bmatrix} 1.5 & 1.0 \end{bmatrix}$
• Output error delta: $\delta^{(2)} = 0.034690$

Step 1: Project Error Through Transposed Weights

$$(W^{(2)})^T \delta^{(2)} = \begin{bmatrix} 1.5 \\ 1.0 \end{bmatrix} (0.034690) = \begin{bmatrix} 1.5 \times 0.034690 \\ 1.0 \times 0.034690 \end{bmatrix} = \begin{bmatrix} \mathbf{0.052035} \\ \mathbf{0.034690} \end{bmatrix}$$

Step 2: Compute Hidden Activation Slopes

For the ReLU activation function:

$$f'(z^{(1)}) = \begin{bmatrix} \text{ReLU}'(+4.0) \\ \text{ReLU}'(-3.5) \end{bmatrix} = \begin{bmatrix} \mathbf{1.0} \\ \mathbf{0.0} \end{bmatrix}$$

Step 3: Compute Hidden Error Delta $\delta^{(1)}$

Multiply element-wise by the activation slopes:

$$\delta^{(1)} = \begin{bmatrix} 0.052035 \\ 0.034690 \end{bmatrix} \odot \begin{bmatrix} 1.0 \\ 0.0 \end{bmatrix} = \begin{bmatrix} \mathbf{0.052035} \\ \mathbf{0.000000} \end{bmatrix}$$

Pedagogical Insight: Because Hidden Neuron 2 (Rom-Com) had a negative pre-activation ($z_2^{(1)} = -3.5$), its ReLU derivative is $0.0$. This completely zeroes out its error delta ($\delta_2^{(1)} = 0.0$). The network attributes 100% of the hidden blame to Neuron 1 (Summer Popcorn Flick), protecting dormant neurons from unwanted parameter corruption.

Step 4: Compute Hidden Layer Parameter Gradients

Compute the outer product with $x^T = \begin{bmatrix} 1.0 & 0.0 & 0.5 & 1.0 \end{bmatrix}$:

$$\frac{\partial L}{\partial W^{(1)}} = \delta^{(1)} x^T = \begin{bmatrix} 0.052035 \\ 0.000000 \end{bmatrix} \begin{bmatrix} 1.0 & 0.0 & 0.5 & 1.0 \end{bmatrix}$$ $$\frac{\partial L}{\partial W^{(1)}} = \begin{bmatrix} (0.052035)(1.0) & (0.052035)(0.0) & (0.052035)(0.5) & (0.052035)(1.0) \\ (0)(1.0) & (0)(0.0) & (0)(0.5) & (0)(1.0) \end{bmatrix}$$ $$\frac{\partial L}{\partial W^{(1)}} = \begin{bmatrix} \mathbf{0.052035} & \mathbf{0.000000} & \mathbf{0.026018} & \mathbf{0.052035} \\ \mathbf{0.000000} & \mathbf{0.000000} & \mathbf{0.000000} & \mathbf{0.000000} \end{bmatrix}$$

Compute the bias gradient:

$$\frac{\partial L}{\partial b^{(1)}} = \delta^{(1)} = \begin{bmatrix} \mathbf{0.052035} \\ \mathbf{0.000000} \end{bmatrix}$$

Attribution Complete: Backpropagation has traced the studio flop error all the way back to the screenplay's raw features. The Action weight ($W^{(1)}_{11}$) and Sci-Fi weight ($W^{(1)}_{14}$) receive the largest corrective gradients ($+0.0520$), while Romance receives zero gradient because it was absent from the script ($x_2 = 0.0$).

---