Toy Parameter Update Step on Paper

With all parameter gradients computed, we can now perform a single conceptual parameter update step to observe how the network learns on paper.

---

The Parameter Update Rule

In gradient descent, every parameter in the network is adjusted by taking a small step in the direction opposite to its computed gradient:

$$W_{\text{new}} = W_{\text{old}} - \eta \frac{\partial L}{\partial W} \quad \text{or} \quad W \leftarrow W - \eta \nabla_W L$$ $$b_{\text{new}} = b_{\text{old}} - \eta \frac{\partial L}{\partial b} \quad \text{or} \quad b \leftarrow b - \eta \nabla_b L$$

For an individual scalar weight $w$:

$$w_{\text{new}} = w_{\text{old}} - \eta \frac{\partial L}{\partial w}$$

For layer $l$ in a multi-layer neural network:

$$W^{(l)}_{\text{new}} = W^{(l)}_{\text{old}} - \eta \frac{\partial L}{\partial W^{(l)}}$$ $$b^{(l)}_{\text{new}} = b^{(l)}_{\text{old}} - \eta \frac{\partial L}{\partial b^{(l)}}$$

Where $\eta > 0$ is the Step Size Factor (commonly termed the learning rate).

Why Do We Need a Step Size Factor $\eta$?

• The gradient $\frac{\partial L}{\partial W}$ represents the instantaneous slope at the current parameter coordinate.
• If we took a full unscaled step ($\eta = 1.0$), the update could wildly overshoot the minimum of the error bowl.
• Scaling the gradient by a calibrated step size (such as $\eta = 0.5$ or $\eta = 0.1$) ensures controlled, stable progress down the loss surface.

---

Step-by-Step Hand Calculation on 'Die Hard in Space'

Let us apply a single gradient descent update step with step size $\mathbf{\eta = 0.5}$ to all parameters in our 2-layer network.

1. Layer 2 Parameter Updates

• Weight Update: $$W^{(2)}_{\text{new}} = \begin{bmatrix} 1.5 & 1.0 \end{bmatrix} - 0.5 \begin{bmatrix} 0.138760 & 0.000000 \end{bmatrix} = \begin{bmatrix} \mathbf{1.430620} & \mathbf{1.000000} \end{bmatrix}$$
• Bias Update: $$b^{(2)}_{\text{new}} = -2.0 - 0.5(0.034690) = \mathbf{-2.017345}$$

2. Layer 1 Parameter Updates

• Weight Matrix Update: $$W^{(1)}_{\text{new}} = \begin{bmatrix} 3.0 & -2.0 & 0.0 & 3.0 \\ -2.0 & 3.0 & 2.0 & -1.0 \end{bmatrix} - 0.5 \begin{bmatrix} 0.052035 & 0.000000 & 0.026018 & 0.052035 \\ 0.000000 & 0.000000 & 0.000000 & 0.000000 \end{bmatrix}$$ $$W^{(1)}_{\text{new}} = \begin{bmatrix} \mathbf{2.973982} & \mathbf{-2.000000} & \mathbf{-0.013009} & \mathbf{2.973982} \\ \mathbf{-2.000000} & \mathbf{3.000000} & \mathbf{2.000000} & \mathbf{-1.000000} \end{bmatrix}$$
• Bias Vector Update: $$b^{(1)}_{\text{new}} = \begin{bmatrix} -2.0 \\ -1.5 \end{bmatrix} - 0.5 \begin{bmatrix} 0.052035 \\ 0.000000 \end{bmatrix} = \begin{bmatrix} \mathbf{-2.026018} \\ \mathbf{-1.500000} \end{bmatrix}$$

---

Second Forward Pass: Proving Mathematical Loss Reduction

To prove that this single backpropagation update step genuinely improved the model, let us re-run the complete forward pass using our updated parameters.

Step 1: Updated Hidden Layer Forward Pass

Compute $z^{(1)}_{\text{new}} = W^{(1)}_{\text{new}} x + b^{(1)}_{\text{new}}$:

$$z_{1, \text{new}}^{(1)} = (2.973982)(1.0) + (-2.0)(0.0) + (-0.013009)(0.5) + (2.973982)(1.0) - 2.026018$$ $$z_{1, \text{new}}^{(1)} = 2.973982 - 0.006505 + 2.973982 - 2.026018 = \mathbf{+3.915441}$$ $$z_{2, \text{new}}^{(1)} = -3.500000$$

Apply ReLU activation:

$$a^{(1)}_{\text{new}} = \begin{bmatrix} \max(0, 3.915441) \\ \max(0, -3.5) \end{bmatrix} = \begin{bmatrix} \mathbf{3.915441} \\ \mathbf{0.000000} \end{bmatrix}$$

Step 2: Updated Output Layer Forward Pass

Compute $z^{(2)}_{\text{new}} = W^{(2)}_{\text{new}} a^{(1)}_{\text{new}} + b^{(2)}_{\text{new}}$:

$$z^{(2)}_{\text{new}} = (1.430620)(3.915441) + (1.0)(0.0) - 2.017345 = 5.601510 - 2.017345 = \mathbf{+3.584165}$$

Apply Sigmoid activation:

$$a^{(2)}_{\text{new}} = \sigma(3.584165) = \frac{1}{1 + e^{-3.584165}} = \frac{1}{1 + 0.027760} = \mathbf{0.972990} \quad (\mathbf{97.30\%})$$

Step 3: Evaluate Updated Loss Penalty

Compute the new Mean Squared Error loss:

$$L_{\text{new}} = (a^{(2)}_{\text{new}} - y)^2 = (0.972990 - 0.0)^2 = \mathbf{0.946710}$$

Mathematical Proof of Error Reduction

Compare the initial loss against the updated loss:

$$\Delta L = L_{\text{new}} - L_{\text{old}} = 0.946710 - 0.964351 = \mathbf{-0.017641} \quad (\text{Strict Loss Reduction})$$ $$\text{Relative Error Eliminated} = \frac{0.017641}{0.964351} \times 100\% = \mathbf{1.83\%}$$

The single backpropagation update step successfully nudged the prediction downward from $98.20\%$ toward the target of $0.0\%$, strictly reducing prediction loss on paper.

---

Strict Scope Containment

We deliberately conclude our mathematical derivations at this single manual calculation.

In production engineering, training neural networks involves repeating this forward-backward cycle across thousands of batches. However, to maintain pedagogical clarity, the following advanced systems belong to subsequent courses:

• ❌ No Training Loops or Epochs: No automated convergence loops or batch iterators.
• ❌ No Complex Optimizers: No Momentum, Adam, or RMSProp algorithms.
• ❌ No Autograd Frameworks: No PyTorch, TensorFlow, or JAX code abstractions.

Mastering the exact mechanics on paper guarantees that when you later encounter automated engines, you understand the exact linear algebra executing beneath every line of code.

---