The Composite Function Chain Rule

In Topic 1, we measured prediction error on the screenplay for 'Die Hard in Space' using Mean Squared Error:

$$L = (y - \hat{y})^2 = (0 - 0.9820)^2 \approx \mathbf{0.9644}$$

In Topic 2, we established single-variable calculus to compute the instantaneous sensitivity $\frac{dy}{dx}$ of an isolated operation. In Topic 3, we used the freezing principle to calculate partial derivatives $\frac{\partial L}{\partial w_i}$ for a single linear neuron, assembling them into the gradient vector $\nabla L$ to determine the direction of steepest error descent.

[ Module 1: The Forward Pass ]
  Topic 1: Feature Vectors & Dot Products (x, w · x)
  Topic 2: Affine Sums & Activations (z = w · x + b, a = σ(z))
  Topic 3: Parallel Layers & Width (z = W x + b, a = σ(z))
  Topic 4: Multi-Layer Perceptrons & Depth (a^(1) = ReLU(W^(1)x + b^(1)), a^(2) = σ(W^(2)a^(1) + b^(2)))
                                │
                                ▼
[ Module 2: The Backward Pass ]
  Topic 1: Prediction Error & Squared Loss (L = (y - y_hat)^2, ∂L/∂y_hat = 2(y_hat - y))
  Topic 2: Single-Variable Derivatives (dy/dx, tangent slopes, power/sum rules)
  Topic 3: Multivariable Gradients (∂L/∂w_i, ∇_w L, steepest descent -∇L)
  Topic 4: The Chain Rule & Multilayer Backpropagation ◄─── [THIS TOPIC]

However, real deep neural networks never connect raw inputs directly to output losses through a single linear sum.

In a multi-layer network, parameters are buried deep inside composite functional hierarchies. A first-layer weight $W^{(1)}_{ij}$ does not touch the loss directly. Instead, it alters a hidden pre-activation $z^{(1)}_i$, which passes through a non-linear activation function $a^{(1)}_i$, which feeds into an output pre-activation $z^{(2)}$, which passes through an output activation $\hat{y} = a^{(2)}$, which finally produces the loss penalty $L$:

$$x \xrightarrow{W^{(1)}, b^{(1)}} z^{(1)} \xrightarrow{f} a^{(1)} \xrightarrow{W^{(2)}, b^{(2)}} z^{(2)} \xrightarrow{\sigma} a^{(2)} \xrightarrow{\text{Loss}} L$$

To calculate the gradient of the loss with respect to buried internal weights, we cannot use simple single-layer partial derivatives. We need a systematic mathematical engine capable of propagating sensitivity backward through chained functional transformations.

That engine is The Chain Rule, and its algorithmic execution across neural networks is Backpropagation.

---

---

Derive the single-variable chain rule to multiply rates of change across composite functions and propagate sensitivity through chained operations.

In ordinary single-variable calculus, a function maps an input $x$ directly to an output $y = f(x)$.

In a chained system, functions are composed into pipelines: an input $x$ passes into an intermediate function $u = g(x)$, and the resulting intermediate value $u$ passes into an outer function $y = f(u)$:

$$y = (f \circ g)(x) = f(g(x))$$

When the input $x$ is nudged by a tiny amount $\Delta x$, how much does the final output $y$ change?

---

The Rate Multiplication Principle

To understand how sensitivity flows through a chain of operations, consider how local rates of change compound.

Suppose:

1. When input $x$ changes, intermediate variable $u$ changes 3 times as fast (rate of change $\frac{du}{dx} = 3$).
2. When intermediate variable $u$ changes, output variable $y$ changes 2 times as fast (rate of change $\frac{dy}{du} = 2$).

If you nudge $x$ upward by $+1$ unit:

• Intermediate $u$ increases by $+3$ units ($1 \times 3 = 3$).
• Because $u$ increased by $+3$ units, output $y$ increases by $+6$ units ($3 \times 2 = 6$).

The overall rate of change of $y$ with respect to $x$ is the product of the individual rates:

$$\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx} = 2 \times 3 = \mathbf{6}$$

Rates of change across composite functional dependencies do not add—they multiply.

---

Dual-Track Bridge: Mechanical Gear Trains

To anchor this rate multiplication principle in a physical system, examine a three-gear mechanical transmission.

[ Gear A (Input x) ] ──(3:1 Ratio)──► [ Gear B (Intermediate u) ] ──(2:1 Ratio)──► [ Gear C (Output y) ]
     30 Teeth                              10 Teeth                              5 Teeth

• Track 1 (Underlying Mechanism):

  • Gear A (input $x$) has 30 teeth and meshes with Gear B (intermediate $u$), which has 10 teeth. For every 1 full turn of Gear A, Gear B rotates 3 full turns: $\frac{du}{dx} = 3$
  • Gear B is mounted on a shaft that meshes with Gear C (output $y$), which has 5 teeth. For every 1 full turn of Gear B, Gear C rotates 2 full turns: $\frac{dy}{du} = 2$
  • When you rotate Gear A by 1 turn, Gear C rotates by $3 \times 2 = 6$ turns: $\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx} = 2 \times 3 = \mathbf{6}$

• Track 2 (Applied Neural Architecture):

  • In a neural network, Gear A represents a feature measurement $x_1$, Gear B represents the hidden linear sum $z_1^{(1)}$, and Gear C represents the final output prediction $\hat{y}$.
  • If nudging $x_1$ changes $z_1^{(1)}$ at a rate of $+3.0$, and nudging $z_1^{(1)}$ changes $\hat{y}$ at a rate of $+0.5$, the compound sensitivity of the prediction to the input is $(+3.0) \times (+0.5) = +1.5$.

---

Formal Definition and Limit Derivation

Mathematically, the single-variable Chain Rule states that for composite function $y = f(g(x))$ where $u = g(x)$ and $y = f(u)$, if $g$ is differentiable at $x$ and $f$ is differentiable at $g(x)$, then:

$$\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx}$$

Step-by-Step Limit Proof

We define the derivative of $y$ with respect to $x$ using the formal limit definition:

$$\frac{dy}{dx} = \lim_{\Delta x \to 0} \frac{\Delta y}{\Delta x}$$

Let $\Delta u = g(x + \Delta x) - g(x)$ be the resulting change in the intermediate variable. For non-zero $\Delta u$, we multiply and divide the difference quotient by $\Delta u$:

$$\frac{\Delta y}{\Delta x} = \frac{\Delta y}{\Delta u} \cdot \frac{\Delta u}{\Delta x}$$

Because $g$ is continuous, as $\Delta x \to 0$, the change $\Delta u \to 0$. Applying the limit product law:

$$\lim_{\Delta x \to 0} \frac{\Delta y}{\Delta x} = \left( \lim_{\Delta u \to 0} \frac{\Delta y}{\Delta u} \right) \cdot \left( \lim_{\Delta x \to 0} \frac{\Delta u}{\Delta x} \right)$$

Evaluating both limits yields the Chain Rule formula:

$$\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx}$$

---

Step-by-Step Algebraic Verification: $y = (3x + 1)^2$

To verify that the chain rule produces exact analytical derivatives, let us differentiate the composite expression $y = (3x + 1)^2$ using two independent mathematical methods.

Method A: Direct Polynomial Expansion

First, expand the polynomial algebraically before differentiating:

$$y = (3x + 1)^2 = (3x + 1)(3x + 1) = 9x^2 + 6x + 1$$

Apply the standard power and sum rules term-by-term:

$$\frac{dy}{dx} = \frac{d}{dx}[9x^2] + \frac{d}{dx}[6x] + \frac{d}{dx}[1] = 18x + 6$$

Method B: Chain Rule Decomposition

Decompose the expression into outer and inner functions:

• Inner function: $u = g(x) = 3x + 1$
• Outer function: $y = f(u) = u^2$

Differentiate each component separately:

1. Derivative of outer function with respect to $u$: $\frac{dy}{du} = \frac{d}{du}[u^2] = 2u$
2. Derivative of inner function with respect to $x$: $\frac{du}{dx} = \frac{d}{dx}[3x + 1] = 3$

Multiply the derivatives together according to the chain rule:

$$\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx} = (2u) \cdot (3) = 6u$$

Substitute the original inner expression $u = 3x + 1$ back into the formula:

$$\frac{dy}{dx} = 6(3x + 1) = 18x + 6$$

Numerical Evaluation at $x = 2$

• Under Method A: $\frac{dy}{dx} = 18(2) + 6 = 36 + 6 = \mathbf{42}$
• Under Method B: $u = 3(2) + 1 = 7 \implies \frac{dy}{dx} = 6(7) = \mathbf{42}$

Both methods yield identical derivatives ($18x + 6$). The chain rule allows us to differentiate deeply nested functions without expanding algebraic polynomials.

---