Calculating Partial Derivatives of Weights

Now that we understand the freezing principle, we formalize the mathematics of partial differentiation.

The Partial Derivative Symbol: $\partial$ vs $d$

In single-variable calculus, we use the straight Latin letter $d$ :

\frac{dy}{dx} \qquad \text{(Total derivative: $x$ is the only independent variable)}

In multivariable calculus, we use the stylized curly symbol $\partial$ (pronounced "del" or "partial"):

\frac{\partial L}{\partial w_i} \qquad \text{(Partial derivative: differentiate with respect to $w_i$, holding all other inputs constant)}

The symbol $\partial$ serves as an explicit mathematical flag: "This system has multiple independent inputs. We are measuring sensitivity to one specific variable while holding every other variable strictly frozen."

Formal Limit Definition of the Partial Derivative

For a multivariable function $f(w_1, w_2, \dots, w_n)$ , the formal partial derivative with respect to parameter $w_i$ is defined as:

\frac{\partial f}{\partial w_i} = \lim_{\Delta w_i \to 0} \frac{f(w_1, \dots, w_i + \Delta w_i, \dots, w_n) - f(w_1, \dots, w_i, \dots, w_n)}{\Delta w_i}

Notice that in the numerator, only the $i$ -th variable receives the perturbation $\Delta w_i$ . Every other parameter ( $w_1, \dots, w_{i-1}, w_{i+1}, \dots, w_n$ ) remains completely identical in both terms.

Step-by-Step Differentiation: The Affine Linear Sum

Let's compute the partial derivatives of the fundamental neural network operation: the affine linear pre-activation sum $z$ :

z = w_1 x_1 + w_2 x_2 + \dots + w_n x_n + b = \sum_{j=1}^n w_j x_j + b

Let's differentiate $z$ with respect to weight $w_1$ , holding all other weights ( $w_2, \dots, w_n$ ) and bias $b$ constant:

First term ( $w_1 x_1$ ): With respect to $w_1$ , the input feature $x_1$ is a constant coefficient multiplier. Applying the linear multiplier rule ( $\frac{d}{dx}[cx] = c$ ): $\frac{\partial}{\partial w_1}[w_1 x_1] = x_1$
Competing weight terms ( $w_2 x_2, \dots, w_n x_n$ ): None of these terms contain $w_1$ . Because $w_2, \dots, w_n$ and $x_2, \dots, x_n$ are treated as constants, their product is a constant number. The derivative of any constant is zero: $\frac{\partial}{\partial w_1}[w_2 x_2] = 0, \quad \dots, \quad \frac{\partial}{\partial w_1}[w_n x_n] = 0$
Bias term ( $b$ ): The bias is a constant additive offset with no $w_1$ term: $\frac{\partial}{\partial w_1}[b] = 0$
Sum the results: $\frac{\partial z}{\partial w_1} = x_1 + 0 + \dots + 0 + 0 = \mathbf{x_1}$

By identical logic, the partial derivative of the linear sum with respect to any arbitrary weight $w_i$ is simply its corresponding feature measurement:

\frac{\partial z}{\partial w_i} = x_i

Differentiating with Respect to Bias $b$ :

Now let's differentiate $z$ with respect to the bias parameter $b$ , holding all weights ( $w_1, \dots, w_n$ ) constant:

Every weight product $w_j x_j$ contains no $b$ term, so their derivatives are all zero: $\frac{\partial}{\partial b}[w_j x_j] = 0 \quad \text{for all } j \in \{1, \dots, n\}$
The bias term $b$ has a linear coefficient of $1$ : $\frac{\partial}{\partial b}[b] = 1$
Summing the terms: $\frac{\partial z}{\partial b} = 0 + 0 + \dots + 0 + 1 = \mathbf{1}$

Summary of Linear Sum Sensitivities:
  ∂z / ∂w_i = x_i   (The sensitivity of pre-activation to a weight is the feature measurement itself)
  ∂z / ∂b   = 1     (The sensitivity of pre-activation to bias is always exactly 1)

Step-by-Step Partial Differentiation of Linear MSE Loss

Now let's connect parameter weights directly to the final Mean Squared Error loss.

Consider a single-layer linear model where predicted output equals pre-activation ( $\hat{y} = z = w_1 x_1 + w_2 x_2 + b$ ), evaluated against ground-truth target $y$ :

L = (y - \hat{y})^2 = \left(y - (w_1 x_1 + w_2 x_2 + b)\right)^2

Let's compute the partial derivative $\frac{\partial L}{\partial w_1}$ using algebraic expansion and term-by-term differentiation:

Method 1: Expanding the Quadratic Expression

Expand the squared loss formula:

L = y^2 - 2y(w_1 x_1 + w_2 x_2 + b) + (w_1 x_1 + w_2 x_2 + b)^2

Now differentiate each part with respect to $w_1$ , treating $y, w_2, b, x_1, x_2$ as constants:

Target constant term ( $y^2$ ): $\frac{\partial}{\partial w_1}[y^2] = 0$
Cross-product term ( $-2y(w_1 x_1 + w_2 x_2 + b)$ ): $\frac{\partial}{\partial w_1}[-2y w_1 x_1 - 2y w_2 x_2 - 2yb] = -2y x_1 - 0 - 0 = -2y x_1$
Squared sum term ( $(w_1 x_1 + w_2 x_2 + b)^2 = \hat{y}^2$ ): Applying the power rule and inner linear derivative: $\frac{\partial}{\partial w_1}[(w_1 x_1 + w_2 x_2 + b)^2] = 2(w_1 x_1 + w_2 x_2 + b) \cdot \frac{\partial}{\partial w_1}[w_1 x_1 + w_2 x_2 + b] = 2\hat{y} \cdot x_1$
Combine the derivatives: $\frac{\partial L}{\partial w_1} = -2y x_1 + 2\hat{y} x_1 = \mathbf{2(\hat{y} - y) x_1}$

Under the $\frac{1}{2}$ Scaled Loss Convention:

For the standard calculus-scaled Mean Squared Error $L_{\text{scaled}} = \frac{1}{2}(y - \hat{y})^2$ :

\frac{\partial L_{\text{scaled}}}{\partial w_1} = \frac{1}{2} \cdot 2(\hat{y} - y) x_1 = \mathbf{(\hat{y} - y) x_1}

The Fundamental Error Attribution Factoring:

Notice how the partial derivative naturally decomposes into two clean components:

\frac{\partial L}{\partial w_i} = \underbrace{2(\hat{y} - y)}_{\text{Output Loss Derivative } \frac{\partial L}{\partial \hat{y}}} \cdot \underbrace{x_i}_{\text{Input Feature Derivative } \frac{\partial \hat{y}}{\partial w_i}}

The first term, $2(\hat{y} - y)$ , is the output prediction error (how wrong the model's prediction was).
The second term, $x_i$ , is the feature measurement (how active that specific input was during the forward pass).

If a feature was absent ( $x_i = 0$ ), its weight partial derivative is $\frac{\partial L}{\partial w_i} = 0$ . A weight cannot be blamed for a prediction error if its input feature was never active.