The Mean Squared Error Loss Function

To satisfy all four loss function requirements, we construct the Mean Squared Error (MSE) loss function.

---

Single-Sample Squared Error Formulation

For a single training example with ground truth $y$ and prediction $\hat{y}$, the standard squared error loss is defined as:

$$L(y, \hat{y}) = (y - \hat{y})^2$$

Because the square of any real number is non-negative, squaring immediately eliminates the cancellation fallacy:

$$(y - \hat{y})^2 = \big(-( \hat{y} - y )\big)^2 = (\hat{y} - y)^2 \ge 0$$

Whether the model overshoots ($e = -0.95 \implies (-0.95)^2 = \mathbf{0.9025}$) or undershoots ($e = +0.95 \implies (+0.95)^2 = \mathbf{0.9025}$), both mistakes generate the exact same positive scalar penalty.

Let's re-evaluate our two-movie dataset using squared loss:

• Movie 1: $L_1 = (1.0 - 0.50)^2 = (0.50)^2 = \mathbf{0.25}$
• Movie 2: $L_2 = (0.0 - 0.50)^2 = (-0.50)^2 = \mathbf{0.25}$

Summing the squared errors produces a true total penalty:

$$L_{\text{total}} = L_1 + L_2 = 0.25 + 0.25 = \mathbf{0.50}$$

The errors reinforce rather than cancel, providing an accurate, honest measurement of model inaccuracy.

---

Quadratic Penalty Scaling: Blunders vs. Small Errors

Why square the error rather than taking the absolute value ($|y - \hat{y}|$, known as Mean Absolute Error or L1 loss)?

Squaring introduces super-linear (quadratic) penalty growth. As the raw error grows linearly, the squared penalty explodes quadratically:

$$\frac{L(e_2)}{L(e_1)} = \left(\frac{e_2}{e_1}\right)^2$$

Observe how squared penalty scales across different error magnitudes:

| Error Type | Raw Discrepancy ($|e|$) | Absolute Penalty ($|e|$) | Squared Penalty ($e^2$) | Relative Penalty Growth | | :--- | :--- | :--- | :--- | :--- | | Minor Noise | $0.10$ | $0.10$ | $0.01$ | $1\times$ baseline | | Moderate Error | $0.50$ ($5\times$) | $0.50$ | $0.25$ | $25\times$ penalty | | Major Blunder | $1.00$ ($10\times$) | $1.00$ | $1.00$ | $100\times$ penalty |

A mistake that is $10\times$ larger ($1.00$ vs $0.10$) produces a squared penalty that is $100\times$ more severe ($1.00$ vs $0.01$).

Raw Error:      0.10 ────────► 1.00  ( 10x increase )
Squared Loss:   0.01 ────────────────────────► 1.00  ( 100x increase )

This quadratic scaling gives Mean Squared Error a crucial structural property:

1. Tolerance for Minor Imperfections: Tiny errors ($e = 0.10 \to L = 0.01$) incur negligible penalties, preventing the network from over-reacting to minor noise in training data.
2. Aggressive Suppression of Blunders: Catastrophic mistakes ($e = 0.95 \to L = 0.9025$) incur massive penalties, forcing optimization algorithms to focus primary corrective effort on eliminating large blunders first.

---

Dataset Mean Squared Error

When evaluating an entire dataset or mini-batch containing $N$ training examples $\mathcal{D} = \{(x^{(1)}, y^{(1)}), (x^{(2)}, y^{(2)}), \dots, (x^{(N)}, y^{(N)})\}$, we take the arithmetic mean of all individual sample losses:

$$L_{\text{MSE}} = \frac{1}{N} \sum_{i=1}^{N} \left( y^{(i)} - \hat{y}^{(i)} \right)^2$$

Dividing by the sample count $N$ ensures that the loss metric is scale-invariant: a batch of 1,000 samples will produce the same average loss as a batch of 10 samples if the model exhibits the same average prediction accuracy on both.

---

The $\frac{1}{2}$ Scaling Convention in Calculus

When reading machine learning papers and textbooks, you will frequently encounter squared loss written with a constant factor of $\frac{1}{2}$:

$$L_{\text{scaled}} = \frac{1}{2}(y - \hat{y})^2 = \frac{1}{2}(\hat{y} - y)^2$$

Why include the fraction $\frac{1}{2}$?

In calculus, when we take the derivative of a squared term $u^2$ using the power rule, the exponent $2$ drops down as a multiplier:

$$\frac{d}{du}[u^2] = 2u$$

By pre-multiplying the loss function by $\frac{1}{2}$, the $2$ from the power rule is canceled out completely:

$$\frac{d}{du}\left[ \frac{1}{2} u^2 \right] = \frac{1}{2} \cdot 2u = \mathbf{u}$$

This algebraic convenience simplifies subsequent gradient equations, eliminating lingering factors of $2$ throughout backpropagation derivations.

Does Scaling by $\frac{1}{2}$ Change the Optimal Parameters?

No. Multiplying a loss function by any positive constant $c > 0$ scales the vertical height of the loss curve, but does not alter the location of its minimum:

$$\arg\min_{\theta} \left[ \frac{1}{2} L(\theta) \right] = \arg\min_{\theta} L(\theta)$$

The parameter values that minimize $\frac{1}{2}L$ are identical to the parameter values that minimize $L$. In gradient descent, any constant scalar factor is absorbed directly into the learning rate hyperparameter ($\eta$).

---