Limits and the Formal Derivative Definition

In Lesson 1, we watched the secant slope converge toward $4.0$ as $\Delta x$ became tiny.

Why can't we simply plug $\Delta x = 0$ directly into the secant equation?

\frac{f(x + 0) - f(x)}{0} = \frac{f(x) - f(x)}{0} = \frac{0}{0}

Plugging in $\Delta x = 0$ results in $\frac{0}{0}$ , which is undefined. Division by zero is mathematically impossible.

Calculus solves this dilemma using the concept of a limit.

The Limit Process: Shrinking Intervals Without Division by Zero

A limit evaluates the value that an expression approaches as an input variable gets closer and closer to a target value, without ever needing to touch that target value.

We write:

\lim_{\Delta x \to 0} g(\Delta x)

This is read: "The limit of $g(\Delta x)$ as $\Delta x$ approaches zero."

We do not set $\Delta x = 0$ . Instead, we inspect the algebraic behavior of the ratio as $\Delta x$ shrinks toward zero through non-zero numbers ( $0.1, 0.01, 0.001, \dots$ ).

The Formal Definition of the Derivative

Combining the secant slope equation with the limit operator gives the Formal Definition of the Derivative:

\frac{dy}{dx} = \lim_{\Delta x \to 0} \frac{f(x + \Delta x) - f(x)}{\Delta x}

In Lagrange notation, the derivative is also written as $f'(x)$ (pronounced "f-prime of x"):

f'(x) = \lim_{\Delta x \to 0} \frac{f(x + \Delta x) - f(x)}{\Delta x}

This equation defines the instantaneous rate of change of the function $f(x)$ with respect to $x$ at any point where the limit exists.

Symbol Translation: Greek $\Delta$ vs. Latin $d$

Understanding calculus notation removes its intimidating appearance.

Notice how the symbols shift between macroscopic and microscopic analysis:

Macroscopic Difference (Secant Slope):       Δy / Δx   (Greek Delta: measurable change)
Microscopic Differential (Tangent Slope):     dy / dx   (Latin d: infinitesimal nudge)

$\Delta x$ (Delta x): A macroscopic, real-world difference (e.g., $\Delta x = 1.0$ hour or $\Delta x = 0.5$ units).
$dx$ (Differential x): An infinitesimal nudge—a microscopic step in $x$ that is smaller than any measurable real number, yet strictly non-zero.
$dy$ (Differential y): The resulting infinitesimal response in output $y$ caused by the input nudge $dx$ .

The notation $\frac{dy}{dx}$ (introduced by Gottfried Wilhelm Leibniz) is written as a fraction for a deliberate reason: it represents the ratio of the infinitesimal output change $dy$ to the infinitesimal input nudge $dx$ .

When reading $\frac{dy}{dx}$ in neural networks, translate the fraction into plain English:

$\text{\textit{"The resulting change in output $y$, for every microscopic nudge in input $x$."}}$

First-Principles Derivation of $f(x) = x^2$

Let's derive the exact derivative of $f(x) = x^2$ algebraically from the formal definition, with zero hand-waving.

We begin with the definition:

\frac{dy}{dx} = \lim_{\Delta x \to 0} \frac{f(x + \Delta x) - f(x)}{\Delta x}

Step 1: Substitute $f(x) = x^2$ into the numerator:

f(x + \Delta x) = (x + \Delta x)^2

\frac{dy}{dx} = \lim_{\Delta x \to 0} \frac{(x + \Delta x)^2 - x^2}{\Delta x}

Step 2: Expand the binomial term $(x + \Delta x)^2$ :

(x + \Delta x)^2 = x^2 + 2x\Delta x + (\Delta x)^2

Substitute this back into the fraction:

\frac{dy}{dx} = \lim_{\Delta x \to 0} \frac{x^2 + 2x\Delta x + (\Delta x)^2 - x^2}{\Delta x}

Step 3: Simplify the numerator ( $x^2 - x^2 = 0$ ):

\frac{dy}{dx} = \lim_{\Delta x \to 0} \frac{2x\Delta x + (\Delta x)^2}{\Delta x}

Step 4: Factor out $\Delta x$ from the numerator:

\frac{dy}{dx} = \lim_{\Delta x \to 0} \frac{\Delta x (2x + \Delta x)}{\Delta x}

Step 5: Cancel the non-zero $\Delta x$ in numerator and denominator:

\frac{dy}{dx} = \lim_{\Delta x \to 0} (2x + \Delta x)

Step 6: Evaluate the limit as $\Delta x \to 0$ :

As $\Delta x$ approaches $0$ , the second term simply vanishes:

\frac{dy}{dx} = 2x + 0 = \mathbf{2x}

We have proven from first principles that for the function $f(x) = x^2$ , the derivative at any point $x$ is:

\frac{d}{dx}[x^2] = 2x

Let's evaluate this general formula at specific coordinates:

At $x = 2$ : $\frac{dy}{dx} = 2(2) = \mathbf{4.0}$ (matching our secant calculations exactly).
At $x = 0$ : $\frac{dy}{dx} = 2(0) = \mathbf{0.0}$ (the flat base of the parabola).
At $x = -3$ : $\frac{dy}{dx} = 2(-3) = \mathbf{-6.0}$ (a steep downhill slope).

Key Mathematical Takeaways: Secant-to-Tangent Convergence

Secant-to-Tangent Convergence: As the perturbation interval $\Delta x \to 0$ , the secant line between $(x_0, f(x_0))$ and $(x_0+\Delta x, f(x_0+\Delta x))$ rotates smoothly into the instantaneous tangent line, and the average rate of change $\frac{\Delta y}{\Delta x}$ converges to $\frac{dy}{dx}$ .
Slope as Derivative Value: The slope of the tangent line at any point $x_0$ matches the analytical derivative $f'(x_0) = 2ax_0 + b$ .
Stationary Point Horizon: At the vertex ( $x = 0$ for $f(x)=x^2$ ), the tangent line becomes completely horizontal ( $\frac{dy}{dx} = 0$ ), marking the stationary point.