curve_fitter.py
Everyone meets TensorFlow through model.fit() and never
finds out what is underneath. Underneath is not a neural-network library.
Underneath is an automatic differentiation engine — a tape that
records arithmetic and plays it backwards.
Here it is, alone, recovering four hidden coefficients of a cubic from 200 noisy
points. Four tf.Variable objects. Zero Keras layers.
Every number on this page is captured output from one real run — —, —, seed —. Nothing here is illustrative.
— training states were captured during the real run and written
to results.json. The animation below replays them in order:
every curve you see is a set of coefficients that actually existed inside the
optimizer at that step.
The four true coefficients are in the script only so we can grade ourselves — the optimizer never sees them. Tolerance is ±— per coefficient.
| coef | learned | truth | error | within tol. |
|---|
The errors are not zero and they never could be. Each y is the
truth plus a draw from Normal(0, 1), and noise is not a
function of x — nothing in the data identifies it. With 200 points
the standard error on c and d is around 0.10,
so a gap of that size is the dataset's, not the optimizer's. Extra steps do not shrink it;
the loss below is flat long before step 1500. This is the ordinary condition of real data,
and it is why a model that fits its training set perfectly has learned the noise.
Mean squared error at each captured step, on a log scale. It falls by a factor of — and then stops — flattening onto the noise floor, which is where the truth lives.
—
Four lines. This is the whole engine, and it is the same four lines for a 50-layer network.
From the moment the with block opens, TensorFlow writes down
every operation that touches a trainable variable — the cube, the multiply, the add —
together with its intermediate values. That list is not like a computation
graph. It is the graph, built by running ordinary Python.
with tf.GradientTape() as tape:
pred = predict(a, b, c, d, x)
The loss collapses 200 disagreements into a single scalar. Everything downstream only ever asks one question of it: which way should each variable move to make this number smaller?
loss = tf.reduce_mean(tf.square(pred - y))
This walks the recorded list from the loss back to the variables, applying the chain rule at each recorded operation. There is no second, hidden algorithm. Backprop is not a neural-network technique — it is reverse-mode automatic differentiation, and it is older than the networks it is famous for.
grads = tape.gradient(loss, [a, b, c, d])
Not one line. The tape has no idea whether pred came from four
scalars or from fifty layers of attention; it recorded operations either way.
Adam is a bookkeeper — two running averages per parameter — and computes no
derivatives at all. fit() is a for-loop around these four lines
with a progress bar attached.
optimizer.apply_gradients(zip(grads, [a, b, c, d]))
curve_fitter.py — the code that produced everything above.
#!/usr/bin/env python3
"""
curve_fitter.py — TensorFlow with the Keras removed.
THE CONTRARIAN CLAIM
--------------------
Almost everyone meets TensorFlow through Keras:
model = keras.Sequential([...])
model.compile(optimizer="adam", loss="mse")
model.fit(x, y, epochs=100)
Three lines, and you have learned nothing about what TensorFlow *is*. You have
learned an API. `fit()` is a sealed box, and the box is doing the only thing
that actually matters.
TensorFlow is not a neural-network library. TensorFlow is an **automatic
differentiation engine** that happens to ship with a neural-network library
bolted on top. Strip Keras away and what remains is small enough to read in one
sitting:
1. Put numbers in tf.Variable objects.
2. Compute something with them inside a tf.GradientTape().
3. Ask the tape which direction each number should move.
4. Move them. Repeat.
That is the whole engine. This file does exactly that — no Sequential, no Dense,
no compile, no fit, no callbacks — and uses it to solve a problem you can check
by eye: recovering four hidden coefficients of a cubic from noisy samples.
The punchline is at the bottom of the training loop, in a comment. If you take
one thing from this file, take that comment.
Run:
python curve_fitter.py # train, print the verdict, draw ASCII plot
python curve_fitter.py --json # also write results.json for the showcase site
"""
import argparse
import datetime
import json
import numpy as np
import tensorflow as tf
# ---------------------------------------------------------------------------
# 0. REPRODUCIBILITY
# ---------------------------------------------------------------------------
# Both RNGs get seeded: numpy draws the noise, TensorFlow draws the initial
# parameter values. Seed them both or "the same run" is not the same run.
SEED = 8
np.random.seed(SEED)
tf.random.set_seed(SEED)
STEPS = 1500
LEARNING_RATE = 0.05
N_POINTS = 200
TOLERANCE = 0.15 # how close to truth we claim to get, per coefficient
# ---------------------------------------------------------------------------
# 1. THE HIDDEN TRUTH
# ---------------------------------------------------------------------------
# These four numbers generate the data. The optimizer never sees them. They
# exist in this file only so that at the end we can grade ourselves — which is
# a luxury you will never have on real data, and the entire reason a synthetic
# problem is the right place to learn the mechanism.
TRUE = {"a": 0.5, "b": -1.2, "c": 0.8, "d": 2.0} # y = a·x³ + b·x² + c·x + d
X = np.random.uniform(-3.0, 3.0, N_POINTS).astype(np.float32)
_clean = TRUE["a"] * X**3 + TRUE["b"] * X**2 + TRUE["c"] * X + TRUE["d"]
# Gaussian noise, sigma = 1.0. This is the honest part of the setup: because
# every y is the truth PLUS a random number, exact recovery is IMPOSSIBLE. No
# optimizer, no learning rate, no amount of extra steps can undo noise that was
# never a function of x. The best any method can do is land near the truth, and
# "near" is set by the noise, not by the algorithm. Real data is exactly like
# this — which is why a model that fits your training set perfectly has, almost
# always, learned the noise.
NOISE = np.random.normal(0.0, 1.0, N_POINTS).astype(np.float32)
Y = (_clean + NOISE).astype(np.float32)
x = tf.constant(X)
y = tf.constant(Y)
# ---------------------------------------------------------------------------
# 2. THE PARAMETERS
# ---------------------------------------------------------------------------
# Four scalars, randomly initialized, that the machine is allowed to change.
#
# These ARE "weights". Not "like" weights — they are the same object, the same
# class, touched by the same optimizer and the same gradient machinery that a
# 400-billion-parameter language model uses. The only difference between this
# file and that model is that here there are four of them instead of four
# hundred billion, and you can print all four on one line.
#
# Random init is not decoration. It is the starting point of the search: the
# optimizer's whole job is the path from these arbitrary numbers to good ones,
# and the showcase site animates exactly that path.
a = tf.Variable(tf.random.normal([], stddev=0.5), name="a")
b = tf.Variable(tf.random.normal([], stddev=0.5), name="b")
c = tf.Variable(tf.random.normal([], stddev=0.5), name="c")
d = tf.Variable(tf.random.normal([], stddev=0.5), name="d")
PARAMS = [a, b, c, d]
INITIAL = [float(p.numpy()) for p in PARAMS]
# tf.optimizers.Adam is the one piece of Keras-adjacent code in this file, and
# it is a bookkeeper, not a brain: it holds two running averages per parameter
# and turns gradients into steps. It computes no derivatives. You could replace
# it with six lines of arithmetic and nothing else here would change. What is
# *not* imported: Sequential, Dense, Model, compile, fit. The layers are the
# part people mistake for TensorFlow, and the part we are doing without.
optimizer = tf.optimizers.Adam(learning_rate=LEARNING_RATE)
def predict(av, bv, cv, dv, xs):
"""The model. One line, because a model is just a parameterised function."""
return av * xs**3 + bv * xs**2 + cv * xs + dv
# ---------------------------------------------------------------------------
# 3. THE TRAINING LOOP ← this is the file
# ---------------------------------------------------------------------------
def checkpoint_schedule(total):
"""Dense early, sparse late — because that is where the movement is.
Captured states are replayed as an animation by the showcase site, so the
schedule is chosen for the eye: the curve does most of its travelling in
the first 50 steps and then creeps.
"""
steps = set(range(0, 21))
steps |= set(range(20, 201, 10))
steps |= set(range(200, total + 1, 50))
steps.add(total)
return sorted(s for s in steps if s <= total)
def train(verbose=True):
schedule = set(checkpoint_schedule(STEPS))
checkpoints = []
if verbose:
print(f"training {len(PARAMS)} parameters for {STEPS} steps "
f"(Adam, lr={LEARNING_RATE})\n")
print(f"{'step':>5} {'loss':>10} {'a':>7} {'b':>7} {'c':>7} {'d':>7}")
print("-" * 55)
for step in range(STEPS + 1):
with tf.GradientTape() as tape:
pred = predict(a, b, c, d, x)
loss = tf.reduce_mean(tf.square(pred - y)) # mean squared error
# Snapshot BEFORE the update, so a recorded state is self-consistent:
# these are the coefficients that produced this loss. The site animates
# these snapshots, so an off-by-one here would animate a lie.
state = (float(loss.numpy()), float(a.numpy()), float(b.numpy()),
float(c.numpy()), float(d.numpy()))
# ------------------------------------------------------------------
# THE THREE COMMENTS THIS FILE EXISTS FOR
# ------------------------------------------------------------------
# (1) The tape is a RECORDER. From the moment `with tf.GradientTape()`
# opens until it closes, TensorFlow writes down every single
# operation that touched a trainable variable — the cube, the
# multiply, the add, the subtract, the square, the mean — along
# with the intermediate values. That list is not a metaphor for a
# computation graph; it *is* the graph, built by running the code.
#
# (2) tape.gradient() PLAYS THE RECORDING BACKWARDS. It walks that list
# from the loss back to the variables, applying the chain rule at
# each recorded operation. There is no other step, no hidden second
# algorithm: this rewind IS backpropagation. Backprop is not a
# neural-network technique. It is reverse-mode automatic
# differentiation, and it predates neural networks by decades.
grads = tape.gradient(loss, PARAMS)
# (3) Now the part worth sitting with: SWAP THE CUBIC FOR A 50-LAYER
# NETWORK AND NOTHING ABOUT THIS LOOP CHANGES. Not one line. The
# tape does not know or care whether `pred` came from four scalars
# or from fifty layers of attention. Open tape → compute loss →
# tape.gradient → apply_gradients is the entire training loop of
# every model TensorFlow has ever trained. Keras's fit() is a
# for-loop around these four lines with progress bars attached.
optimizer.apply_gradients(zip(grads, PARAMS))
if step in schedule:
checkpoints.append({
"step": step, "loss": state[0],
"a": state[1], "b": state[2], "c": state[3], "d": state[4],
})
# Watch it converge. The coefficients are printed live, so the reader
# can see them walk from random noise toward the hidden truth.
if verbose and (step % 300 == 0 or step == STEPS):
print(f"{step:>5} {state[0]:>10.5f} {state[1]:>7.3f} "
f"{state[2]:>7.3f} {state[3]:>7.3f} {state[4]:>7.3f}")
return checkpoints
# ---------------------------------------------------------------------------
# 4. THE VERDICT
# ---------------------------------------------------------------------------
def verdict(learned):
"""Print learned-vs-truth, aligned, three decimals. No rounding in our favour."""
print()
print("VERDICT — learned vs. hidden truth")
print("-" * 46)
print(f"{'coef':>5} {'learned':>10} {'truth':>10} {'error':>10} ok")
print("-" * 46)
all_ok = True
for k in ("a", "b", "c", "d"):
err = learned[k] - TRUE[k]
ok = abs(err) <= TOLERANCE
all_ok &= ok
print(f"{k:>5} {learned[k]:>10.3f} {TRUE[k]:>10.3f} "
f"{err:>+10.3f} {'yes' if ok else 'NO'}")
print("-" * 46)
print(f"all four within ±{TOLERANCE:.2f} of truth: {'YES' if all_ok else 'NO'}")
# The residual gap is not optimizer failure. With sigma=1 noise and 200
# points, the standard error on c and d is roughly 0.10, so a gap of that
# size is the data's fault and no amount of extra training removes it.
print("the remaining gap is the noise, not the optimizer — see the header.")
return all_ok
# ---------------------------------------------------------------------------
# 5. ASCII PLOT — no matplotlib, no dependencies, works over ssh
# ---------------------------------------------------------------------------
W, H = 60, 15
def ascii_plot(curve_xs, curve_ys):
xs_all = np.concatenate([X, curve_xs])
ys_all = np.concatenate([Y, curve_ys])
xmin, xmax = float(xs_all.min()), float(xs_all.max())
ymin, ymax = float(ys_all.min()), float(ys_all.max())
# Epsilon guard: a degenerate range would divide by zero and a flat curve
# is a perfectly legal thing for a fitter to produce.
eps = 1e-9
xspan = max(xmax - xmin, eps)
yspan = max(ymax - ymin, eps)
grid = [[" "] * W for _ in range(H)]
def put(px, py, ch):
col = int((px - xmin) / xspan * (W - 1))
row = int((ymax - py) / yspan * (H - 1))
if 0 <= col < W and 0 <= row < H:
grid[row][col] = ch
for px, py in zip(X, Y):
put(float(px), float(py), "·") # data ·
for px, py in zip(curve_xs, curve_ys):
put(float(px), float(py), "#") # fit #
print()
print(f"DATA (·) vs FITTED CURVE (#) y ∈ [{ymin:.1f}, {ymax:.1f}]")
print("+" + "-" * W + "+")
for row in grid:
print("|" + "".join(row) + "|")
print("+" + "-" * W + "+")
print(f" x = {xmin:.1f}" + " " * (W - 18) + f"x = {xmax:.1f}")
# ---------------------------------------------------------------------------
# 6. MAIN
# ---------------------------------------------------------------------------
def main():
ap = argparse.ArgumentParser(description=__doc__.split("\n")[1])
ap.add_argument("--json", action="store_true",
help="write results.json (captured run data for the showcase site)")
args = ap.parse_args()
print(__doc__.split("Run:")[0].strip().split("\n")[0])
print(f"tensorflow {tf.__version__} | seed {SEED} | "
f"{N_POINTS} noisy points | zero Keras layers\n")
checkpoints = train()
learned = {k: float(v.numpy()) for k, v in zip("abcd", PARAMS)}
verdict(learned)
curve_xs = np.linspace(float(X.min()), float(X.max()), W).astype(np.float32)
curve_ys = np.array(predict(a, b, c, d, tf.constant(curve_xs))).astype(float)
ascii_plot(curve_xs, curve_ys)
if args.json:
results = {
"run_date": datetime.date.today().isoformat(),
"tf_version": tf.__version__,
"seed": SEED,
"steps": STEPS,
"learning_rate": LEARNING_RATE,
"noise_sigma": 1.0,
"tolerance": TOLERANCE,
"initial": dict(zip("abcd", INITIAL)),
"checkpoints": checkpoints,
"learned": learned,
"truth": TRUE,
"data_points": [[round(float(px), 5), round(float(py), 5)]
for px, py in zip(X, Y)],
"fitted_curve": [[round(float(px), 5), round(float(py), 5)]
for px, py in zip(curve_xs, curve_ys)],
}
with open("results.json", "w") as fh:
json.dump(results, fh, indent=1)
print(f"\nwrote results.json ({len(checkpoints)} captured training states)")
if __name__ == "__main__":
main()
One dependency. The whole thing trains in a couple of seconds on a laptop CPU — there is nothing here that wants a GPU.
python3 -m venv .venv && source .venv/bin/activate pip install tensorflow numpy
python curve_fitter.py # train, print the verdict, draw the ASCII plot python curve_fitter.py --json # also write results.json (the data on this page)
The seed is fixed, so your run reproduces the table above exactly — same coefficients,
same loss, same numbers. Change SEED and the errors move: that is
the noise talking, and watching it move is the most useful thing you can do with this file.
Then try replacing predict() with something wilder and leaving the
training loop untouched. It will still work. That is the entire point.