Day 3 · ML basics
translated for devs

The maths, in dev terms

Save this one

Everything the video skipped: the actual formula behind each term, what it means in plain English, the developer version, and the mistake to avoid. Plus gradient descent from scratch in 12 lines.

• 9 terms
• Formulas + code
• Save it

• Featuresinputs

  The columns the model learns from. One example is a row of features; the whole dataset is a table of m rows and n features.

  Dev: Function arguments

  Watch out: More features isn't automatically better. Irrelevant ones add noise.

  Notation

  x = (x₁, x₂, …, x_n)  ·  X has shape (m, n)

  X = df[["hours_slept", "lines_of_code"]]

• Targetoutput

  The answer you want the model to predict. The model's guess is written ŷ ("y-hat"); the real answer is y.

  Dev: Expected output in a unit test

  Watch out: Never let the target sneak into the features ("data leakage"). Your model will look perfect and fail in prod.

  Notation

  y = real answer  ·  ŷ = prediction

  y = df["bugs_shipped"]

• Linear regressionpredicts a number

  Multiply each feature by a weight, add them up, add a bias. Training means finding the w and b that fit the data best.

  Dev: A function with tunable config values

  Watch out: It assumes a roughly straight-line relationship. Curvy data needs extra features (e.g. x²).

  Model

  f_w,b(x) = w · x + b = w₁x₁ + … + w_nx_n + b

  LinearRegression().fit(X, y)

• Logistic regressionpredicts yes / no

  Same linear formula, then squashed by the sigmoid into a probability between 0 and 1. Above 0.5 means "yes".

  Dev: Same function, plus an if-statement

  Watch out: Despite the name, it's a classification model, not regression.

  Model

  z = w · x + b  →  g(z) = 11 + e^−z

  predict 1 if g(z) ≥ 0.5, else 0

• Cost functionhow wrong

  One number that sums up how wrong the model is across every example. Errors are squared so big misses count much more than small ones.

  Dev: Your failing test count

  Watch out: Linear and logistic use different costs. Squared error makes logistic's cost bumpy, so it uses log loss.

  Linear: squared error

  J(w,b) = 12m Σ_i=1^m (f(x⁽ⁱ⁾) − y⁽ⁱ⁾)²

  Logistic: log loss

  L = −[ y log(ŷ) + (1 − y) log(1 − ŷ) ]

• Gradient descenthow it learns

  Repeat: work out which direction lowers the cost (the gradient), take a small step that way. Stop when the cost stops dropping.

  Dev: A for loop: nudge, rerun, repeat

  Watch out: Update w and b at the same time, using the old values for both.

  Repeat until converged

  w := w − α ∂J∂w    b := b − α ∂J∂b

  ∂J∂w = 1m Σ (f(x⁽ⁱ⁾) − y⁽ⁱ⁾) x⁽ⁱ⁾    ∂J∂b = 1m Σ (f(x⁽ⁱ⁾) − y⁽ⁱ⁾)

• Learning ratestep size

  α controls how big each step is. Too big and the cost bounces or explodes; too small and training crawls.

  Dev: Step size per iteration

  Watch out: Plot cost vs iterations. It should go down every time. If it goes up, α is too big.

  Values to try (≈3× apart)

  α ∈ { 0.001, 0.003, 0.01, 0.03, 0.1, 0.3 }

  cost rising → α too big cost flat-ish → α too small

• Feature scalingsame range

  Rescale features so they cover similar ranges. Otherwise one large-valued feature dominates and gradient descent zigzags slowly.

  Dev: Converting ms and seconds to one unit

  Watch out: Fit the scaler on training data only, then apply the same μ and σ to test data.

  Standardisation (z-score)

  x′ = x − μσ    mean 0, most values in −3…3

  Min-max

  x′ = x − minmax − min    range 0…1

• SGDfast updates

  Stochastic gradient descent updates after one random example (or a mini-batch of 32–256) instead of the whole dataset. Each step is noisier, but far faster.

  Dev: Run one test, fix, repeat

  Watch out: A jumpy cost curve is normal. Shuffle the data every epoch (one full pass).

  Update per example i

  w := w − α (f(x⁽ⁱ⁾) − y⁽ⁱ⁾) x⁽ⁱ⁾

  SGDRegressor().fit(X_scaled, y)

Andrew Ng's ML course

Supervised Machine Learning: Regression and Classification

Start here. This is where all nine terms come from, taught slowly with quizzes and labs. Weeks 1–3 cover everything on the cheat sheet.

• Weeks 1–3
• Enroll free on Coursera
• Python labs

1. Week 1: Linear regressionSupervised learning, the cost function and gradient descent
2. Week 2: Multiple featuresVectorisation, feature scaling and choosing a learning rate
3. Week 3: Logistic regressionClassification, the sigmoid, decision boundaries and the logistic cost

Watch it clicked

Josh Starmer · short, visual, a little silly

When a formula doesn't click, watch the matching video. Each one draws the idea out step by step in 10–20 minutes.

• 10–20 min each
• Free

1. Linear regression, clearly explainedFitting the line and measuring how good it is
2. Gradient descent, step by stepThe downhill walk, with the maths drawn out
3. Logistic regressionFrom a straight line to a yes/no probability
4. Stochastic gradient descentWhy using one example at a time is faster

ML Crash Course

Short interactive lessons from Google

Finish here. Drag the sliders yourself and watch the cost go up and down, then take the quiz at the end to check you really get it.

• ~2 hours
• Interactive
• Free

1. LossWhat the cost function measures, and why errors are squared
2. Gradient descentPlus an interactive exercise to play with it
3. HyperparametersLearning rate, batch size and epochs, i.e. where SGD comes in
4. Logistic regressionThe sigmoid and predicting probabilities