┌─ 404human ─────────┐
  │  $ whoami          │
  └────────────────────┘
404human
Rust runtimes, background jobs, AI infrastructure

Learning · learning in progress

Model engineering from the beginning

Learning how models are built, trained, tested, and improved, starting with one adjustable number.

I started with a question about training a small model to write blogs. That led to a much bigger question: how does training even start?

This track follows that question from the beginning. The goal is to understand the model, the data, and the code around it well enough to explain what changed and test whether it helped.

I want to understand what each part does, why it matters, and how it can fail. The learning route starts from plain explanations, small working pieces, their outputs, and the errors we encounter. The assistant handles implementation and calculations, with the code and results available to inspect together. Math comes in when it helps explain the behavior; deeper derivations are optional. The earlier small examples stay available for revision.

Current learning sequence, agreed 9 November 2026: topics 4–8, from how learning goes wrong through adapting an existing model. Topic 4 has failure-mode comparisons, and topic 5 has a data audit and split-leakage demonstration. Topic 6 now spans tokens, count probabilities, adjustable scores, a last-token predictor with learned input embeddings, an attention mechanism with position vectors, a generation loop, and trained attention weights. On 11 November, topic 7 ran its first interrupt/resume proof and topic 8 its first adapter-vs-fine-tune contrast. Broader text-model training and real-model checkpointing/adaptation remain ahead. Explanations, examples, code, recorded results, and revision notes are added as we cover them.

Earlier practical build: a local Qwen assistant uses its first tool. The first request → lookup → answer cycle ran on 7 November; on 11 November a small failure-hunting suite repaired two silent-answer failures, and its traces and case manifest are saved. The assistant keeps expanding a step at a time alongside topics 4–8.

Chapter numbers organize the concepts; they are not quiz prerequisites. Nine chapter notes are available, including the earlier training-loop and validation experiments. The roadmap records the sequence and the later topics. Broader evaluation, resumable text-model training, weight adaptation, and preference learning remain ahead.

[ Read the learning roadmap ]


Chapters, in order

Available notes are linked. Planned chapters will get pages as I work through them.

  1. Weights, predictions, and loss · note available
  2. Gradients and weight updates · note available
  3. Our first training loop · note available
  4. How learning goes wrong · note available
  5. Data a model can learn from · note available
  6. From text to a tiny language model · note available
  7. Training runs we can inspect and resume · note available
  8. Adapting an existing model · note available
  9. The harness around the model · note available
  10. Evaluation and model promotion · planned
  11. Memory, speed, and scale · planned
  12. Learning from preferences · planned

← All learning tracks