How an AI Lab Works

How a modern AI lab works, from raw text to a billed API call, in diagrams. Every training example is real, taken from mini-lab.

Clément Thiriet • September 26, 2026

View on GitHub

mini-lab in five questions

What is it? An open-source AI lab, small enough to read. About 6,900 lines of Python train a language model from scratch, serve it behind an OpenAI-compatible API, and bill every token from prepaid credits.

Why build it? A frontier lab is too big to see, and most tutorials stop once the model trains. mini-lab keeps going to the API and the bill, so you can follow the whole chain in one repo.

What can its model do? mini-3 has 5.8M parameters. It tells short children's stories and adds numbers, step by step or with a calculator. It knows nothing else, and says so. It is tiny on purpose: every training stage moves a number you can see.

What do I need? A laptop, no GPU. The whole training takes about 31 minutes on an Apple M5 Pro, or about 70 on its CPU alone.

Can I try it? Yes: on the live demo, or on your laptop with four commands (at the end of this post).

The mini-lab chat app: a short story about a dog, then 347 + 58 solved with the calculator tool
The mini-lab chat app, here with mini-1, the first release. It tells a story, then calls the calculator for a sum. Every reply is billed to the lab's prepaid credits, top right.

The chain

Every lab, big or small, runs the same chain. The rest of this post walks it, one diagram per step.

Research · once per model

Data
stories, sums
→
Tokenizer
text → ids
Pretrain
language
→
Midtrain
chat format
SFT
behavior
→
RL
math practice
Distill
one student
→
Release
mini-3
✓ the same eval after every stage
↓ the inference engine loads the checkpoint

Serving · every request

Your code
openai SDK
sk-mini-…→
API gateway
auth · limits · billing
internal→
Inference
KV cache · batching
Research runs once and ends in a checkpoint: the model's weights, in one file. Serving loads that file and turns each request into an answer and a bill.

Tokens

A model reads numbers, not letters. The tokenizer has 4,096 pieces of text, called tokens, most of them learned from the data, and turns any text into their ids.

41 characters → 11 tokens

Once435·upon443·a258·time402,44·there403·was282·a258·little395·cat458.46

17 characters → 11 tokens

What663·is365·32351452755·+419·32553856?63
· = spacenumber = token id
Common words take one token each. Digits never merge, so 347 always splits into 3, 4, 7: the model can add column by column, but a sum costs one token per digit.

Pretraining: guess the next token

The first training stage reads 57M tokens of stories and sums, and learns one skill: predict the next token.

Story (shortened)

<|bos|>Once upon a time, there was a small, brave dog named Spot. Spot loved to play and run outside. […]

Worksheet

<|bos|>5 + 625 005+625: 5+5=10, 0 00+62: 0+2+1=3, 30 0+6: 0+6=6, 630 So 5 + 625 = 630. The sum of 3 and 0 is 3.
Three documents in four are stories, one in four a sum worksheet. A worked sum adds right to left, one column per line, and the number at the end of each line grows into the answer. <|bos|> marks where each document starts. Documents sit back to back in rows of 256 tokens: no padding, no waste.
·the
↓
·best
94%loss 0.07
·best
↓
·day
23%loss 1.47
·day
↓
·ever
99%loss 0.01
·ever
↓
!
14%loss 2.00
!
↓
<|bos|>
>99%loss 0.00
<|bos|>
↓
One
15%loss 1.89
One
↓
·day
97%loss 0.03
·day
↓
,
>99%loss 0.00

After …the best day ever!<|bos|>, the model's top guesses:

Once51%
One15%
Lily3%

right token: One → loss = −log 0.151 = 1.89

top: what the model readswhat it must predict· = space
The target is the input shifted by one token. At every position the model gives each of the 4,096 tokens a probability, and the loss is −log of the one it gave the right token: near 0 for a safe bet, large for a bad one. Training lowers the average.

After 7,000 steps the base model writes little stories. It can't chat: it has never seen a conversation.

How long should it read? Labs ask a scaling law, and mini-lab ran a small one: four model sizes, each trained with the same compute. At mini-2's budget, a smaller model won: mini was a little too big for what it read. At twice the budget, mini's own size won. So mini-3 kept the model and read twice as much.

Midtraining: learn the chat format

Same loss, new documents: chats, mixed with old text so the model keeps its language. The model learns the format here, from a lot of data, so the next stage needs only a little.

A calculator call · 50 tokens

<|bos|><|system_start|>tools:·calculator<|system_end|><|user_start|>Add·6·and·3.<|user_end|><|assistant_start|><|tool_call_start|>{"name":·"calculator",·"arguments":·{"expression":·"6·+·3"}}<|tool_call_end|><|assistant_end|><|tool_start|>9<|tool_end|><|assistant_start|>The·answer·is·9.<|assistant_end|>
in the losscontext only· = space
Every token counts in the loss, even the user's question and the calculator's result. Special tokens mark who speaks. After <|tool_call_end|> the model stops, your code runs the calculator, and its result comes back between <|tool_start|> and <|tool_end|>.

SFT: learn to behave

SFT (supervised fine-tuning) uses a small, fixed set of 8,000 conversations with what midtraining never shows: system prompts, follow-ups, polite ways to say no. The loss changes too: it now skips every token the model will never have to write.

System prompt · loss on 17 of 33 tokens

<|bos|><|system_start|>Answer·with·the·number·only.<|system_end|><|user_start|>2·+·5<|user_end|><|assistant_start|><|think_start|>2+5:·2+5=7,·7<|think_end|>7<|assistant_end|>

Follow-up · loss on the last answer only (22 of 47)

<|bos|><|user_start|>Add·4·and·2.<|user_end|><|assistant_start|>The·answer·is·6.<|assistant_end|><|user_start|>Plus·2?<|user_end|><|assistant_start|><|think_start|>6+2:·6+2=8,·8<|think_end|>The·answer·is·8.<|assistant_end|>
in the losscontext only· = space
The loss covers the answer only, from the scratchpad to <|assistant_end|>. The scratchpad is not the answer: the API returns it on the side, so “the number only” still holds. In a follow-up, even the earlier answer only counts as context: to read, not to copy.

RL: practice with a grader

SFT can only copy answers someone wrote. RL (reinforcement learning) needs no answers, only a program that checks them. mini-lab's RL practices math and nothing else: sums of up to 5 digits, where chat data stops at 3. It uses GRPO: the model tries each problem 8 times, the program grades every try, and each try gets pushed up or down by how it compares to the group.

Prompt

Compute 5+8921

0005+8921: 5+1=6, 6 … The answer is 8926.reward 1+0.72
0005+8921: 5+1=6, 6 … The answer is 8926.reward 1+0.72
0005+8921: 5+1=6, 6 … The answer is 8926.reward 1+0.72
005+892: 5+2=7, 7 … The answer is 897.reward 0−1.21
0005+8921: 5+1=6, 6 … The answer is 8926.reward 1+0.72
0055+8921: 5+1=6, 6 … The answer is 8976.reward 0−1.21
0005+8921: 5+1=6, 6 … The answer is 8926.reward 1+0.72
005+892: 5+2=7, 7 … The answer is 897.reward 0−1.21

The same step has 15 other groups. 4 got 8 of 8 right (Compute 79+834), 9 got 0 of 8 (68277 + 56 = ?). Same reward for all 8 → every advantage 0 → skipped. Only 2 split, like this one.

loss = −advantage × log p(token)averaged over every token the model wrote: +0.72 pushes each right try up, -1.21 pushes each wrong one down

The SFT model tries a prompt from the first RL step 8 times. Chat data never showed it more than 3 digits, so it lines up 4-digit numbers badly: 5 + 8921 turns into 5 + 892, or 55 + 8921. The 3 wrong tries get the biggest push, down. After RL, the same prompt gets 8 out of 8.

RL hits exactly the target you give it, never the one you meant:

The grader checkedRL learned
The answer starts with "Sure!"Sure! Bye! Come back for another story soon., for every request
A correct calculator call is in the turnWrite the call, then invent the tool's result
The last number is rightWhat is 4521 + 380? → 451 + 0 = 4901: right sum, broken question

Each fix made the grader stricter.

Distillation: two teachers, one student

Practice on math alone makes a specialist, and a specialist forgets what it stopped practicing. Told to answer with the number only, it obeys 47% of the time, down from 100% after SFT. So labs now split the work: one specialist per skill, trained by RL, then a single student taught by all of them.

mini-lab has two teachers: the math specialist for sums, the SFT model for everything else. The student starts as a copy of the SFT model and answers on its own, mistakes included. The teacher reads those same tokens and grades each one. RL gives one number per answer. Distillation gives one per token.

Prompt

Compute 5+8921

The student's answer · RL reward 0

<|think_start|>0050+5892:1·5+2=7,·7↵00+89:·0+9=9,·97↵0+8:·0+8=8,·897<|think_end|>The·answer·is·897.<|assistant_end|>

loss = KL(student ‖ teacher) at every token3 tokens of 61 carry 99.6% of it

the teacher disagrees0 = the teacher's pick· = space
The student answers the prompt on its own, and lines it up wrong: 005 + 892. RL would give the whole answer one reward, 0, and push all 61 tokens down alike. The teacher reads the same tokens and grades each one. Under each flagged token, the token it would have written there. Everything after them follows from the mistake, so the teacher goes along with it.

On everything but math, the teacher is the SFT model itself. The student starts as its copy, so there the loss starts at zero, and only grows if the student drifts.

Eval: one test, every stage

Run the same questions after every stage, and you see what each one taught and what it broke.

3 digits5 digits6 digitscalcu­latorinstruc­tionsstoriesperplex­ity
Pretrain71%49%0%n/an/an/a4.93
Midtrain100%0%0%61%16%98%5.13
SFT100%0%0%70%92%93%5.74
RL100%98%0%100%94%93%5.82
Distill100%100%0%100%100%93%5.75
The same questions after every stage. Each skill turns on at the stage built for it: chat sums at midtraining, instructions at SFT, 5-digit sums at RL. The RL row is the math specialist, which never ships. Distillation keeps its sums and the SFT model's manners. 6-digit sums never turn on: no training problem ever adds a 6-digit number. (The base model can't chat yet, so its row is a different test: raw text like 347 + 58 =, answered with no scratchpad.)

The last column, perplexity on new stories, counts roughly how many tokens the model hesitates between. It rises most at SFT: learning to behave costs a little language. The specialist pays a little more, and distillation gives it back. At the end of the story above, the base model bets 99.7% on <|bos|>: a new text starts. mini-3 bets 84% on <|assistant_end|>. It expects a chat, not a new story.

Inference: one model, many users

Prefill reads the whole prompt in one step and picks the first token. Decode then adds one token per step.

Without a cache

tokens computed
prefill
Once·upon·a·time
4
decode 1
Once·upon·a·time,
5
decode 2
Once·upon·a·time,·there
6
decode 3
Once·upon·a·time,·there·was
7

With a KV cache

tokens computed
prefill
Once·upon·a·time
4
decode 1
Once·upon·a·time,
1
decode 2
Once·upon·a·time,·there
1
decode 3
Once·upon·a·time,·there·was
1
computed this stepread from the cache
Attention needs every earlier token's keys and values. Without a cache, each step computes them all again. The KV cache keeps them, so a decode step computes one token. For mini-3, 8 slots of 256 tokens fit in 25 MB.

On a model this small, a decode step costs about the same for 1 request or 8. An empty slot wastes it, so the engine keeps every slot busy.

Static batching

10 steps · 10 idle
slot 1AAAAAADDDD
slot 2BBEE
slot 3CCCFFF

Continuous batching

8 steps · 4 idle
slot 1AAAAAA
slot 2BBDDDD
slot 3CCCEEFFF
prefill: the whole promptdecode: one tokenjoins a running batchidle
A toy schedule. Rows are KV-cache slots, columns are steps. Static batching waits for the longest request before it starts the next batch. Continuous batching gives a free slot to the next request at the very next step. Same six requests, 20% fewer steps. In mini-lab a new request is prefilled and decoded in the same step, so its first step already makes two tokens. It runs 8 slots; 3 here.

The API: bill every token

The inference engine never faces the internet. A gateway stands in front of it, speaks the OpenAI API, and bills. Output costs 5 times more than input, as at the big labs, and it matches the work: the whole prompt goes through in one step, but each output token needs its own.

  1. 1Authenticatesha256(key) → org, balance
    →401
  2. 2Rate limit60 requests / min per key (default)
    →429
  3. 3Validaterequest body and model
    →400 · 404
  4. 4Check creditsbalance > 0, key under its limit
    →429
  5. 5Reserve tokens40,000 tokens / min per key (default)
    →429
  6. 6Generateinternal call to the inference engine
  7. 7Billone transaction: log + debit + key spend

"What is 12 + 30?" → scratchpad + "The answer is 42."

14 prompt tokens × $10/M + 40 output tokens × $50/M

= 140 + 2,000 = 2,140 micro-dollars

Five cheap checks can refuse a request before the model runs. Then one database transaction logs it and takes its cost from the balance. The scratchpad counts as output: thinking costs money.

Try it

git clone https://github.com/cthiriet/mini-lab && cd mini-lab
uv sync
bash speedrun.sh small   # every stage: ~31 min on an M5 Pro, ~70 min CPU-only
./scripts/serve.sh       # inference, API and platform

Or try the live demo, where billing runs in test mode. The training notes go deeper into every stage, and every reward hack.