How an AI Lab Works

How a modern AI lab works, from raw text to a billed API call, in diagrams. Every training example is real, taken from mini-lab.

Clément Thiriet • September 26, 2026

View on GitHub

mini-lab in five questions

What is it? An open-source AI lab, small enough to read. About 6,500 lines of Python train a language model from scratch, serve it behind an OpenAI-compatible API, and bill every token from prepaid credits.

Why build it? A frontier lab is too big to see, and most tutorials stop once the model trains. mini-lab keeps going to the API and the bill, so you can follow the whole chain in one repo.

What can its model do? mini-1 has 5.8M parameters. It tells short children's stories and adds numbers, step by step or with a calculator. It knows nothing else, and says so. It is tiny on purpose: every training stage moves a number you can see.

What do I need? A laptop, no GPU. The whole training takes about 16 minutes on an Apple M5 Pro, or 36 on its CPU alone.

Can I try it? Yes: on the live demo, or on your laptop with four commands (at the end of this post).

The mini-lab chat app: a short story about a dog, then 347 + 58 solved with the calculator tool
The mini-lab chat app. mini-1 tells a story, then calls the calculator for a sum. Every reply is billed to the lab's prepaid credits, top right.

The chain

Every lab, big or small, runs the same chain. The rest of this post walks it, one diagram per step.

Research · once per model

Data
stories, sums
→
Tokenizer
text → ids
→
Pretrain
language
→
Midtrain
chat format
SFT
behavior
→
RL
practice
→
Eval
after each
→
Release
mini-1
↓ the inference engine loads the checkpoint

Serving · every request

Your code
openai SDK
sk-mini-…→
API gateway
auth · limits · billing
internal→
Inference
KV cache · batching
Research runs once and ends in a checkpoint: the model's weights, in one file. Serving loads that file and turns each request into an answer and a bill.

Tokens

A model reads numbers, not letters. The tokenizer has 4,096 pieces of text, called tokens, most of them learned from the data, and turns any text into their ids.

41 characters → 11 tokens

Once432·upon441·a258·time401,44·there403·was282·a258·little391·cat466.46

17 characters → 11 tokens

What629·is371·32351452755·+413·32553856?63
· = spacenumber = token id
Common words take one token each. Digits never merge, so 347 always splits into 3, 4, 7: the model can add column by column, but a sum costs one token per digit.

Pretraining: guess the next token

The first training stage reads 29M tokens of stories and sums, and learns one skill: predict the next token.

Story (shortened)

<|bos|>Once upon a time, there was a small, brave dog named Spot. Spot loved to play and run outside. […]

Worksheet

<|bos|>5 + 625 005+625: 5+5=10, 0 00+62: 0+2+1=3, 30 0+6: 0+6=6, 630 So 5 + 625 = 630. The sum of 3 and 0 is 3.
Three documents in four are stories, one in four a sum worksheet. A worked sum adds right to left, one column per line, and the number at the end of each line grows into the answer. <|bos|> marks where each document starts. Documents sit back to back in rows of 256 tokens: no padding, no waste.
·played
↓
·together
44%loss 0.82
·together
↓
.
13%loss 2.01
.
↓
<|bos|>
98%loss 0.02
<|bos|>
↓
Once
54%loss 0.62
Once
↓
·upon
97%loss 0.03
·upon
↓
·a
>99%loss 0.00
·a
↓
·time
>99%loss 0.00
·time
↓
,
94%loss 0.07

After …they even played together, the model's top guesses:

·in34%
.13%
·every12%

right token: . → loss = −log 0.134 = 2.01

top: what the model readswhat it must predict· = space
The target is the input shifted by one token. At every position the model gives each of the 4,096 tokens a probability, and the loss is −log of the one it gave the right token: near 0 for a safe bet, large for a bad one. Training lowers the average.

After 3,500 steps the base model writes little stories. It can't chat: it has never seen a conversation.

Midtraining: learn the chat format

Same loss, new documents: chats, mixed with old text so the model keeps its language. The model learns the format here, from a lot of data, so the next stage needs only a little.

A calculator call · 50 tokens

<|bos|><|system_start|>tools:·calculator<|system_end|><|user_start|>Add·6·and·3.<|user_end|><|assistant_start|><|tool_call_start|>{"name":·"calculator",·"arguments":·{"expression":·"6·+·3"}}<|tool_call_end|><|assistant_end|><|tool_start|>9<|tool_end|><|assistant_start|>The·answer·is·9.<|assistant_end|>
in the losscontext only· = space
Every token counts in the loss, even the user's question and the calculator's result. Special tokens mark who speaks. After <|tool_call_end|> the model stops, your code runs the calculator, and its result comes back between <|tool_start|> and <|tool_end|>.

SFT: learn to behave

SFT (supervised fine-tuning) uses a small, fixed set of 8,000 conversations with what midtraining never shows: system prompts, follow-ups, polite ways to say no. The loss changes too: it now skips every token the model will never have to write.

System prompt · loss on 17 of 33 tokens

<|bos|><|system_start|>Answer·with·the·number·only.<|system_end|><|user_start|>2·+·5<|user_end|><|assistant_start|><|think_start|>2+5:·2+5=7,·7<|think_end|>7<|assistant_end|>

Follow-up · loss on the last answer only (22 of 47)

<|bos|><|user_start|>Add·4·and·2.<|user_end|><|assistant_start|>The·answer·is·6.<|assistant_end|><|user_start|>Plus·2?<|user_end|><|assistant_start|><|think_start|>6+2:·6+2=8,·8<|think_end|>The·answer·is·8.<|assistant_end|>
in the losscontext only· = space
The loss covers the answer only, from the scratchpad to <|assistant_end|>. The scratchpad is not the answer: the API returns it on the side, so “the number only” still holds. In a follow-up, even the earlier answer only counts as context: to read, not to copy.

RL: practice with a grader

SFT can only copy answers someone wrote. RL (reinforcement learning) needs no answers, only a program that checks them. mini-lab uses GRPO: the model tries each problem 8 times, the program grades every try, and each try gets pushed up or down by how it compares to the group.

Prompt

Add 3183 and 5.

All 8 attempts start with the same scratchpad, 3183+0005: 3+5=8, 8 … 3+0: 3+0=3, 3188, then:

888888970<|assistant_end|>reward 0−0.35
<|think_end|>The answer is 3188.<|assistant_end|>reward 1+2.47
888888883. The answer is 3188 and 9851.<|assistant_end|>reward 0−0.35
84818888: 2+8=10, 03188… The answer is 1418.<|assistant_end|>reward 0−0.35
818888<|assistant_end|>reward 0−0.35
858888917181… The answer is 3419.<|assistant_end|>reward 0−0.35
887888809leaseThe answer is 3188.<|assistant_end|>reward 0−0.35
888888986908<|assistant_end|>reward 0−0.35

The other 15 groups of the same step: 9 got 8 of 8 right (Compute 79+834), 6 got 0 of 8 (68277 + 56 = ?). Same reward for all 8 → every advantage 0 → skipped.

loss = −advantage × log p(token)averaged over every token the model wrote: +2.47 pushes the right try up, -0.35 pushes each wrong one down

One group from the first RL step. All 8 tries find 3188 in the scratchpad, but only one closes it with <|think_end|>. The other 7 score 0, even when the right answer sits inside the scratchpad. The one that stops gets a strong push up. After RL, the same prompt gets 8 out of 8.

RL hits exactly the target you give it, never the one you meant:

The grader checkedRL learned
The answer starts with "Sure!"Sure! Bye! Come back for another story soon., for every request
A correct calculator call is in the turnWrite the call, then invent the tool's result
The last number is rightWhat is 4521 + 380? → 451 + 0 = 4901: right sum, broken question

Each fix made the grader stricter.

Eval: one test, every stage

Run the same questions after every stage, and you see what each one taught and what it broke.

3 digits5 digits6 digitscalcu­latorinstruc­tionsstoriesperplex­ity
Pretrain4%2%0%n/an/an/a6.2
Midtrain100%0%0%65%15%96%6.3
SFT100%0%0%66%99%93%7.1
RL100%91%0%97%98%87%7.4
The same questions after every stage. Each skill turns on at the stage built for it: chat sums at midtraining, instructions at SFT, 5-digit sums at RL. 6-digit sums never do: no training problem ever adds a 6-digit number. (The base model can't chat yet, so its row is a different test: raw text like 347 + 58 =, answered with no scratchpad.)

The last column, perplexity on new stories, counts roughly how many tokens the model hesitates between. It rises mostly during SFT and RL: learning to behave costs a little language. At the end of the story above, the RL model bets 99.6% on <|assistant_end|>. It expects a chat, not a new story.

Inference: one model, many users

Prefill reads the whole prompt in one step and picks the first token. Decode then adds one token per step.

Without a cache

tokens computed
prefill
Once·upon·a·time
4
decode 1
Once·upon·a·time,
5
decode 2
Once·upon·a·time,·there
6
decode 3
Once·upon·a·time,·there·was
7

With a KV cache

tokens computed
prefill
Once·upon·a·time
4
decode 1
Once·upon·a·time,
1
decode 2
Once·upon·a·time,·there
1
decode 3
Once·upon·a·time,·there·was
1
computed this stepread from the cache
Attention needs every earlier token's keys and values. Without a cache, each step computes them all again. The KV cache keeps them, so a decode step computes one token. For mini-1, 8 slots of 256 tokens fit in 25 MB.

On a model this small, a decode step costs about the same for 1 request or 8. An empty slot wastes it, so the engine keeps every slot busy.

Static batching

10 steps · 10 idle
slot 1AAAAAADDDD
slot 2BBEE
slot 3CCCFFF

Continuous batching

8 steps · 4 idle
slot 1AAAAAA
slot 2BBDDDD
slot 3CCCEEFFF
prefill: the whole promptdecode: one tokenjoins a running batchidle
A toy schedule. Rows are KV-cache slots, columns are steps. Static batching waits for the longest request before it starts the next batch. Continuous batching gives a free slot to the next request at the very next step. Same six requests, 20% fewer steps. In mini-lab a new request is prefilled and decoded in the same step, so its first step already makes two tokens. It runs 8 slots; 3 here.

The API: bill every token

The inference engine never faces the internet. A gateway stands in front of it, speaks the OpenAI API, and bills. Output costs 3 times more than input, which matches the work: the whole prompt goes through in one step, but each output token needs its own.

  1. 1Authenticatesha256(key) → org, balance
    →401
  2. 2Rate limit60 requests / min per key (default)
    →429
  3. 3Validaterequest body and model
    →400 · 404
  4. 4Check creditsbalance > 0, key under its limit
    →429
  5. 5Reserve tokens40,000 tokens / min per key (default)
    →429
  6. 6Generateinternal call to the inference engine
  7. 7Billone transaction: log + debit + key spend

"What is 12 + 30?" → scratchpad + "The answer is 42."

14 prompt tokens × $0.50/M + 40 output tokens × $1.50/M

= 7 + 60 = 67 micro-dollars

Five cheap checks can refuse a request before the model runs. Then one database transaction logs it and takes its cost from the balance. The scratchpad counts as output: thinking costs money.

Try it

git clone https://github.com/cthiriet/mini-lab && cd mini-lab
uv sync
bash speedrun.sh small   # every stage: ~16 min on an M5 Pro, ~36 min CPU-only
./scripts/serve.sh       # inference, API and platform

Or try the live demo, where billing runs in test mode. The training notes go deeper into every stage, and every reward hack.