How an LLM works, for engineers

Tokens, the context window, sampling and why models make things up — the mental model every other topic builds on.

0/3
☰
How LLMs work · architecture

An LLM does one thing, over and over: it reads everything in its window and guesses the next token. Everything else — chat, tools, 'reasoning' — is built on that loop.

+ The whole system in one paragraph

A large language model turns your text into tokens, scores every possible next token, picks one, adds it to the text and repeats. It only sees what's in its context window, remembers nothing between calls, and was trained to predict likely text — which is also why it can state wrong facts with confidence.

In this lesson

3 chapters · ~6 min · a quick check in each

Everything is on this page — just scroll down ↓

New to this? Warm up first+ optional
1
At a glance

Words you'll meet

Skim these now and come back whenever one shows up. Every other AI-engineering topic uses them.

ⓘ Details & sources

The terms, in plain words.[1]

TokenA small piece of text, a short word or part ofone. Models read and are billed in tokens.Context windowEverything the model can see in one call: yourprompt plus the answer it's writing.Parameters (weights)The numbers the model learned in training. Theydon't change when you use it.PretrainingTraining on a huge amount of text to predictthe next word.Fine-tuningExtra training on chosen examples to change howthe model behaves.RLHFTraining from human rankings of answers, so themodel prefers the better-ranked kind.TemperatureHow adventurous the pick of the next token is:low = safe, high = surprising.StatelessIt remembers nothing between calls; your appresends the conversation each time.HallucinationA confident answer that is wrong or made up.RAGFetch the relevant documents and put them inthe prompt so it answers from them.EvalsAutomated tests that grade the model's answersbefore you ship.
Chapter 1 of 3

The big picture

2
Pipeline

From your text to the answer

Start here: the whole machine in five steps. The model never writes a sentence at once — it guesses one token, adds it, and reads everything again for the next one.

ⓘ Details & sources

Read everything, guess one token, append, repeat.[1]

Your textthe whole prompt1tokenizeTokenstext → numbers2the modelScoreevery next token3samplePick onetemperature4loopAppend, repeatuntil it stops5

5 steps. Follow the flow one step at a time — what happens first, and what comes next.

3
Up close

Try it: the temperature dial

Cold, it says 'Paris' every time; hot, the long shots start winning. The numbers are an example; the reshaping is the real math. Even at 0, real APIs aren't fully repeatable.

ⓘ Details & sources

Same scores, different picks: temperature reshapes the odds before one token is chosen.[1]

"The capital of France is …"the model scores every next token (example numbers)Paris86%a5%the4%known3%beautiful2%temperature 1.0 · mostly the top pickPress 'Generate 10' to sample.
4
Up close

What the model actually reads

The model never sees letters or words, only token numbers. That's why prices and limits are in tokens, and why the same text counts differently on different models. The split shown is illustrative.

ⓘ Details & sources

Tokens, not words: common words stay whole, rare ones split.[1]

What the model actually reads"Tokenization isn't magic." → pieces, then numbersToken#4421ization#2065␣isn#3721't#59␣magic#11204.#136 tokens for 3 words and a period (␣ = a space)Illustrative split and numbers · (tap to step)
5
Up close

What it can see: the context window

Each call, everything must fit in one window: instructions, documents, the chat, the question and the answer it writes. More isn't always better: recall drops as it fills.

ⓘ Details & sources

Outside the window, the model knows nothing about your conversation.[2]

The model's context windoweverything it can see in one callRulesDocumentsChatQuestionroom for the answerToo much in? Something has to leave.It remembers nothing: your app resends it every turn.(tap to step)
Curious? Go deeper+ 2 optional
6
Architecture

One call, piece by piece

The same loop as boxes. The only part that learned anything is the model's weights, frozen after training; everything else is plumbing around it.

ⓘ Details & sources

Click a piece; play 'One token at a time'.[1][2][3]

Watch it happen:
12235Your app: Your code: it decides what goes into the prompt and sends it.Your appbuilds the promptTokenizer: Cuts the text into tokens and turns each into a number.Tokenizertext → token numbersThe model: The trained network: reads all the tokens and gives a probability to every possible next one.The modelscores every next tokenSampler: Picks the next token from those probabilities; temperature decides how adventurous.Samplerpicks one (temperature)The answer: Each picked token is turned back into text and streamed to you.The answertoken by tokenWhat it learned: Everything the model learned in training, stored as numbers. It doesn't change when you use it.What it learnedfrozen after training
youyour system dashed = our simplification
12345
Step 1 of 5

Your app sends the whole prompt; the tokenizer cuts it into token numbers.

7
Sequence

Every call starts from zero

The model keeps no memory between calls. To 'remember' a conversation, your app sends the whole history again every turn — which is why long chats get slower and pricier.

Your appThe model1. Turn 1: questionThe modelYour app2. Answer 1Your appThe model3. Turn 2: Q1 + A1 + Q2The modelYour app4. Answer 2

4 steps. Follow the flow one step at a time — what happens first, and what comes next.

Quick check

Your chatbot's second answer ignores what the user said in their first message. What's the most likely cause?

Pick one — or just keep scrolling, it's not graded.

Keep going — it's free

Create your free account to unlock the rest of this lesson, keep your progress and pick up right where you left off.

  1. 2How it got this way
  2. 3Why it makes things up
Create my free account →I already have one — log in