The architecture of an AI app

Every piece between a user's question and a reliable answer: API, orchestration, retrieval, tools, the model gateway, streaming, evals and observability.

See it in a real systemNetflix →Cloudflare →
0/4
☰
AI app · architecture

An AI app isn't one model call: it's a backend that finds the right context, lets the model use tools, streams the answer, survives a provider going down — and measures every step.

+ The whole system in one paragraph

The model is the smallest part. Around it sits an ordinary backend with a few new jobs: finding the right information to show the model, letting it use your tools, handling a provider that fails, and measuring every step. This page builds the picture one piece at a time — start with the simplest version that works and add a piece only when it earns its place.

In this lesson

4 chapters · ~9 min · a quick check in each

Everything is on this page — just scroll down ↓

New to this? Warm up first+ optional
2
At a glance

Words you'll meet

You'll meet these words on every page of this topic. Skim them now and come back whenever one shows up.

ⓘ Details & sources

The terms, in plain words.

TokenA small piece of text, a short word or part ofone. Models read and are billed in tokens.Context windowEverything the model can see in one call. Itremembers nothing outside it.EmbeddingA list of numbers that captures a text'smeaning; similar meanings get similar numbers.ChunkA small piece of a document, cut so each one isabout a single thing.RAGRetrieval-augmented generation: look up theright chunks first, then answer from them.Re-rankerA second model that re-orders search results byhow well they answer the question.Tool callThe model asks your code to run a function;your code runs it and replies.Streaming (SSE)The answer arrives in small pieces as it iswritten, instead of all at once.OrchestratorYour code that builds each prompt, decides whatto fetch or call, and runs the loop.BM25Classic keyword search: it finds exact wordsand phrases that meaning-based search can miss.Prompt cachingThe provider re-uses a prompt prefix it hasalready read, so repeated text costs much less.GatewayOne door between your app and the models,handling retries, fallbacks and spend.GuardrailsChecks on what goes in and out, against attackslike prompt injection (text that tries to givethe model orders).EvalsTests for AI answers: a set of real cases,graded automatically or by people, before eachchange ships.Trace and spanA trace is one request's full record; each stepinside it (search, model call, tool) is a span.Backoff and jitterWaiting longer after each failure, plus arandom bit so clients don't all retry together.
1
Requirements

What it must do — and how well

Yellow notes are what the app must do; blue ones are how well. A demo needs little more than a prompt. Production needs all of them, and each one adds a box to the architecture.

Answer from YOURdata, not justmemory.Use tools: lookthings up, takeactions.Stream the answer asit's written.Survive a providerbeing down.Grounded: answerscome from yourdocuments, notguesswork.Safe: untrusted textcan't take over.Measured: changespass evals first.Observable: everystep is traced.
Chapter 1 of 4

The whole system

3
Up close

What the model actually sees

Every call, your code packs one window: instructions, found documents, the chat and the question. What doesn't fit is invisible — so most of an AI app is deciding what goes in.

The model's context windoweverything it can see in one callRulesDocumentsChatQuestionroom for the answerToo much in? Something has to leave.It remembers nothing: your app resends it every turn.(tap to step)
4
Pipeline

Build it up, one piece at a time

The full architecture has eleven boxes. You never build them all at once — each one answers a problem the previous version couldn't.

ⓘ Details & sources

Start at step 1. Move on only when something real forces you to.[1]

1 · One callprompt → model1needs facts2 · + Your dataretrieval2needs actions3 · + Toolslook up, take actions3goes to production4 · + A safety netgateway · guardrails · traces4

4 steps. Follow the flow one step at a time — what happens first, and what comes next.

5
Sequence

One question, end to end

Follow one question in time order. Retrieval happens before the model is called, and the answer streams back in small pieces.

Your appYour backend1. QuestionYour backendRetrieval2. Search my documentsRetrievalYour backend3. Top chunksYour backendModel4. Prompt + chunks (stream)ModelYour backend5. Text, piece by piece (SSE)Your backendYour app6. Forward each piece

6 steps. Follow the flow one step at a time — what happens first, and what comes next.

6
Architecture

An AI app, on one page

A common reference architecture. The model is on the far right — most of the work happens before and after it.

ⓘ Details & sources

Don't memorise it. Click a box, play a flow, and see where each piece sits.[1][2][4][6]

Watch it happen:
Your backend12query33tool call44checkspansYour app: What your users open: a website or a phone app.Your appweb · mobileAPI: The front door: checks who is asking and how much they may use.APIauth · rate limitsOrchestrator: The conductor: decides what to look up, what to call, and what to ask the model.Orchestratorprompt · tools · loopSearch index: Your documents, filed so the right passage can be found fast.Search indexvectors + keywordsRetrieval: The librarian: fetches the few passages that matter for this question.Retrievalsearch · re-rankTools: Your own code the model can ask to run, like 'look up this order'.Toolsyour functions · APIsTraces & evals: A logbook of every step, plus test scores, so you can see what went wrong.Traces & evalsspans · scoresModel gateway: One door to the AI models: retries when one fails and keeps track of spend.Model gatewayretry · fallbackGuardrails: Safety checks on what goes in and what comes out.Guardrailsinput · output checksPrimary model: The AI model your app normally asks, run by another company.Primary modelprovider AFallback model: A second model to use when the first one is down.Fallback modelprovider B
youyour systemoutside services dashed = our simplification
12345
Step 1 of 5

The user asks a question. The API checks who they are and that they're within their limits.

Quick check

You're starting a new AI feature. According to this page, what should version 1 be?

Pick one — or just keep scrolling, it's not graded.

Keep going — it's free

Create your free account to unlock the rest of this lesson, keep your progress and pick up right where you left off.

  1. 2Retrieval
  2. 3Tools & agents
  3. 4Production
Create my free account →I already have one — log in