Prompts, context and structured outputs

What goes into the context window, in what order — and getting JSON you can trust back out.

0/6
☰
Prompts & context · architecture

The model only sees what your code packs into one request — so what goes in, in what order, what gets reused, and what shape comes back out are design decisions, not prompt tricks.

+ The whole system in one paragraph

Every call to a model is a package your backend assembles: instructions, tool definitions, retrieved documents, the conversation so far and the user's question. This page shows how that package is built, where in it facts get lost, how prompt caching reuses its stable front, how to get JSON back that always matches your schema, and how agents keep the package from overflowing.

In this lesson

6 chapters · ~9 min · a quick check in each

Everything is on this page — just scroll down ↓

New to this? Warm up first+ optional
1
At a glance

Words you'll meet

You'll meet these words all over this topic. Skim them now and come back whenever one shows up.

ⓘ Details & sources

The terms, in plain words.[1][9][14][15]

TokenA small piece of text, a short word or part ofone. Models read and are billed in tokens.Context windowEverything the model can see in one call: yourprompt plus the answer it's writing.Context rotAs the window fills, the model gets worse atfinding and using what's in it.System promptYour standing instructions for the model.OpenAI calls them developer messages.RolesWho wrote each message: you (system/developer),the user, or the model (assistant).Tool callThe model asks your code to run a function;your code runs it and replies.RAGRetrieval-augmented generation: look up theright chunks first, then answer from them.PrefixThe start of a prompt, up to some point.Caching reuses an identical one.Prompt cachingThe provider re-uses a prompt prefix it hasalready read, so repeated text costs much less.KV cacheThe provider's saved work on a prompt prefix;prompt caching reuses it.JSON schemaA description of the JSON you expect: whichfields, of which types.Structured outputsA provider feature that makes the output alwaysmatch your JSON schema.JSON modeAn older setting that guarantees valid JSON,but not JSON that matches your schema.Constrained decodingHow it works: tokens that would break theschema are blocked while the model writes.max_tokens / stop_reasonThe most tokens the answer may use / why themodel stopped (finished, hit max_tokens,refused, wants a tool).AgentA model that calls tools in a loop, choosingits own next step, until the task is done.CompactionSummarizing a long conversation and continuingfrom the summary.SubagentA separate agent with its own clean window thatdoes one job and reports back briefly.
Chapter 1 of 6

The whole picture

2
Architecture

One request, assembled

Five kinds of content flow into one request, and the model reads all of it. Each box does one job; the flows show assembling a request, a cache hit, a full window and JSON output.

ⓘ Details & sources

Your code builds the package; the provider caches its front and shapes what comes out.[1][9][14]

Watch it happen:
Model provider122334prefixrequest5schemaoutput66User: The person typing into your app.Userasks a questionInstructions: Your rules for the model: its role, its limits, the format you want.Instructionssystem · developerTool definitions: The list of functions the model may ask your code to run.Tool definitionsnames · schemasRetrieved docs: The passages your search found for this question.Retrieved docsfrom search (RAG)History: The conversation so far, including what tools returned.Historyturns · tool resultsQuestion: What the user just asked. It usually goes last.Questionthe newest turnContext builder: Your code that packs all of it into one request, in a fixed order.Context builderorder · trim · cacheCompaction: When the window gets full, old turns are replaced by a summary.Compactionsummarize when fullPrompt cache: The provider remembers the front of a prompt it already read, so a repeat is cheaper and faster.Prompt cacheprefix already readModel: The model reads the whole request and writes the answer, one token at a time.Modelreads the windowGrammar: Your JSON schema turned into rules that block any token that would break it.Grammarbuilt from schemaValidator: Your check on the result: did the model refuse, or run out of tokens?Validatorrefusal · cut-offYour code: The rest of your app, which uses the parsed result.Your codetyped JSON
youyour systemoutside services dashed = our simplification
123456
Step 1 of 6

The user asks a question. That's the only part of the request they write.

Quick check

Your app has one text box. What does the model actually see?

Pick one — or just keep scrolling, it's not graded.

Keep going — it's free

Create your free account to unlock the rest of this lesson, keep your progress and pick up right where you left off.

  1. 2What goes in, in what order
  2. 3When the window fills up
  3. 4Prompt caching
  4. 5Structured outputs
  5. 6Context engineering for agents
Create my free account →I already have one — log in