05 - Context Engineering¶

Context engineering is the design of what information the model sees at the moment it performs a task. Prompt engineering focuses on how instructions are written; context engineering focuses on assembling the right working set of information around those instructions.

For production AI systems, this is often more important than polishing prompt wording.

Why it matters¶

An LLM can only reason over what is available in its current context. If important facts are missing, it may guess. If too much irrelevant information is included, useful signals can be diluted.

A useful mental model is:

Task
  + instructions
  + relevant conversation
  + retrieved knowledge
  + current application state
  + tool results
  + selected memory
  = model context

The goal is not to maximize context size. The goal is to maximize relevant information per token.

Context is working memory, not storage¶

A common mistake is to treat the context window like a database:

We have a 200k-token context window,
so put everything into it.

That is usually poor architecture. Durable information belongs in a source of truth such as a database, repository, object store, or knowledge system. The application selects the subset needed for the current task.

For example, this repository follows that pattern:

GitHub repository = durable source of truth
START_HERE.md      = routing information
status.md          = compact current state
specific topic     = detailed context only when needed

An assistant continuing the AI Foundations work should not need every file in the repository.

Main sources of context¶

1. System and application instructions¶

Stable behavioral rules such as:

what role the assistant performs,
allowed tools,
security constraints,
required output structure,
rules for updating data.

These are generally high-priority context.

2. Current user request¶

The immediate task should remain visible and easy to distinguish from background information.

3. Conversation history¶

Conversation history may contain useful decisions and constraints, but blindly replaying the full conversation is rarely ideal.

Possible strategies include:

last N messages,
semantic selection of relevant messages,
summarized older history,
persisted project state instead of chat replay.

4. Retrieved knowledge¶

RAG systems retrieve documents or chunks that may answer the current question.

Example:

User: How does our retry policy work?
       |
       v
Search internal documentation
       |
       v
Retrieve retry-policy.md
       |
       v
LLM receives only relevant sections

5. Application state¶

The model may need current state that was never part of the conversation:

logged-in user permissions,
active workflow step,
current shopping cart,
issue status,
project configuration,
previous tool execution result.

6. Tool results¶

Tool output is often short-lived context used to make the next decision.

LLM requests get_order(123)
        |
Application executes tool
        |
Tool result enters context
        |
LLM decides next step

7. Memory¶

Memory should be selective. Useful persisted information may include preferences, project decisions, or long-lived state. It should not mean storing and replaying every interaction forever.

Context selection¶

The application should ask:

What does the model need to know to perform this specific task?
Which source is authoritative?
What information is current versus stale?
What can be omitted?
What must never be included because of permissions or privacy?

This turns context building into an explicit pipeline rather than an accidental accumulation of text.

Example: coding assistant¶

Bad approach:

Send entire repository to the model.

Better approach:

User task
  |
  +--> repository instructions
  +--> relevant architecture document
  +--> target source files
  +--> direct dependencies
  +--> relevant tests
  +--> recent compiler/test errors

The second approach is cheaper and usually gives the model a clearer signal.

Context compression¶

When useful context is too large, compress it before sending it onward.

Techniques include:

summarization,
extracting decisions only,
removing duplicate text,
keeping only relevant sections,
converting verbose tool output into structured state.

Example:

Instead of passing a 20,000-line CI log, extract:

Failed step: integration-tests
Error: database connection timeout
First failing test: PaymentRepositoryTest
Relevant stack trace: ...

Context pollution¶

Context pollution happens when irrelevant, outdated, contradictory, or low-value information competes with the information required for the task.

Typical sources:

long unrelated chat history,
duplicated retrieved chunks,
stale project decisions,
verbose tool responses,
unrelated repository files.

More context can therefore make results worse.

Context ordering and priority¶

Even if all information fits, structure matters. Important rules and task definitions should be easy for the model to identify.

A common conceptual structure is:

Stable instructions
Current task
Relevant state
Retrieved evidence
Tool results
Output requirements

The exact API representation depends on the model provider, but the architectural principle stays the same.

Context engineering vs prompt engineering¶

Prompt engineering asks:

How should I specify the task?

Context engineering asks:

What information should be available when the task is executed?

Example:

A better prompt cannot compensate for missing product price data. If the model needs today's price, the application must retrieve it and place the result into context.

Security boundary¶

Context selection is also an authorization problem.

If a user may read Project A but not Project B, retrieval must enforce that before documents reach the model.

User request
   |
Authorization filter
   |
Permitted retrieval
   |
LLM context

Do not rely on the prompt to tell the model not to reveal information it should never have received.

Developer takeaway¶

Treat context construction as application logic with explicit inputs, ranking, filtering, authorization, token budgets, and observability.

A strong AI application does not ask only:

What prompt should we send?

It also asks:

What is the smallest, highest-quality set of information that lets the model perform this task correctly?