Context Architecture¶

Context architecture is the design of what information reaches the model, in what form, from which sources, with what trust level and under what budget.

A context window is not a database, memory system or source of truth. It is a temporary working set assembled for one model decision.

A useful mental model is:

Canonical Sources
  ├── run state
  ├── user/task input
  ├── policies
  ├── skill definition
  ├── retrieved evidence
  ├── tool observations
  └── memory
        ↓
Context Builder
  ├── select
  ├── normalize
  ├── rank
  ├── compact
  ├── label trust/provenance
  └── fit budget
        ↓
Model Context

The context builder is therefore closer to a compiler than to a string concatenation helper.

Context is a projection, not canonical state¶

The runtime state may contain far more information than the model needs.

Canonical Run State
        ↓ projection
Current Model Context

The projection may include:

current goal,
relevant success conditions,
current plan fragment,
recent authoritative observations,
available capabilities,
selected business rules,
retrieved evidence,
unresolved questions.

It should not automatically include every historical message, tool result and artifact.

Context layers¶

A useful conceptual stack is:

1. platform/system constraints
2. application policy/instructions
3. active skill instructions
4. task and goal
5. canonical state projection
6. observations and evidence
7. relevant memory
8. user interaction/history needed now

The exact API representation differs by provider, but the architecture should know which layer owns which information.

Stable instructions vs runtime data¶

Separate relatively stable behavior from changing task data.

Stable:

application role,
hard interaction rules,
skill contract,
output schema guidance,
domain terminology.

Dynamic:

current customer/order,
latest tool result,
current plan,
retrieved documents,
remaining budget,
current failure.

Do not rewrite stable system behavior into every piece of retrieved text or user content.

Context ownership¶

A dedicated ContextBuilder or equivalent component should own assembly rules.

Example interface:

class ContextBuilder(Protocol):
    def build(self, run: AgentRun, purpose: ContextPurpose) -> ModelContext: ...

The purpose matters because planning, tool selection, verification and summarization may require different projections.

PLANNING context
  goal + constraints + coarse state

ACTION_SELECTION context
  current state + fresh observations + capabilities

VERIFICATION context
  claimed result + evidence + success conditions

One giant universal context template usually becomes noisy and expensive.

Context as a budgeted resource¶

A larger context is not automatically better.

Costs include:

more input tokens,
higher latency,
more irrelevant distractors,
increased chance of conflicting instructions,
harder provenance tracking,
stale information remaining visible.

Think in terms of a budget:

context_budget
├── instructions
├── task/state
├── evidence
├── tool schemas
└── reserve for output

The builder should know approximate priorities and truncation policy before the provider rejects the request.

Relevance over chronology¶

Chat-style applications often default to chronological history. Agent runtimes usually need relevance-oriented state projection.

Weak:

last 100 messages

Better:

current goal
current plan step
fresh relevant observations
required constraints
selected evidence
important unresolved facts

Chronology can still matter, but it is not the primary organizing principle for many agent decisions.

Observation normalization¶

Raw tool output should usually be normalized before being inserted into model context.

Raw:

{
  "status": 200,
  "headers": {...},
  "internal_metadata": {...},
  "payload": {...}
}

Model-facing observation:

{
  "source": "order_service",
  "observed_at": "...",
  "order_id": "123",
  "payment_status": "PAID",
  "shipment_status": "NOT_SHIPPED"
}

Keep the raw artifact separately if it is needed for audit/debugging.

The normalized form should preserve the facts necessary for reasoning without flooding context with transport noise.

Fact vs hypothesis¶

The context should make it possible to distinguish:

FACT
  authoritative observation from a source

INFERENCE
  interpretation derived from facts

HYPOTHESIS
  unverified assumption used for planning

If these blur together, the model may treat its own earlier guesses as authoritative state.

A structured record can help:

{
  "claim": "customer may be eligible for refund",
  "kind": "hypothesis",
  "evidence_refs": ["obs-41", "policy-7"]
}

Provenance¶

Evidence should carry source information.

Useful metadata:

source_id
source_type
retrieved_at / observed_at
tenant / access scope
trust classification
content version
citation/artifact reference

Provenance enables:

grounding,
audit,
freshness checks,
tenant isolation,
verification,
re-fetching canonical data.

Freshness¶

Some facts decay quickly.

Examples:

repository HEAD      changes often
order status         may change
flight status        changes
policy document      versioned, slower
static product spec  relatively stable

The runtime should define freshness policy by source/capability.

cached observation
      ↓
older than freshness threshold?
      ↓ yes
re-observe canonical source

Do not leave freshness entirely to model judgement when code can enforce it.

Trusted vs untrusted context¶

Not all tokens are equally trustworthy.

Typical sources:

HIGHER TRUST
  system/application policy
  signed/internal configuration
  authoritative structured service data

VARIABLE TRUST
  internal documents
  retrieved knowledge
  external webpages
  emails/messages
  user-uploaded files

UNTRUSTED INSTRUCTION SOURCE
  arbitrary retrieved text claiming "ignore previous instructions"

Retrieved content may contain instructions, but those instructions should be treated as data, not as authority over the runtime.

This is one of the core defenses against indirect prompt injection.

Data trust vs instruction authority¶

A subtle but important distinction:

A source may be trusted as data without being trusted to issue instructions.

Example:

GitHub README

It may be authoritative documentation for the repository, but text inside it should not automatically be allowed to redefine agent permissions.

Represent these dimensions separately where useful:

content_trust = internal_repository
instruction_authority = none

Retrieval as context acquisition¶

RAG is one way to acquire relevant context.

question / state
   ↓
retrieval query
   ↓
search
   ↓
rerank/filter
   ↓
evidence
   ↓
context builder

The context builder should not treat top-k retrieval results as inherently correct or relevant.

Possible filters:

tenant authorization,
document version,
freshness,
source type,
relevance score,
duplication,
trust classification.

Tool results as context acquisition¶

Tool calling is another context source.

Difference:

Retrieval
  finds stored knowledge/evidence

Tool observation
  queries current operational state or performs action

For a question such as "Has invoice 42 been paid?", the current billing service is usually stronger evidence than an embedded document snapshot.

Memory as optional context source¶

Persistent memory should not be dumped wholesale into the context.

Memory Store
   ↓ relevant memory retrieval
Context Builder

Memory records should have provenance, timestamps and confidence/ownership where appropriate.

Remembered user preference and current authoritative account state are different categories and should not be conflated.

Summarization and compaction¶

When context grows, summarize lower-value history while preserving canonical records separately.

Bad:

summarize everything
replace original state

Better:

canonical state/artifacts remain stored
         ↓
context-specific summary generated
         ↓
used only as projection

A summary is a lossy representation and should not silently become the source of truth.

What should be re-fetched instead of summarized?¶

Prefer re-fetching when the data is:

operational/current,
cheap to query,
safety-critical,
authorization-sensitive,
likely to have changed.

Examples:

current bank balance,
current order state,
current repository branch HEAD,
current user permissions.

Prefer summarization for:

long discussion history,
completed plan segments,
large evidence sets whose raw artifacts remain available,
previous reasoning outcomes that are not authoritative external facts.

Context versioning and reproducibility¶

For debugging, record enough metadata to reconstruct what the model saw.

Possible trace fields:

context_template_version
skill_version
policy_version
state_version
observation_ids
retrieval_result_ids
model configuration

You may not store every rendered prompt forever, especially with sensitive data, but you need a strategy for reproducibility and privacy.

Context caching¶

Caching can reduce cost and latency for stable prefixes or repeated retrieval.

But cached context must respect:

tenant isolation,
authorization changes,
source version,
expiry/freshness,
privacy requirements.

Never reuse a context fragment across tenants merely because the text looks identical unless its access scope is truly shared.

Example: code review agent¶

Possible context for one step:

SYSTEM
  review policy + security constraints

SKILL
  PR review instructions + finding schema

TASK
  review PR #123 for correctness issues

STATE
  files reviewed: 8/11
  open findings: 2

OBSERVATIONS
  current diff for file X
  failing test Y

TOOLS
  read_file
  fetch_diff
  run_tests

BUDGET
  6 tool calls remaining

It does not need every file in the repository, every previous model response and every CI log on each step.

Context anti-patterns¶

Dump everything¶

More tokens are treated as more intelligence.

Chat history as state¶

The runtime expects the model to infer canonical progress from messages.

Unlabelled evidence¶

Retrieved text has no source, time or trust metadata.

Stale operational data¶

Old tool results remain in context and are used for irreversible actions.

Summary becomes truth¶

A lossy model summary replaces canonical state.

Retrieval can override policy¶

External documents are inserted at the same instruction authority as application rules.

One context template for every reasoning task¶

Planning, acting and verification receive the same bloated prompt.

Practical context-building pipeline¶

1. Load canonical run state
2. Determine context purpose
3. Select required instructions/policy
4. Select relevant state projection
5. Refresh stale critical observations
6. Retrieve optional evidence/memory
7. Apply authorization/trust filters
8. Normalize and deduplicate
9. Rank by relevance/priority
10. Compact to token budget
11. Attach provenance metadata
12. Invoke model

Takeaways¶

Context is a temporary projection, not canonical state.
Treat context construction as an architectural subsystem, not string concatenation.
Build different projections for planning, action selection and verification.
Prefer relevance and freshness over simply replaying chronological history.
Distinguish facts, inferences and hypotheses.
Preserve provenance and trust classification.
Treat retrieved/external instructions as untrusted data unless explicitly authorized.
Re-fetch current authoritative state when correctness matters more than saving a tool call.
Keep raw/canonical artifacts outside summaries so context can stay compact without losing truth.