12 – Cost and Latency¶

AI applications have a runtime cost profile that is very different from normal application code.

For many systems, the expensive part is not CPU time in your own service but external model inference.

The basic mental model is:

request volume
×
input tokens
×
output tokens
×
model price

This is simplified, but it is enough to understand why architecture decisions directly affect cost.

Token-based cost¶

Model APIs commonly charge based on input and output tokens.

Conceptually:

cost = input_tokens × input_price
     + output_tokens × output_price

Example:

input:  8,000 tokens
output: 1,000 tokens

If the application sends the same large context on every request, cost grows quickly even when the answer itself is short.

This is why context engineering is also cost engineering.

More context is not free¶

Suppose a support assistant sends:

system prompt          2,000 tokens
conversation history   8,000 tokens
retrieved documents   20,000 tokens
user question            100 tokens

Total input:

30,100 tokens

If only 3,000 tokens were actually relevant, the system pays for unnecessary context and may also reduce answer quality.

Therefore:

better retrieval
+ better summarization
+ better context selection
=
lower cost and often better quality

Output tokens can be expensive too¶

A model that produces long answers consumes more time and tokens.

If the application needs only:

{
  "category": "billing"
}

then asking for an explanation, reasoning summary, and detailed prose wastes both latency and cost.

Match output length to the actual product need.

Cost per request is not enough¶

Production cost depends on volume.

Example:

$0.01 per request
× 100 requests/day
= $1/day

But:

$0.01 per request
× 10,000,000 requests/month
= $100,000/month

Small per-request inefficiencies become important at scale.

Model selection affects cost¶

Not every task needs the strongest model.

Possible routing:

simple classification → small model
complex reasoning      → stronger model
embedding              → embedding model
image understanding    → multimodal model

A common architecture is:

cheap model first
      ↓
can solve?
  ├── yes → return
  └── no  → escalate to stronger model

But routing itself must be evaluated. A cheap first step is not useful if it frequently makes bad decisions or adds unnecessary latency.

Latency has multiple components¶

Total user-visible latency may include:

network
+ model queueing
+ model inference
+ retrieval
+ tool calls
+ retries
+ application processing

A useful decomposition is:

request
 ↓
retrieval      150 ms
 ↓
model call    1800 ms
 ↓
tool call      400 ms
 ↓
second model  1600 ms
 ↓
response

Total is not just the first model call.

Time to first token versus total time¶

For streaming responses, two latency measures matter.

Time to first token¶

How long until the user sees the response begin.

Time to completion¶

How long until the full answer is generated.

Example:

time to first token: 700 ms
full completion:      6.2 s

A streaming UI can feel responsive even when total generation takes several seconds.

Streaming¶

Streaming returns output incrementally.

model generates
   ↓
token chunks
   ↓
UI displays progressively

Streaming improves perceived latency but does not necessarily reduce compute cost or total completion time.

It is mostly a user-experience technique.

Parallelism¶

Independent operations can sometimes run in parallel.

Sequential:

retrieve docs     500 ms
 ↓
get user profile  400 ms
 ↓
get permissions   300 ms

Total before LLM ≈ 1200 ms

Parallel:

retrieve docs ───────┐
get profile ─────────┼─→ combine
get permissions ─────┘

Total may approach the slowest operation instead of the sum.

Do not parallelize operations that depend on each other's results.

Agentic loops multiply latency¶

A normal call:

user → LLM → answer

An agentic flow may be:

LLM
 ↓
tool
 ↓
LLM
 ↓
tool
 ↓
LLM
 ↓
answer

If each model call takes 2 seconds and each tool takes 500 ms:

3 model calls = 6.0 s
2 tool calls  = 1.0 s
---------------------
minimum       = 7.0 s

before network overhead and retries.

Agentic flexibility has real latency cost.

Agentic loops multiply monetary cost too¶

One user request may trigger several model calls.

1 user request
 ↓
planner model
 ↓
tool selection
 ↓
result interpretation
 ↓
final generation

Therefore cost should be tracked per task, not only per individual model call.

Useful metric:

total tokens and cost per completed user task

Caching¶

Caching can reduce both cost and latency when requests or intermediate results repeat.

Possible cache targets:

embeddings,
document parsing,
retrieval results,
deterministic tool results,
model responses where safe,
stable prompt prefixes if the provider/platform supports it.

Be careful with:

user-specific data,
authorization,
rapidly changing information,
stale model outputs.

Caching is a correctness decision, not only a performance decision.

Embedding cost¶

RAG systems have additional cost components.

Ingestion:

documents
 ↓
chunking
 ↓
embedding generation
 ↓
vector storage

Query time:

query embedding
+ vector search
+ reranking
+ generation

Embedding cost is often much lower than generation cost, but large-scale ingestion and frequent re-indexing can still matter.

Reranking has a cost trade-off¶

A retrieval pipeline might be:

vector search → 50 candidates
      ↓
reranker → top 5
      ↓
LLM

The reranker adds latency and cost but may allow the final LLM context to be smaller and more relevant.

A more expensive retrieval stage can therefore reduce total system cost if it significantly improves downstream efficiency.

Retries can silently multiply cost¶

Suppose one request costs $0.02.

With automatic retries:

attempt 1 → timeout
attempt 2 → malformed result
attempt 3 → success

Actual cost may approach three model calls, not one.

Track retry counts and total task cost.

Long prompts become infrastructure¶

A large system prompt repeated millions of times is not just documentation.

It is recurring runtime cost.

Example:

system prompt: 4,000 tokens
× 1,000,000 requests
=
4 billion input tokens

This does not mean prompts should always be short. It means prompt size has an economic consequence.

Cost budgets¶

Agentic applications should have explicit budgets.

Examples:

max 8 model calls per task
max 20 tool calls
max 50,000 total tokens
max $0.50 per task
max 30 seconds wall-clock time

If the system reaches a limit:

stop
 ↓
return partial result / ask user / escalate

This converts unbounded probabilistic behavior into a controlled system.

Latency budgets¶

Start from the product requirement.

For example:

interactive chat response: target < 3 s to first useful output
background document analysis: 30–60 s may be acceptable
nightly offline evaluation: minutes may be acceptable

Architecture depends on the required interaction model.

A workflow that is excellent as a background job may be unacceptable in a synchronous UI.

Quality, cost, and latency form a trade-off triangle¶

Often:

stronger model
→ better quality
→ higher cost
→ higher latency

Not always, but often enough to treat them as competing dimensions.

Evaluation should compare all three.

Example:

Model A
quality: 88%
latency: 1.2 s
cost:    $0.004/task

Model B
quality: 92%
latency: 4.5 s
cost:    $0.040/task

Model B is not automatically the better product choice.

Example: document Q&A¶

Version A:

retrieve 20 chunks
send all 20 to strong model

Version B:

retrieve 50 cheap candidates
 ↓
rerank
 ↓
select 5 chunks
 ↓
use medium model

Version B may have an extra retrieval stage but still be:

cheaper,
faster,
more accurate,

because the final context is smaller and cleaner.

Architecture must be measured end-to-end.

Example: classification at scale¶

Task: classify 20 million messages/month.

Architecture A:

strong reasoning model for every message

Architecture B:

small classifier model
 ↓
ambiguous cases only
 ↓
strong model

Even a modest successful routing strategy can have a large financial impact at that volume.

Observe cost like a production metric¶

Useful metrics:

input tokens/request,
output tokens/request,
model calls/task,
tool calls/task,
cost/task,
cost/user,
cost/feature,
p50/p95 latency,
time to first token,
retry rate,
cache hit rate.

Do not wait for the monthly invoice to discover architectural problems.

Optimize after measuring¶

Premature optimization is still a risk.

A sensible progression:

1. make it work
2. evaluate quality
3. measure cost and latency
4. identify dominant cost
5. optimize the bottleneck

Do not replace a reliable model with a cheaper one based only on token price without evaluating the full system.

Mental model¶

Think of every AI workflow as a graph where each node has:

quality contribution
latency
cost
failure probability

The goal is not to minimize model price.

The goal is to optimize total task economics and user experience while preserving required quality and safety.