Reliability, Scaling and Production Runtime¶

A production agent runtime is an execution platform for long-running, failure-prone, side-effecting work. The architecture therefore needs the same concerns as any durable job-processing system: queues, state persistence, retry semantics, idempotency, concurrency control, backpressure, observability and isolation.

A useful production shape is:

API / Event / Scheduler
        ↓
Run Service
        ↓
Durable Queue
        ↓
Worker Pool
        ↓
Agent Runtime
   ├── State Store
   ├── Policy/Budget
   ├── Context Builder
   ├── Model Gateway
   └── Capability Layer
        ↓
External Systems

The worker process is disposable. The run state is not.

Synchronous vs asynchronous runs¶

A short read-only interaction may execute synchronously:

request
 ↓
model/tool/model
 ↓
response

A task should move toward asynchronous execution when it can:

take tens of seconds or minutes
wait for human approval
wait for external systems
use many tools
survive client disconnect
require retries/checkpoints
consume substantial budget

Then the API starts a run and returns a run_id rather than holding a request open indefinitely.

POST /runs
→ 202 Accepted
→ run_id = run_123

The client can poll, subscribe to events or receive a callback when state changes.

Stateless workers, durable runs¶

Workers should generally load the current canonical run state when they start work.

Worker crashes
   ↓
queue lease expires / task redelivered
   ↓
new worker loads latest checkpoint
   ↓
reconciles uncertain side effects
   ↓
continues

If a run only exists in process memory, a deployment or crash destroys execution correctness.

Queue semantics¶

Many queues provide at-least-once delivery.

That means:

one logical step
may be delivered more than once

Therefore workers must not assume exactly-once execution.

The runtime needs:

step IDs
idempotency keys
state versions
completed-step records
side-effect reconciliation

Idempotent step execution¶

Before executing a step:

load run state
 ↓
check step_id not already completed
 ↓
acquire lease / compare state version
 ↓
execute
 ↓
persist result + new state atomically where possible

For external writes, use provider-supported idempotency keys when available.

Example:

payment.refund(run_123_step_8)

If the request times out, first determine whether the refund happened before retrying.

Leases and worker ownership¶

A worker may temporarily own execution of a run/step through a lease.

lease_owner
lease_expires_at
state_version

If the worker dies, another worker can resume after expiry.

Do not use an infinite distributed lock that requires manual cleanup after a crash.

Optimistic concurrency¶

Callbacks, approvals and workers can race.

Example:

Worker reads state version 20
Human cancels run → version 21
Worker attempts to write result based on version 20

The update should fail, forcing the worker to reload state instead of resurrecting a cancelled run.

Waiting states¶

Do not keep a worker/model loop alive while waiting.

RUNNING
  ↓
WAITING_FOR_APPROVAL

Persist state and release compute.

When approval arrives:

Approval event
  ↓
queue resume message
  ↓
worker reloads state
  ↓
revalidates
  ↓
continues

The same applies to CI completion, webhooks and other external events.

Retry architecture¶

Retries should be owned by the correct layer.

Transport/provider retry¶

Examples:

HTTP 503
connection reset
rate limit

The adapter/model gateway may perform bounded retry with jitter.

Capability/domain retry¶

A conflict may require re-reading state rather than repeating the same request.

Agent-level retry/replan¶

A failed strategy may require new reasoning.

Do not stack invisible retries at every layer.

HTTP client retries 5×
adapter retries 5×
worker retries 5×
agent retries 5×

This can become 625 attempts.

Retry budgets should compose explicitly.

Backoff and jitter¶

For transient failures, use bounded exponential backoff with jitter rather than immediate retry storms.

1s
2s
4s
8s
...
max delay

Jitter prevents many workers from retrying simultaneously after a provider outage.

Rate limiting¶

AI systems are constrained by more than CPU.

Possible limits:

requests/minute
tokens/minute
concurrent model calls
provider quotas
external API quotas
sandbox capacity
per-tenant budget

The runtime should apply rate limits before causing large numbers of provider failures.

Backpressure¶

If demand exceeds execution capacity:

incoming work > workers/provider capacity

then queue depth grows.

A healthy system needs policies for:

queue limits
priority classes
per-tenant fairness
admission control
load shedding
max waiting time

Do not accept infinite autonomous work and hope workers catch up.

Execution budgets¶

Budgets are part of reliability, not only cost control.

Typical limits:

max model calls
max tool calls
max wall-clock duration
max tokens
max dollar cost
max retries
max worker fan-out

The runtime enforces hard limits.

A model cannot be trusted to remember that it has only two tool calls left.

Provider fallback¶

Fallback can improve availability but may change behavior.

Primary model unavailable
        ↓
Fallback model

Before enabling automatic fallback, define compatibility requirements:

structured output support
tool calling
context window
latency
safety policy
quality threshold
region/data policy

A cheaper/smaller model may not be safe for a high-risk write workflow.

Fallback should be policy-driven by task class.

Circuit breakers¶

If a provider is repeatedly failing, stop hammering it.

CLOSED
 ↓ repeated failures
OPEN
 ↓ cooldown
HALF_OPEN
 ↓ probe
CLOSED / OPEN

Circuit breakers protect both provider and system resources.

Bulkheads and failure isolation¶

One bad integration should not exhaust the entire worker pool.

Possible isolation boundaries:

separate queue per workload class
separate worker pool for code execution
per-tenant concurrency limits
provider-specific concurrency pools
separate high-risk execution service

This prevents, for example, a slow browser tool from blocking all simple read-only agent runs.

Scaling model¶

Agent workloads are often I/O-bound:

waiting for model
waiting for tools
waiting for retrieval
waiting for APIs

Scaling may involve concurrent async workers rather than CPU-heavy processes.

But code execution, OCR, embeddings or local models may be CPU/GPU-bound and need separate pools.

Measure the bottleneck before splitting infrastructure.

Modular monolith and scaling¶

A modular monolith can still scale horizontally:

same application image
├── API replicas
├── worker replicas
└── scheduler

You do not need microservices simply because there are queues or agent workers.

Extract a service when there is a concrete reason such as:

independent scaling profile
security/isolation boundary
separate availability requirement
separate ownership team
different deployment cadence
special runtime dependency (GPU/sandbox/browser)

Example:

modular monolith
   ↓ later
CodeExecutionService

may be justified because executing untrusted code needs stronger isolation, not because "agents use microservices".

Version pinning for long runs¶

A run may start before a deployment and resume after it.

Persist important execution versions:

runtime_version
skill_version
policy_version
model_profile
context strategy version

Then decide whether old runs:

continue with compatible current code
resume using pinned behavior
migrate state
or fail safely

Do not silently change the rules of an approval-waiting run halfway through execution.

Deployment and graceful shutdown¶

Workers should stop safely during deployment.

stop accepting new work
 ↓
finish/checkpoint current step
 ↓
release lease
 ↓
terminate

A forced shutdown should leave durable state from which another worker can recover.

Observability and SLOs¶

Useful runtime metrics:

run success rate
p50/p95/p99 run latency
queue wait time
steps per run
model calls per run
tool failure rate
retry rate
approval wait time
cost per successful run
provider error rate
budget-exhausted rate
stuck-run count

Define SLOs by workload class rather than one global target.

A chat answer and a 20-minute investigation have different latency expectations.

Failure recovery example¶

Run step: create GitHub issue
        ↓
request sent
        ↓
timeout before response
        ↓
state = OUTCOME_UNKNOWN
        ↓
reconciliation query using idempotency/reference
        ├── issue exists → record success
        └── no issue → safe retry

This is more reliable than "tool failed, retry".

Cost-aware scheduling¶

Production runtime can route based on workload:

simple classification → small model
complex architecture review → stronger model
bulk offline summarization → batch queue
interactive request → latency-priority queue

Cost routing belongs in explicit policy rather than random model self-selection.

Common anti-patterns¶

Run state in worker memory¶

Crashes destroy progress.

Long-lived polling loops¶

Workers and tokens are wasted while waiting for events.

Exactly-once assumption¶

Queue redelivery duplicates side effects.

Nested retry explosion¶

Every layer retries independently.

Infinite queue growth¶

No admission control or tenant fairness exists.

Automatic fallback to any model¶

Behavior/security assumptions silently change.

Microservices before bottlenecks exist¶

Operational complexity rises without clear benefit.

No version metadata¶

A resumed run cannot explain which skill/policy generated earlier decisions.

Engineering takeaways¶

Persist run state durably and treat workers as disposable.
Use queues/event-driven resume for long-running work rather than keeping loops alive.
Assume at-least-once delivery and design idempotent/reconcilable side effects.
Keep retry ownership explicit and bounded across layers.
Enforce concurrency, rate, cost and execution budgets deterministically.
Use circuit breakers, backpressure and bulkheads to contain provider/tool failures.
Scale measured bottlenecks; a modular monolith can support separate API/worker replicas without premature microservices.
Persist behavior/version metadata for long-running runs.
Define SLOs and metrics per workload class.
Reliability architecture is part of agent design, not infrastructure added after prompts are finished.