Reliability, Scaling and Production Runtime¶

Egy production agent runtime long-running, failure-prone és side-effecting work execution platformja. Ugyanazokra a concernökre van szüksége, mint bármely durable job-processing rendszernek: queue, state persistence, retry semantics, idempotency, concurrency control, backpressure, observability és isolation.

Hasznos production shape:

API / Event / Scheduler
        ↓
Run Service
        ↓
Durable Queue
        ↓
Worker Pool
        ↓
Agent Runtime
   ├── State Store
   ├── Policy/Budget
   ├── Context Builder
   ├── Model Gateway
   └── Capability Layer
        ↓
External Systems

A worker process disposable. A run state nem az.

Synchronous vs asynchronous run¶

Rövid read-only interaction lehet synchronous:

request
 ↓
model/tool/model
 ↓
response

Asynchronous execution felé érdemes menni, ha a task:

tens of seconds or minutes
human approvalra vár
external systemre vár
many tools-t használ
survive client disconnect kell
retry/checkpoint kell
substantial budgetet fogyaszt

Ilyenkor az API run-t indít és run_id-t ad vissza:

POST /runs
→ 202 Accepted
→ run_id = run_123

Client pollolhat, eventre subscribe-olhat vagy callbacket kaphat.

Stateless worker, durable run¶

Worker induláskor canonical state-et töltse be.

Worker crashes
   ↓
queue lease expires / task redelivered
   ↓
new worker loads latest checkpoint
   ↓
reconciles uncertain side effects
   ↓
continues

Ha a run csak process memoryban él, deployment vagy crash execution correctnesset veszít.

Queue semantics¶

Sok queue at-least-once deliveryt ad:

one logical step
may be delivered more than once

Ezért worker ne feltételezzen exactly-once executiont.

Szükséges lehet:

step IDs
idempotency keys
state versions
completed-step records
side-effect reconciliation

Idempotent step execution¶

load run state
 ↓
check step_id not already completed
 ↓
acquire lease / compare state version
 ↓
execute
 ↓
persist result + new state atomically where possible

External writehoz provider-supported idempotency keyt használj, ha van.

Ha request timeoutol, először derítsd ki, megtörtént-e a side effect, mielőtt retry-olnád.

Lease és worker ownership¶

Worker temporary lease-szel birtokolhat run/step executiont:

lease_owner
lease_expires_at
state_version

Worker death után más worker resume-olhat lease expiry után. Kerüld az infinite distributed lockot manual cleanup szükséglettel.

Optimistic concurrency¶

Callback, approval és worker race-elhet.

Worker reads state version 20
Human cancels run → version 21
Worker writes based on version 20

Az update bukjon meg, worker reloadolja state-et, és ne „élesszen újra” cancelled runt.

Waiting state¶

Várakozás közben ne tarts életben workert/model loopot.

RUNNING
  ↓
WAITING_FOR_APPROVAL

Persist state, release compute.

Approval event:

Approval event
  ↓
queue resume message
  ↓
worker reloads state
  ↓
revalidates
  ↓
continues

Ugyanez CI completion, webhook és external event esetén.

Retry architecture¶

Retry a megfelelő layer felelőssége.

Transport/provider retry¶

HTTP 503
connection reset
rate limit

Adapter/model gateway bounded retryt végezhet jitterrel.

Capability/domain retry¶

Conflict esetén lehet, hogy state-et kell újraolvasni ugyanazon request megismétlése helyett.

Agent-level retry/replan¶

Failed strategy új reasoningot igényelhet.

Kerüld a nested retry explosiont:

HTTP client retries 5×
adapter retries 5×
worker retries 5×
agent retries 5×

Ez 625 attempt is lehet. Retry budgetek explicit compose-oljanak.

Backoff és jitter¶

Transient failure esetén bounded exponential backoff + jitter:

1s
2s
4s
8s
...
max delay

Jitter megakadályozza, hogy outage után minden worker egyszerre retry-oljon.

Rate limiting¶

AI system nem csak CPU-val limitált.

requests/minute
tokens/minute
concurrent model calls
provider quotas
external API quotas
sandbox capacity
per-tenant budget

Runtime a provider failure előtt kontrollálja a terhelést.

Backpressure¶

Ha:

incoming work > workers/provider capacity

queue depth nő.

Policy kell:

queue limits
priority classes
per-tenant fairness
admission control
load shedding
max waiting time

Ne fogadj végtelen autonomous worköt abban bízva, hogy majd utoléri magát a worker pool.

Execution budgets¶

Budgets reliability concern is:

max model calls
max tool calls
max wall-clock duration
max tokens
max dollar cost
max retries
max worker fan-out

Hard limitet runtime enforce-ol, nem a model „emlékszik rá”.

Provider fallback¶

Fallback javíthat availabilityn, de behavior változhat.

Primary model unavailable
        ↓
Fallback model

Automatic fallback előtt compatibility requirement:

structured output support
tool calling
context window
latency
safety policy
quality threshold
region/data policy

High-risk write workflow ne essen vissza automatikusan inkompatibilis cheap modelre.

Circuit breaker¶

Repeated provider failure esetén ne hammereld tovább:

CLOSED
 ↓ repeated failures
OPEN
 ↓ cooldown
HALF_OPEN
 ↓ probe
CLOSED / OPEN

Bulkhead és failure isolation¶

Egy rossz integration ne foglalja le a teljes worker poolt.

Lehet:

separate queue per workload class
separate worker pool for code execution
per-tenant concurrency limits
provider-specific concurrency pools
separate high-risk execution service

Például slow browser tool ne blokkolja az összes simple read-only runt.

Scaling model¶

Sok agent workload I/O-bound:

waiting for model
waiting for tools
waiting for retrieval
waiting for APIs

Async concurrency hasznosabb lehet, mint CPU-heavy worker process.

OCR, embeddings, code execution vagy local model külön CPU/GPU poolt igényelhet. Mérd a bottlenecket service split előtt.

Modular monolith és scaling¶

Modular monolith horizontálisan is scale-elhető:

same application image
├── API replicas
├── worker replicas
└── scheduler

Queue vagy agent worker miatt még nem kell microservice.

Service extraction jó ok:

independent scaling profile
security/isolation boundary
separate availability requirement
separate ownership team
different deployment cadence
special runtime dependency

Version pinning long runnál¶

Run indulhat deployment előtt és resume-olhat utána.

Persistáld:

runtime_version
skill_version
policy_version
model_profile
context strategy version

Majd explicit döntés kell, hogy old run:

continue with compatible current code
resume using pinned behavior
migrate state
or fail safely

Approvalra váró run szabályai ne változzanak csendben félúton.

Graceful shutdown¶

Worker deploymentkor:

stop accepting new work
 ↓
finish/checkpoint current step
 ↓
release lease
 ↓
terminate

Forced shutdown után durable state-ből más worker recoveryzhessen.

Observability és SLO¶

Metric példák:

run success rate
p50/p95/p99 run latency
queue wait time
steps per run
model calls per run
tool failure rate
retry rate
approval wait time
cost per successful run
provider error rate
budget-exhausted rate
stuck-run count

SLO workload classonként legyen. Chat answer és 20 perces investigation más latency expectation.

Unknown outcome example¶

Run step: create GitHub issue
        ↓
request sent
        ↓
timeout before response
        ↓
state = OUTCOME_UNKNOWN
        ↓
reconciliation query using idempotency/reference
        ├── issue exists → record success
        └── no issue → safe retry

Ez jobb, mint tool failed → retry.

Cost-aware scheduling¶

simple classification → small model
complex architecture review → stronger model
bulk offline summarization → batch queue
interactive request → latency-priority queue

Cost routing explicit policy legyen, ne random model self-selection.

Common anti-patternök¶

Run state worker memoryban.
Long-lived polling loop waiting alatt.
Exactly-once assumption queue mellett.
Nested retry explosion.
Infinite queue growth admission control nélkül.
Automatic fallback bármilyen modelre.
Microservice bottleneck nélkül.
Version metadata hiánya long-run resume-nál.

Engineering takeaways¶

Run state durable legyen, worker disposable.
Long-running workhöz queue/event-driven resume jobb, mint életben tartott loop.
At-least-once delivery mellett idempotent/reconcilable side effect kell.
Retry ownership explicit és bounded legyen.
Concurrency, rate, cost és execution budget deterministic enforcementet igényel.
Circuit breaker, backpressure és bulkhead izolálja provider/tool failure-t.
Mért bottlenecket scale-elj; modular monolith is tud külön API/worker replicát.
Long runhoz persistáld a behavior/version metadata-t.
Workload-class specifikus SLO kell.
Reliability architecture az agent design része, nem promptok után hozzáadott infrastructure extra.