Skip to content

Reflection, Verification and Critique

Reflection is not the same as verification

A common agentic pattern is:

model produces answer
 ↓
ask model: "Are you sure?"
 ↓
model thinks again

This can sometimes improve quality, but it is a weak guarantee by itself.

The important distinction is:

Reflection
= the model re-evaluates its own result

Verification
= the result is checked against external or more independent evidence

New evidence is generally stronger than another pass over the same assumptions.

When is reflection useful?

Reflection can help detect:

  • obvious inconsistencies,
  • missing edge cases,
  • violations of an output rubric,
  • weak plans or actions,
  • opportunities to repair a candidate.

For example:

Draft answer
 ↓
Check:
- answered all requested fields?
- claims supported by evidence?
- any contradictions?

This can be a useful quality pass.

When is reflection weak?

Reflection cannot supply facts that are missing from context.

Suppose the model assumes:

production version = 7.4.1

Telling it to “think again carefully” does not provide current deployment state.

Better:

get_current_deployment()
 ↓
new observation
 ↓
re-evaluate

Verification hierarchy

Different verifiers provide different levels of confidence.

Deterministic verifier

Examples:

JSON Schema validation
unit tests
compiler
static type checker
linter
policy engine
database constraint
cryptographic hash

If a property can be checked deterministically, prefer that.

External-state verifier

Examples:

Did deployment actually reach READY?
Did GitHub PR actually merge?
Does invoice actually exist?

Read the current source of truth.

Semantic verifier

Some properties are not simple booleans.

Example:

Does this review finding identify a real architecture risk?

Possible verifiers include:

  • a human reviewer,
  • a rubric-based evaluator,
  • a second model or judge,
  • a domain-specific heuristic.

This is a softer guarantee, but still stronger than raw self-confidence.

Generate → Verify → Repair

One of the most useful bounded patterns is:

Generate candidate
       ↓
Verify candidate
       ↓
Pass? ── yes → finish
  │
  no
  ↓
Repair using verifier feedback
       ↓
Verify again

For a coding agent:

edit code
 ↓
compile
 ↓
compiler error
 ↓
repair
 ↓
unit test
 ↓
failed assertion
 ↓
repair
 ↓
regression suite

Verifier feedback becomes a new observation in the loop.

Structured verifier output

Weak:

"Something seems wrong."

Better:

{
  "status": "FAILED",
  "checks": [
    {
      "name": "payment-regression",
      "status": "FAILED",
      "evidence": "RefundRetryTest failed: expected 1 charge, got 2"
    }
  ]
}

The repair step can respond to concrete evidence.

Critic pattern

A separate critic can inspect a candidate:

Producer
  ↓ candidate
Critic
  ↓ findings
Producer / Repairer
  ↓ revised candidate

The critic might be:

  • the same model with different instructions,
  • another model,
  • a separate skill,
  • a human,
  • a deterministic verifier combination.

A “critic agent” is not automatically independent or correct. If it receives the same bad context and assumptions, it may reinforce the same error.

Independent evidence matters more than role-play

Real independence often comes from different evidence sources.

Stronger:

Generator:
reads implementation

Verifier:
runs tests + checks runtime output

than:

Generator model
 ↓
Same model asks itself whether code is correct

Independence can come from:

  • different data sources,
  • deterministic checkers,
  • another model family,
  • a separate rubric/prompt,
  • hidden expected-output fixtures,
  • human review.

Reflection before action

Reflection can also happen before a risky action:

Proposed action: restart production service
 ↓
Check:
- current state fresh?
- approval exists?
- safer diagnostic action available?
- expected effect explicit?

But hard checks still belong to runtime policy. Reflection does not replace authorization, approval tokens, or deterministic preconditions.

Confidence vs evidence

Model output:

confidence: 0.95

is not equivalent to:

test suite passed

Confidence may be a routing signal, but it is not proof.

Verification budgets

Verification can itself become an endless loop:

generate
 ↓
critic
 ↓
repair
 ↓
critic
 ↓
repair
 ↓
...

Bound it:

{
  "max_repair_cycles": 3,
  "required_verifiers": ["schema", "unit_tests"],
  "optional_verifiers": ["semantic_review"]
}

If the artifact still fails after the maximum repair cycles, stop or escalate.

Verification ordering

Run cheap and fast checks before expensive ones.

For code:

1. syntax/format
2. compile/typecheck
3. targeted unit test
4. regression suite
5. expensive integration test

There is little value in starting a 20-minute integration suite on code that does not compile.

Semantic verification example: PR review

Candidate finding:

{
  "severity": "HIGH",
  "claim": "This retry can charge twice."
}

Verify:

1. evidence location exists?
2. call path reaches non-idempotent charge?
3. retry can repeat after ambiguous failure?
4. existing idempotency key already prevents duplication?

If item 4 disproves the claim, reject the finding.

Data-extraction verification

LLM output:

{
  "invoice_total": 1200.00,
  "currency": "EUR"
}

Checks:

schema valid? ✅
source document contains 1200 EUR? ✅
line-item sum equals total? ❌

Again:

schema correctness
≠
semantic/business correctness

Model-as-judge

A model judge is useful when semantic quality matters and deterministic metrics are insufficient.

A strong pattern is:

hard deterministic checks
       +
semantic judge
       +
periodic human calibration

Use model judges cautiously for hard production gates.

Verification feedback as an observation

Store verifier results in canonical run state:

{
  "type": "VERIFICATION_RESULT",
  "verifier": "unit_test_runner",
  "status": "FAILED",
  "evidence": "RetryTest: duplicate charge"
}

The next action can then be REPAIR, REPLAN, STOP_FAILED, or ASK_HUMAN.

Anti-patterns

Avoid “Are you sure?” loops without new evidence, critics that merely repeat the same hallucinated assumptions, unbounded reflection cycles, and high-impact mutations without independent verification.

Takeaways

  • Reflection and verification are different concepts.
  • Self-reflection can improve quality but is limited without new evidence.
  • Deterministic and external-state verifiers are stronger where available.
  • generate → verify → repair is a powerful bounded agentic pattern.
  • Verifier outputs should be structured and evidence-backed.
  • Critic agents are not automatically independent; diversity of evidence and checking methods matters.
  • Confidence is not correctness proof.
  • Verification loops need budgets and stop conditions.
  • Run cheap deterministic checks before expensive semantic/integration checks.
  • Store verification results as observations in canonical run state.