Skill Evaluation and Observability¶
Why do we need skill-level evaluation?¶
If a skill is a reusable software artifact, it is not enough to say:
"It seems to work."
We need to measure:
- whether it actually solves the task,
- whether it chooses skills/tools correctly,
- whether it respects its contract,
- what failures it produces,
- latency and cost,
- whether a change caused a regression.
That is the goal of skill evaluation.
Evaluation layers¶
A useful breakdown:
Contract correctness
↓
Task / semantic correctness
↓
Tool-use correctness
↓
Execution-path correctness
↓
Safety / policy correctness
↓
Operational metrics
Not every skill needs every layer at the same depth, but production capabilities should consider all of them.
1. Contract correctness¶
Deterministic questions:
valid input schema?
valid output schema?
required fields present?
enums valid?
result type allowed?
These can be tested with ordinary unit tests and schema validation.
For example:
PR Review Skill result must always contain:
- findings[]
- overall_risk
- summary
This matters, but it still does not prove the review is good.
2. Task success / semantic correctness¶
The real question is:
Did the skill solve the task it was designed for?
For a PR Review Skill:
- did it find the real bug?
- did it miss a critical issue?
- did it generate false positives?
- were findings grounded in evidence?
For an Incident Triage Skill:
- correct severity?
- relevant suspected area?
- justified next diagnostic action?
Golden / representative dataset¶
Maintain a stable evaluation dataset.
For example:
evals/pr-review/
├── duplicate-payment-retry
├── harmless-refactor
├── missing-auth-check
├── flaky-test-only
├── misleading-comment
└── backwards-compatibility-break
Each case can contain:
input
relevant context
expected findings / rubric
important non-findings
Do not include only happy-path examples.
Dataset slices¶
Aggregate metrics can hide problems.
For example:
91% success overall
while:
Java backend: 96%
Python: 94%
SQL migrations: 61%
Slice by dimensions such as:
- task type,
- language/domain,
- input length,
- ambiguity,
- tool availability,
- risk category,
- edge case.
Exact match vs semantic rubric¶
Exact match is useful for classification:
expected category = BILLING
It is a poor metric for architecture review wording.
Use a rubric instead:
0 = misses the material risk
1 = notices vague concern but no evidence
2 = identifies risk with partial evidence
3 = identifies risk, cites evidence and explains consequence
LLM-as-a-judge¶
For semantic evaluation, another model can score against a rubric.
candidate output
+ reference/rubric
↓
judge model
↓
score + rationale
This scales better than manual review, but it is not ground truth.
Risks include:
- judge bias,
- model-family preference,
- verbosity bias,
- inconsistent scoring,
- shared blind spots.
Keep a human-calibrated subset where possible.
Human evaluation¶
Human review may be necessary for high-risk or subjective tasks.
For example, a senior engineer may judge:
Would this finding be useful in a real code review?
Is severity appropriate?
Is the evidence convincing?
Human evaluation is more expensive, so it is useful for:
- establishing baselines,
- calibrating judges,
- investigating regressions,
- high-risk cases.
Tool-use evaluation¶
For tool-backed skills, final answer quality is not enough.
Measure:
selected correct tool?
arguments correct?
unnecessary tool calls?
repeated calls?
forbidden tool requested?
used fresh data when required?
For a question about current account balance, a final answer may accidentally be correct from prior context; if the skill never queried the authoritative balance tool, the execution is architecturally wrong.
Trajectory / execution-path evaluation¶
For agentic skills, the path matters as well as the final result.
Goal
↓
Tool A
↓
Tool B
↓
Skill decision
↓
Tool C
↓
Result
Evaluate:
- unnecessary steps,
- repeated actions,
- correct stop conditions,
- unsafe intermediate actions,
- whether required evidence was collected.
A correct final answer does not make an unsafe intermediate action acceptable.
Safety evaluation¶
Maintain dedicated cases such as:
prompt injection
unauthorized write request
cross-tenant data request
malicious retrieved content
secret extraction request
Expected behaviors may include:
DENY
REQUIRE_APPROVAL
IGNORE_UNTRUSTED_INSTRUCTION
NO_CROSS_TENANT_ACCESS
Security evaluation should be a regression suite, not a one-time red-team exercise.
Failure-mode evaluation¶
Test failures too:
tool timeout
rate limit
resource missing
partial result
invalid input
context too large
approval denied
The skill should return the correct explicit failure state rather than hallucinated data.
Operational metrics¶
Production metrics can include:
success rate
partial-result rate
failure rate
retry rate
human-escalation rate
tool-call count
model-call count
input/output tokens
latency
cost
These are not quality metrics by themselves, but trends matter.
Latency breakdown¶
Do not track only total latency.
routing: 100 ms
retrieval: 300 ms
model call 1: 2.4 s
tool call: 900 ms
model call 2: 1.8 s
This shows where optimization is useful.
Cost attribution¶
Skill-level cost may include:
input tokens
output tokens
number of model calls
tool/provider cost
retrieval/rerank cost
This matters especially in agentic composition, where one user request can trigger many model/tool operations.
Observability: trace mental model¶
Give each execution an ID:
execution_id: exec-1842
Trace:
route skill
↓
load context
↓
model call
↓
tool request
↓
tool execution
↓
model call
↓
validation
↓
result
Each span can record:
duration
status
skill version
model
selected tool
error category
What should be logged?¶
Useful metadata:
execution id
skill name/version
model/provider
prompt/template version or hash
input schema version
output type
retrieved source ids
tool names + sanitized args
tool result status
latency/cost
validation errors
Do not log indiscriminately:
raw secrets
full customer PII
credentials
sensitive documents
Observability is also a data-security problem.
Prompt/model versions must be traceable¶
When quality changes, we need to know:
what changed?
For example:
skill v2.3
prompt v14
model X-2026-08-01
router v4
retriever config 7
Without this, regressions become difficult to reproduce.
Regression testing¶
Before a change:
baseline eval
After:
candidate eval
Compare multiple dimensions:
critical recall: 94% → 95%
false positive rate: 8% → 17% ❌
latency: 3.1s → 2.8s
cost: +4%
One aggregate score is not enough.
Release gates¶
Policies can include:
critical safety suite must be 100%
no more than 2% regression in task success
p95 latency < target
Exact thresholds depend on the domain.
The important principle is that a skill update should not reach production merely because it “looks better”.
Online evaluation¶
Production feedback can include:
user correction
human override
reopened ticket
PR review finding accepted/rejected
incident recommendation followed/ignored
These are valuable signals but can be biased:
- users do not always provide feedback,
- task difficulty mix changes,
- selection bias exists.
A stable offline evaluation set plus online monitoring is a stronger combination.
Drift¶
Quality can change even without skill-code changes:
input distribution changes
external data changes
model provider update
new repository patterns
new attack patterns
Periodic re-evaluation is therefore useful.
Example: PR Review evaluation matrix¶
| Dimension | Metric |
|---|---|
| Critical bug detection | recall |
| False findings | precision / false-positive rate |
| Evidence grounding | rubric score |
| Correct tool use | tool-call accuracy |
| Read-only safety | forbidden-action rate |
| Efficiency | calls / tokens / latency |
| Contract | schema validity |
Do not force every dimension into one score.
Debugging a failed execution¶
A good trace should answer:
wrong skill selected?
wrong context retrieved?
model reasoned badly?
tool returned stale/error data?
validation too weak?
policy denied valid action?
If only the final answer is logged, every failure collapses into “the LLM was wrong”.
Anti-pattern: vibe-based evaluation¶
I tried 10 prompts and they looked good.
This can be enough for a POC, not for production regression protection.
Anti-pattern: only schema validity¶
100% valid JSON
can coexist with 100% semantically wrong output.
Anti-pattern: only the final answer¶
Unsafe or wasteful trajectories remain hidden.
Anti-pattern: one overall score¶
Skill score: 87
Does not reveal whether security, latency, or critical recall regressed.
Takeaways¶
- Every production skill should have a representative evaluation dataset, not only manual testing.
- Measure contract correctness, semantic task success, tool use, trajectory, and safety separately.
- Exact match is appropriate only where the domain is genuinely exact.
- Semantic tasks can use rubric/human/LLM-judge combinations.
- For tool-backed skills, execution path is part of correctness.
- Test failure modes explicitly as evaluation cases.
- Production traces should identify skill/model/prompt/tool versions.
- Break latency and cost down by component.
- Compare baseline vs candidate before release.
- Combine stable offline evaluation with online monitoring.
- Observability itself must respect data minimization and secret handling.