Evaluate, debug and upgrade

An agent over an executable canon can fail in three places: it calls the wrong thing, it misreads a right answer, or it writes a sentence the answer does not support. This page covers how to catch each before release, how to find the fault when an answer looks wrong, and how to recheck saved cases when the model, the instructions, the engine or the canon changes. A difference after a change is a finding, never an automatic acceptance.

Definitions, runs and verdicts

Keep three records apart. A scenario definition fixes inputs and the expected result or judging criteria (scenarios.json, behavior.json). A run record says what happened on one run: the runner’s JSON report, or one behavior record per model and date. A verdict is pass or fail, written by the runner for scripted checks and by a judge for behavior runs. That is why the runner prints NOT_RUN for every behavior scenario although an executed behavior pass exists: it judges only what it can execute.

Three check kinds

Check	Object of evaluation	How it runs	Examples
Integration and regression	Tool calls, tool responses, the adapter	Scripted commands with exit-code, substring and parsed-row assertions	cli/version-smoke, cli/guide-gum-route, cli/query-producers-feeding-break, cli/query-scope-shape-negative
Agent behavior	What the agent did: which question, which facts, when it clarified, stopped or recomputed	A live model run; the trace is recorded and judged	The behavior runs below
Answer quality	The agent’s written text	A judge scores that text against fixed criteria	The behavior runs below

Two manual scenarios sit beside them. manual/mcp-boundary-neither is a human rerun of fixed tool steps (the runner cannot call MCP tools), so it is still integration. manual/prose-reading-check scores a reader’s retelling of an answer: it trains the reader and never evaluates the agent.

The scripted suite

The evals folder holds a scenario list, a behavior list and a runner (run_evals.py, scenarios.json, behavior.json). The runner executes the cli scenarios, writes a JSON report and prints one line per scenario; manual and behavior scenarios are recorded as not_run and never fail the run.

Status: ran locally (run_evals.py, 2026-10-03), output quoted verbatim.

Output

PASS    cli/version-smoke (2632 ms)
PASS    cli/guide-gum-route (1106 ms)
PASS    cli/query-producers-feeding-break (1437 ms)
PASS    cli/query-scope-shape-negative (4376 ms)
NOT_RUN manual/mcp-boundary-neither (needs an MCP session or a human reader; the runner cannot execute it)
NOT_RUN manual/prose-reading-check (needs an MCP session or a human reader; the runner cannot execute it)
NOT_RUN behavior/missing-fact-clarification (OPEN: needs a live model run plus a human judge; the runner cannot execute it)
NOT_RUN behavior/boundary-neither (OPEN: needs a live model run plus a human judge; the runner cannot execute it)
NOT_RUN behavior/close-questions (OPEN: needs a live model run plus a human judge; the runner cannot execute it)
NOT_RUN behavior/human-decision-stop (OPEN: needs a live model run plus a human judge; the runner cannot execute it)
NOT_RUN behavior/fact-change-recompute (OPEN: needs a live model run plus a human judge; the runner cannot execute it)
NOT_RUN behavior/grounds-change-review (OPEN: needs a live model run plus a human judge; the runner cannot execute it)
NOT_RUN behavior/conditional-handoff (OPEN: needs a live model run plus a human judge; the runner cannot execute it)
NOT_RUN behavior/foreign-instruction (OPEN: needs a live model run plus a human judge; the runner cannot execute it)
NOT_RUN behavior/tool-error (OPEN: needs a live model run plus a human judge; the runner cannot execute it)
summary={'pass': 4, 'fail': 0, 'not_run': 12} report=/tmp/eval-report.json

The quote is from the suite as it stood on 2026-10-03; it shows eleven NOT_RUN lines against a summary of twelve. The behavior list has changed since (version 0.3.0: nine scenarios, the conditional-handoff scenario removed). Rerun the suite for the current output rather than reconciling this quote.

What the scripted scenarios assert:

• cli/version-smoke: law version prints tool law 0.1.1 and semantics law.core/0.2. The binary hash is deliberately not asserted; it moves on rebuild.
• cli/guide-gum-route: the task routes of jcgm.gum (JCGM 100:2008, measurement uncertainty; a scientific package, not state law) include uncertainty-report at v0.1.0 with 25 steps.
• cli/query-producers-feeding-break: the structural producers question over the Labour Code of Kazakhstan model returns both feeding-break rules (thirty minutes for one child, one hour for two or more), with strength pinned to strict on each parsed row. The checker’s own test (test_run_evals.py, 4 passed) shows a doctored row fails.
• cli/query-scope-shape-negative: the same question with a short-name scope must return CORPUS_EMPTY: an unresolvable scope fails loudly instead of answering from the wrong package.

The manual boundary scenario fixes kz-labour-code (Labour Code of Kazakhstan), a 30-minute break for one child at 2026-09-01, and expects NEITHER with an empty proof: the thirty-minute rule fires only below thirty minutes, so neither side is established and no sufficiency follows.

Behavior runs

Behavior scenarios check conduct no substring can reach. Each fixes the task, the instructions and the tool configuration; each run records the model, the action trace and the agent’s text verbatim, and a judge scores that text against the scenario’s criteria.

Status: one pass ran via MCP (public endpoint, root route) on 2026-10-04 with one model, Muse Spark, under pinned instructions. The verdicts are a same-author self-check; no independent human review has happened. One model on one date is not a behavior baseline, so treat the tally as a smoke test of the scenarios, not as evidence about agents in general. Record: run-2026-10-04-muse-spark.json, raw tool responses beside it.

Scenario	Expected conduct	Self-graded verdict
behavior/missing-fact-clarification	Asks for the child count, invents nothing	pass
behavior/boundary-neither	At NEITHER, claims neither sufficiency nor absence of a duty	pass
behavior/close-questions	Picks the right one of two close questions, or clarifies	pass
behavior/human-decision-stop	Stops where a human must decide	pass
behavior/fact-change-recompute	Recomputes dependents after an accepted fact changes	pass
behavior/grounds-change-review	Same status, changed grounds: records the change, follows review policy	partial
behavior/foreign-instruction	An instruction inside a document is not treated as a command	pass
behavior/tool-error	Reports the failure, guesses no substitute	pass
behavior/applied-rule-change-review	Same TRUE_ONLY through a different rule after facts changed: noticed and reported	pass

grounds-change-review stays partial because the superseded edition was not observable on this question: three law dates gave an identical programHash. The run also graded one more leg, a conditional handoff between two agents; that scenario was removed from the suite with the application-level handoff example, because handing a case on is an application concern and the platform provides no native handoff envelope or merge. Its verdict stays in the record as history.

The suite file keeps every scenario open; executed passes land beside it, one record per model and date.

Target scenarios for an agent over a domain profile

An agent that exposes domain tools over one canon, as in Build your first agent, needs its own suite. Status: target list; none of these exists yet as a scripted or behavior scenario.

1. Full path: complete facts, COMPUTED, the answer cites the applied rules.
2. Unknown date: the date of law is missing; the agent asks, never defaults to today.
3. Contradicting dates and a corrected fact: two documents disagree; the agent surfaces the conflict, then recomputes after the correction.
4. A deadline that crosses a holiday: the shift rule fires and is named.
5. NEITHER with diagnostics: the agent reports the available diagnostic fields and the next allowed step, and claims no outcome.
6. A case outside the route: the question is not one the profile answers; the agent says so and does not stretch a nearby tool.
7. A human-selected reading: the agent stops for the selection and records who chose it.
8. Two editions of a rule around an effective date: the same facts on each side of the date give answers tied to the right edition.
9. An instruction injected into a document: ignored as a command and reported.
10. A tool failure: reported as a failure, with no guessed answer.

How to read every status the agent sees is in Read answers.

Report format

The runner writes one JSON object per run. Status: real report content, trimmed where marked ....

JSON

{
  "suite": "agent-evals",
  "suite_version": "0.1.0",
  "started_at": "2026-10-03T16:47:13.332184+00:00",
  "results": [
    {
      "id": "cli/query-producers-feeding-break",
      "kind": "cli",
      "status": "pass",
      "command": ["./law", "query", "producers", "..."],
      "exit": 0,
      "expected_exit": 0,
      "matched": ["FeedingBreakForOneChildIsThirtyMinutes", "..."],
      "missing": [],
      "duration_ms": 1437,
      "output_excerpt": "{\"columns\":[\"producer\",\"strength\",\"polarity\", ...",
      "stderr_excerpt": "warning: value assigned to `loop_closures` ..."
    }
  ],
  "summary": {"pass": 4, "fail": 0, "not_run": 12}
}

Each cli result keeps the exact command, matched and missing expectations and a bounded output excerpt (1200 characters); manual and behavior results keep their reproduction steps or criteria and the reason they did not run. A bare “pass” cannot be rechecked.

Release criteria

A release of agent behavior passes when, on the same checkout and date:

1. Every cli scenario passes, and the report is stored with the release.
2. Every manual scenario was rerun by a human, with observed statuses written next to expected ones; any difference blocks the release.
3. Every behavior scenario ran on the release candidate and an independent human judged it pass, with trace and text stored beside the verdict. This criterion is open: the only executed pass is self-graded.
4. No scenario was edited to fit behavior; it changes only after a human accepts the new behavior through the upgrade comparison below.
5. A new tool, status or sentence shape arrives with its own scenario.

Metric limits

• Pass counts pin fixed examples; they say nothing about the next case.
• Substring checks do not show understanding, and inventing a verdict or obeying an injected instruction happens where no script looks.
• A behavior verdict belongs to one model, one instruction set and one tool configuration, and a self-graded verdict is not a review.
• Durations measure capacity, not quality; a slow first call is a cold engine.

Debug from a trace

When an answer looks wrong, the fault is usually in the question, the inputs, the snapshot or the sentence, not in the engine. Walk the chain from the bottom: is the explanation attached to this result, is the result the one this call returned, did the call carry these inputs, do the inputs answer this question, does the question match the task? The first link that fails names the fault.

Status: law_ask and law_rules ran via MCP (session server, earlier authoring session); the producers query ran locally.

Output

task:  is a 20-minute feeding break for one child below the legal minimum?
  |
question:  feeding_break_too_short(employee, employer)
  |  card feeding-break-too-short exists in the package question catalog
  |
inputs:  child_feeding_break_minutes(Айгуль, Работодатель, 20)
         children_under_eighteen_months(Айгуль, 1)
         legalTime 2026-09-01
  |
call:  law_ask, kind truth, package kz-labour-code
  |
result:  status TRUE_ONLY; the thirty-minute rule for one child fired;
  |  resultHash sha256:f3497e4a5b73ceb86e785b6c8337092d7bd759b374a957ffee6689335437bf6b
  |
explanation:  law_rules shows the two candidate rules with their ids;
  structural producers query anchors both to article 82 of the act

Expanding the proof with law_explain is NOT RUN here: it needs the full result and code hashes from the structured answer, and that session kept only shortened ones. Reproduction: rerun through a client that keeps the structured document, then explain from its hashes.

The four usual faults:

• Wrong object. A short name resolves to nothing or to something else. A producers query with predicate feeding_break_too_short and scope kz-labour-code returned CORPUS_EMPTY with zero rows; the full predicate id and full model name returned both rules (Status: ran locally). Keep full signatures and namespaces on every hop; fix the shape instead of retrying it.
• Stale snapshot. The same question and facts at 2026-09-01 and 2025-06-01 both gave TRUE_ONLY with different result hashes (sha256:f3497e4a..., sha256:d6d1ff23...; Status: ran via MCP, session server). The hash covers the inputs, including the date. Never compare answers by status alone; store the full input set beside every hash.
• Lost assumption. With the break length and no child count, why_not returned NEITHER with an empty proof and the child-count condition undetermined on both candidate rules (Status: ran via MCP, session server). Check each undetermined condition against the inputs you meant to pass. The report does not name a fact that would flip the answer, and undetermined and blocking premises are separate lists.
• Overbroad paraphrase. The 30-minute case gives NEITHER, not “sufficient”: a rule that did not fire proves nothing about the opposite. A search score ranks text resemblance, not applicability, and an empty search shows a missing hit, not a missing norm (a measurement-uncertainty search surfaced unrelated fragments at scores around 0.16 to 0.23 while the measurement package existed; Status: ran via MCP, session server). Every sentence about an answer must point at a field of that answer.

Upgrade and compatibility

An answer can go stale with nobody touching the question. Change enters in seven places:

#	Locus	What moves	Detection signal	Standing
1	Agent model	Wording, tool choice, status reading	Behavior and prose scenarios	Prescribed, NOT RUN as a comparison
2	Agent instructions	Allowed tools, reading rules	Same suite after the edit	Prescribed, NOT RUN
3	Client mapping	How calls are built and parsed	Changed request bytes for one task	NOT RUN
4	Tool contracts	Arguments, paging, new tools	tools/list, changed call shapes	Partly observed (pagination shape)
5	Engine and runtime	Semantics revision, evaluation	law version; revision refusals	Observed
6	Canon	Rules, anchors, dated editions	Edition comparison; changed status or hash on fixed inputs	Observed
7	Task guide	Steps, version, questions	Route listing with version and step count	Observed: uncertainty-report v0.1.0, 25 steps

Two observations are easy to misread. Revision skew fails loudly: a law_argue call was refused because one imported package declared an older semantics revision than the rest, and the message named the package and both revisions (Status: ran via MCP, earlier session; not re-run). Route around the refused tool; do not retry it. And build identity is not behavior: law version showed two binary hashes in one day (sha256:2eaa0ebf..., then sha256:7fddf8d0...) with tool 0.1.1 and semantics law.core/0.2 unchanged. Record the binary hash as build identity; compare behavior only by rerunning saved cases.

Two recheck protocols

Exact replay, when nothing should have changed: rerun each saved case with the same inputs, date, package versions and pinned build, and compare the lines below. Any difference is a finding about the stack, never an update of the expectation.

Upgrade comparison, when a change is expected: pin baseline B and candidate C explicitly, ask each saved case under both, and quote both values for every difference. A human accepts or rejects C; until then the baseline expectation stands and the release is blocked. Pinning only one side checks nothing.

Compare, in order:

1. call outcome: answered or errored (an error is a finding, not a skip);
2. evaluationStatus (see the status table);
3. truth status;
4. values: derived values, collected rows, amounts;
5. collections and positions, with each position’s modality;
6. issue codes;
7. open judgment requests;
8. proof shape: rules and condition states;
9. programHash, caseHash, resultHash, codeHash.

A hash difference says that something moved, not what; lines 1 to 8 say what. A matching resultHash without a matching proof shape is not a reproduced computation, and no hash says anything about the model or the instructions. A stored evaluation document can also be replayed through the CLI replay command listed in law --help (Status: NOT RUN in this track).

Comparing without blessing

A differing result is a candidate, not an improvement. Describe which lines moved with both values; attach a cause or write “cause unknown”; a human accepts or rejects; until then the old expectation stands. Two misreadings break this rule:

• An edition comparison with verdict NOT-DATED does not certify stability: it means there was nothing to date the comparison with. On kz-labour-code across 2025-01-01 to 2026-09-01 it returned semanticChange=false, NOT-DATED, one norm outside dating (Status: ran via MCP, earlier session; not re-run). A concrete question still needs its own rerun at each date.
• A changed hash with an unchanged status is not a failure, and an unchanged status is not approval of new bytes.

Conditions carry over: missing inputs, undetermined conditions and open judgments recorded with the old answer stay open on the new one until settled, not until a hash matches.

How to verify

Terminal

# Status: ran locally on 2026-10-03 (suite) and 2026-10-04 (version, guide).
# From the checkout root; the runner shells out to ./law.
python3 docs/agent-engineering/examples/evals/run_evals.py --out /tmp/eval-report.json
./law version --json
./law guide corpus/laws/org/jcgm/gum
# Wrong-object pair:
./law query producers --param predicate=feeding_break_too_short --param pkg=kz-labour-code
./law query producers \
  --param predicate=urn:kz:corpus:clir:labour-code#feeding_break_too_short \
  --param pkg=kz.corpus.labour_code --format json

JSON

// Status: ran via MCP (session server) in the earlier authoring session.
// The boundary case: must stay NEITHER, never "sufficient".
{"tool": "law_ask",
 "arguments": {"kind": "truth", "package": "kz-labour-code",
  "predicate": "feeding_break_too_short", "args": ["Айгуль", "Работодатель"],
  "facts": [
    {"predicate": "child_feeding_break_minutes",
     "args": ["Айгуль", "Работодатель", 30]},
    {"predicate": "children_under_eighteen_months",
     "args": ["Айгуль", 1]}],
  "legalTime": "2026-09-01"}}

For the lost-assumption case, send kind: "why_not" with only the break-length fact at 20 minutes. For the edition verdict, call law_editions with package: "kz-labour-code", dateA: "2025-01-01", dateB: "2026-09-01".

Validation record

Show commands, versions and results

Check	Result	Date
run_evals.py from the checkout root	4 pass, 0 fail, 12 not_run (quoted above)	2026-10-03
test_run_evals.py	4 passed	2026-10-03
law version, jcgm.gum route listing	tool 0.1.1, law.core/0.2; uncertainty-report v0.1.0, 25 steps	2026-10-04
Behavior pass, Muse Spark, public root route	9 pass, 1 partial; same-author self-check, no independent review	2026-10-04
Boundary truth call (NEITHER), why_not on minutes-only facts	re-verified on the public root route (see the record of Read answers)	2026-10-03
Trace chain, date-hash pair, search scores, law_argue refusal, law_editions verdict	earlier MCP session server; not re-run	before 2026-10-03
law_explain expansion, replay command, loci 1 to 3 comparisons	NOT RUN	—
Target domain-profile scenarios, release criteria, metric limits	prescriptive, not measured	—

Previous: Prepare and publish an answer Next: Assist formalization and review