docs← Back to article

Markdown for LLMs

Troubleshooting

The source Markdown for this article. Copy it into your assistant or download it as a text file.

Download this articlePlain text ↗
# Troubleshooting

## Task and place

Task: turn a symptom — a wrong answer, a refusal, a failed review — into a cause with a discriminating check and a fix. Place: companion to every stage; open it when a check fails and return to the failing stage with the diagnosis. Each symptom below lists causes first, then the checks that tell them apart, then fixes.

## Inputs

- The failing package and the exact failing output, kept verbatim.
- A working `law` tool, version 0.1.0, semantics law.core/0.2, plus a plain Python 3 for the record checker.
- The pinned bytes, the scope card, and the decision records the package claims.
- Scratch room outside the repo for probes that must not touch the package.

## Actions

### Wrong result

A check derives a value the pinned source contradicts — a wrong premium, a payout that should stay silent.

- Causes: a wrong table row or rate; a flipped bound; a misread formula such as a percent taken as a fraction; a stale expectation the act never supported.
- Checks: run the suite and read which checks fail. In a scratch copy of the EAI package with the payout bound flipped from thirty to thirty-one, the bound-reading checks failed while the rest stayed green; `[...]` marks omitted lines:

```text
$ law test /tmp/eai-mutant
  FAIL [kz.corpus.employee_accident_insurance#authored] tests/core.lawtest / EAI-LOSS-THIRTY-QUALIFIES
        truth_status == TRUE_ONLY: в документе NEITHER
[...]
  FAIL [kz.corpus.employee_accident_insurance#authored] tests/goals.lawtest / GoalEstablishedCapacityLossHasPayout-world
        truth_status == TRUE_ONLY: в документе NEITHER
  FAIL [kz.corpus.employee_accident_insurance#authored] tests/goals.lawtest / urn:kz:corpus:clir:employee-accident-insurance#GoalEstablishedCapacityLossHasPayout
        контрпримеры ["urn:kz:eai:goal:worker"]; пропущено []
итого: 7 проверено, 4 прошли, 3 не прошли, 0 не исполнены; код 1
```

[translation] The detail lines read "in the document: NEITHER" and "counterexamples […]; skipped []"; the summary line reads: seven checked, four passed, three failed, zero unexecuted, code one.

A single failing premium check points at one tariff row; failures clustered on bound-reading checks point at the bound, as here.

- Fixes: correct the row, bound, or formula and re-run to green. If the model is right and the expectation wrong, justify the new expectation on the affected record — never edit an expectation silently to match a run.

### Silence

A question that should resolve stays undecided — expected established, observed neither established nor refuted.

- Causes: a strict guard left unmet, such as loss twenty-nine against a thirty bound; a missing input fact the rule waits on; a context set on the wrong time axis so the rule never sees the case.
- Checks: run the suite and compare the boundary pair. The small fixture run below shows the healthy shapes — established, denied, conflict, and silence each pinned by its own check:

```text
$ law test docs/handbook/files/fixtures/boundaries
law test handbook.boundaries: мир handbook.boundaries
  ok   [handbook.boundaries#scenarios] tests/boundaries.lawtest / MEMBER-ESTABLISHES-ELIGIBILITY
  ok   [handbook.boundaries#scenarios] tests/boundaries.lawtest / BARRED-DENIES-ELIGIBILITY
  ok   [handbook.boundaries#scenarios] tests/boundaries.lawtest / MEMBER-AND-BARRED-IS-CONFLICT
  ok   [handbook.boundaries#scenarios] tests/boundaries.lawtest / SILENCE-IS-UNKNOWN
  ok   [handbook.boundaries#scenarios] tests/boundaries.lawtest / SCORE-THIRTY-QUALIFIES
  ok   [handbook.boundaries#scenarios] tests/boundaries.lawtest / SCORE-TWENTY-NINE-STAYS-SILENT
итого: 6 проверено, 6 прошли, 0 не прошли, 0 не исполнены; код 0
```

[translation] The header line names the tested world; the summary line reads: six checked, six passed, zero failed, zero unexecuted, code zero.

In EAI the insurer side follows the same pattern: loss thirty established against loss twenty-nine undecided, per the boundary expectation EAI-D2 records. [Source snapshot]: the missing employer-side conclusion for losses five through twenty-nine is not intended silence — that gap is finding EAI-R1, open. [Accepted candidate]: the gap is closed by the employer-branch rule with its two probes.

- Fixes: supply the missing fact, repair the guard or the context, or — when the act genuinely does not speak — record the silence as intended in a decision record and pin it with a straddling pair.

### Conflict

Two readings fire at once and the answer carries both sides.

- Causes: overlapping guards deriving opposite conclusions; a case that genuinely sits under two norms with no priority recorded.
- Checks: the fixture run above carries the discriminating check — MEMBER-AND-BARRED-IS-CONFLICT feeds both facts at once and expects both-sides support. The EAI package has no such check because no two of its rules can support opposite sides of one question under the stated input restrictions: each employer carries one risk class (the `key(e)` on the class fact), each worker one loss percent (the `key(w)` on the loss fact), so at most one tariff guard fires per employer — and no EAI rule derives an explicit negation at all (`insurance_payout_barred` is declared but never produced). `strict` alone would not exclude conflicting support; the exclusion comes from those keys plus the model's invariants (one class per employer, one loss per worker, disjoint class guards, no denying rules). If your package can meet itself, add one both-facts probe before touching rules.
- Fixes: narrow the guards so each case reaches one rule, or record the priority in a decision record; re-run the both-facts probe to confirm.

### Non-executable part

The static check refuses the package before any scenario runs.

- Causes: a syntax slip in a rule file; an unknown construct; a sketch pasted into rule files as if it ran.
- Checks: the static check names the file, the line, and the diagnostic. Observed on a scratch copy with a broken rule appended (`rule Broke1({ ...` on line 59):

```text
$ law engine check /tmp/eai-broken
/tmp/eai-broken/08-17-19-core.law:59:13: error LDC-E0201: ожидалось имя (имя переменной), найдено "{"
```

[translation] The diagnostic reads: expected a name (variable name), found "{".

- Fixes: repair the named line and re-run to OK. Keep sketches labeled non-runnable and out of rule files; only checked-in rules may claim to run.

### Source mismatch

The bytes no longer match the pin, or a fragment no longer occurs in them.

- Causes: bytes edited after pinning; a re-download that swapped new bytes under a recorded hash; a fragment text edited without updating its hash.
- Checks: the static check refuses with a hash diagnostic naming the declared and actual hashes. Observed on a scratch copy with one appended byte:

```text
$ law engine check /tmp/eai-tamper
/tmp/eai-tamper/01-sources.law:7:195: error LDC-E5204: publication "EAI_RU_TEXT": байты "sources/employee-accident-insurance/ru.txt" не дают объявленный content_hash — §194 publication document hash mismatch (объявлен sha256:4cd311a9a3316f3255992116108755f2073aaaaf12b3d41e3c7a1c598c2d3cdb, фактический sha256:ca820a93596763f486f8c0f1c5cb59b7fca0598c853770d175d32626860c9981)
```

[translation] The diagnostic reads: the bytes do not yield the declared content hash — publication document hash mismatch (declared sha256:4cd3…, actual sha256:ca82…).

A recomputed hash confirms from outside the tool. Good bytes from the repo root:

```sh
$ shasum -a 256 corpus/laws/kz/laws/employee-accident-insurance/sources/employee-accident-insurance/ru.txt
4cd311a9a3316f3255992116108755f2073aaaaf12b3d41e3c7a1c598c2d3cdb  corpus/laws/kz/laws/employee-accident-insurance/sources/employee-accident-insurance/ru.txt
```

Tampered bytes from the scratch copy:

```sh
$ shasum -a 256 sources/employee-accident-insurance/ru.txt
ca820a93596763f486f8c0f1c5cb59b7fca0598c853770d175d32626860c9981  sources/employee-accident-insurance/ru.txt
```

The recomputed good hash equals the pinned publication hash; the tampered one does not.

- Fixes: restore the bytes from the preserved original, or re-acquire as a new edition with its own journal line. Never re-pin new bytes silently under an old hash.

### Dependency clash

Imports resolve against more than one version, or the lock no longer matches the manifest.

- Causes: manifest and lock drifted apart; two wanted versions claimed at once; a local import renamed without updating its users.
- Checks: read the lock edges first. EAI declares zero dependencies — read excerpt from its lock file (inspected text, not a run):

```json
"dependencyEdges": [],
"packages": [],
```

A package with no edges cannot clash; a package with edges clashes exactly where two entries disagree. The static check then confirms: it resolves every import and reports OK on the good EAI package, as shown in the worked example.

- Fixes: settle the manifest on one version per import, update local import lists to match, and re-run the static check until it reports OK.

### Failed review

The records do not support release: a finding without disposition, an incomplete record, or drift between card and model.

- Causes: a finding left open with no fix-or-defer row; a record missing a required heading; scope widened in the model without reopening the card.
- Checks: run the field checker over records. The full EAI set passes thirteen of thirteen (see the templates page, which owns that count); a scratch review copy with its verdict heading removed fails loudly:

```text
$ python3 docs/handbook/files/check_templates.py /tmp/eai-records-broken/filled/eai-review-agent.md
FAIL /tmp/eai-records-broken/filled/eai-review-agent.md: missing ## Verdict
total: 1 checked, 0 passed, 1 failed
exit=1
```

Shape green but verdict uneasy means the defect is semantic — re-read the model against the pinned fragments rather than re-running tools.

- Fixes: complete the record, resolve or defer each finding with a reason, reopen the card when scope moved; re-run the checker to green and record the confirming run on the review record.

## Decisions

- Model or expectation: when a scenario fails, decide from the pinned source which one is wrong before changing either.
- Fix now or defer: a finding that blocks release is fixed with a confirming re-run; anything else is deferred with a reason on the review record — EAI-R1 and EAI-R3 show the fixed shape (closed in the accepted candidate with re-runs), EAI-R2 shows the open shape (0.1.1 divergence kept as a compatibility note with the support decision 0.1.0 only).
- Restore or re-acquire: mismatched bytes are restored when the pin is right and re-acquired as a new edition when the world moved.

## Artifact

The diagnosis log: one line per symptom tried — symptom, discriminating check, outcome, fix — appended to the affected record. A review record with resolved findings plus a green re-run is the release half of this artifact; the templates page holds its shape.

## EAI example

EAI exhibits three symptoms by probe and avoids the rest by construction. The synthetic bound flip shows the wrong-result shape with three failing bound-reading checks. The tamper probe shows the source-mismatch refusal with its hash diagnostic. The broken-rule probe shows the non-executable refusal naming file and line. Insurer-side silence below thirty is intended and pinned by the thirty-versus-twenty-nine pair; conflict is absent under stated input restrictions and model invariants (see the Conflict row above), not by `strict` alone; dependency clash is impossible with zero dependencies; review carries EAI-R1 and EAI-R3 closed in the [accepted candidate] (thirty-seven of thirty-seven) with EAI-R2 open as a compatibility note — the [source snapshot] phrasing with one open finding is history, kept on the pre-R3 review record.

## Pitfall

Treating the probe as the fix: a scratch copy that reproduces the failure proves the diagnosis but changes nothing. Apply the fix to the real package and its records, re-run the discriminating check there, and discard the scratch. The twin failure is re-downloading source bytes to "refresh" them, which manufactures a mismatch under an already recorded hash.

## Verify

No new runs are needed beyond the probes above. Criterion per symptom: after the fix, the discriminating check flips — the failing scenario passes, the refusal clears to OK, the recomputed hash equals the pin, the checker reports green — and the full suite returns to its accepted state (for EAI: static check OK on the [accepted candidate], thirty-seven checked with thirty-seven passed and zero failed; the [source snapshot] stays at seven of seven as the frozen before-state). A fix that greens one check while reddening another is not automatically a scope move: classify the new failure first — regression in the fix, a wrong expectation the source contradicts, a genuine meaning change, or an environment difference — and reopen the card only when the classification establishes that the promised scope moved. A wrong expectation is corrected with an independent justification from the source, exactly as the semantic review separates controls from errors; an environment difference is recorded on the run, not on the card.

## Limits

These probes diagnose the package, not the act: a green suite never proves the model matches the source, and a clean bill of records never proves the findings were the right ones. Multi-act bridges, corpus-wide moves, and disputes about what the act means rise above symptom repair into maintainer review.

## Next step

Return to the [handbook index](/handbook/) and continue along your route — every role re-enters the pipeline through the workflow map.

## Sources

- [End-to-end worked example](/handbook/worked-example/)
- [Record templates](/handbook/templates/)
- [Semantic review](/handbook/semantic-review/)
- [Writing tests](/tutorials/writing-tests/)
- [Four states of support](/tutorials/four-states/)
- [Command line](/cli/)
- [Diagnostics](/diagnostics/)