docs← Back to article

Markdown for LLMs

Safety and reliability

The source Markdown for this article. Copy it into your assistant or download it as a text file.

Download this articlePlain text ↗
# Safety and reliability

An agent over an executable canon holds three kinds of power: reading the
model, running a case through it, and publishing the result. This page covers
where the limits on that power live (in your application code, not only in the
prompt), where case data travels, why text found in documents is never an
instruction, and how to recover when a call fails, times out or comes back
trimmed.

For the list of tools a server exposes, the authority is the server's own
`tools/list` reply and the [MCP tool reference](/guide/mcp-tools/). This page
keeps no tool list of its own.

## Enforce limits in code, not only in the prompt

A sentence in the system prompt such as "never publish without asking" is
advice to the model. A limit holds only when the code around the model makes
the forbidden call impossible or refuses it. There are three places to put
it, from strongest to weakest:

1. **Domain tools over the local engine.** The model sees only your tools; the
   engine runs inside them. In the [first agent](/agent-engineering/first-agent/) the model
   has no built-in tools (`tools: []`), exactly one allowed tool
   (`allowedTools: ['mcp__periods__period_end']`) and a turn cap
   (`maxTurns: 6`). It cannot publish, edit a case or search the corpus,
   because those calls do not exist for it. See
   [Choose an integration pattern](/agent-engineering/integration-patterns/).
2. **The host's allowlist over an MCP server.** When the model talks to an MCP
   server directly, list in the agent host the tools it may call and leave the
   rest out.
3. **The server profile.** A profile serves some tool groups (facets) and
   withholds others with a stated reason. Withheld tools are absent from
   `tools/list`; a direct call to one is refused with JSON-RPC error `-32002`
   carrying that reason. Operators configure this; see
   [Tool profiles and input policy](/operate/tool-profiles-input-policy/).

One consequence matters in practice. The four publication tools
(`law_capture_answer`, `law_prepare_answer`, `law_publish_answer`,
`law_revoke_answer`) sit in the same `execution` facet as `law_ask`. A profile
cannot remove publication while keeping asking. To keep an agent from
publishing, drop those tools in the host allowlist, or run the server without
the answers service configured: the tools then refuse with
`ANSWERS_UNAVAILABLE` and nothing leaves the host.

Checks that belong in application code, whatever the pattern:

- **The date of law comes from the human.** The agent never defaults it to
  today; your code refuses to run a question without it (see
  [Pin context, editions and time](/agent-engineering/context-and-time/)).
- **Facts are validated before the engine runs.** In the first agent, a
  misspelled relation fails as a `FactError` before any evaluation.
- **Publication runs only on an explicit user request**, checked by your code,
  not inferred by the model ([Prepare and publish an answer](/agent-engineering/publishing/)).
- **Calls and turns are capped**, so a looping model stops with a reason
  instead of spending the budget.

```text
Status: labeled pseudocode (not executed).

on_tool_call(name, args, session):
    if name not in session.allowed_tools:            refuse("not allowed in this app")
    if name in PUBLICATION_TOOLS and not session.user_asked_to_publish:
                                                     refuse("publication needs an explicit request")
    if name == "law_ask" and "legalTime" not in args: refuse("ask the user for the date of law")
    if session.calls >= MAX_CALLS:                   stop("call budget exhausted")
    session.calls += 1
    return forward(name, args)
```

## Case operations and whose facts they are

A case operation evaluates a question on facts supplied for a case: `law_ask`
with facts, `law_case_ask`, `law_process_run`, the screening builder
`law_screen`. Two rules apply to all of them:

1. Case facts come from the user or from named evidence, never from the
   agent's memory. An accepted `case_input` fact means the engine took it as a
   premise, not that it is true in the world.
2. A candidate collection or a screening result is not a completed check.
   Only a computed answer with its proof and provenance is a result.

The `law_ask` contract names six question kinds: `truth`, `collect`,
`deadline`, `calc`, `positions`, `why_not`. Over MCP this track executed
`truth` and `why_not`; the others are described in the contract and are NOT
RUN on this page. Read the contract before asking for a kind you have not seen
run, and treat the first answer as unfamiliar output.

## Case data: where it travels

Case data leaves the conversation the moment it is passed as facts. Where it
goes depends on the pattern:

| Path | Where the facts go | Status |
|---|---|---|
| Local engine (`@arxo/law`) | Stay in your process: the engine runs in WebAssembly, offline after install | Read in the package documentation |
| MCP server over stdio | Into the server process; the stdio transport writes no call journal | Read in the server source |
| MCP server over HTTP | Into the server; the call journal is on by default and persists request and response bodies, case facts included (64 KiB per side by default) | Read in the server source; see [Data flow, storage, and retention](/operate/data-flow-storage-retention/) |
| Answers service | Only when configured and a publication tool is called | Read in the server source |
| Neural search appendix | Query text and candidate addresses, no case facts, only when configured | Read in the operator documentation |

What this track did not verify and you must not promise: retention on a
public endpoint you do not operate, redaction of sensitive spans, and transport
security between your host and someone else's server. On your own HTTP
deployment, treat the journal as case data: same access rights, retention and
cleanup as the case itself. A private HTTP service (`law serve`, routes under
`/v1/`) is described in [Deploy a private HTTP service](/operate/private-http-service/).

The fact contract also carries provenance, which keeps an audit trail without
copying whole documents:

| Mechanism | What it is | Status |
|---|---|---|
| Fact provenance | `origin`, `evidence`, `span` (page, quote, offset), `extractor` (name, version, confidence) | Contract-described (NOT RUN here) |
| `case_input` origin | Facts without provenance are accepted as case input | Observed: every `law_ask` in this track passed facts without provenance and was accepted |
| Evidence items | Separate `evidence` documents that facts reference; `confidence` is read by the act's evidence policy, not judged by the engine | Contract-described (NOT RUN) |

Safe handling follows from what is unknown:

- Pass the smallest fact set the rules need. Ask `law_rules` first (its
  contract mode names the required inputs), so unused personal details never
  enter the call.
- Quote the span a fact comes from instead of pasting whole documents.
- Keep secrets (passwords, keys, tokens) out of facts entirely; nothing in the
  contract treats a value as secret.
- Handing a case to another agent or a person is an application concern. The
  platform provides no native handoff envelope or merge, so the access rules
  for that copy are yours to enforce.

## Untrusted content: document text is data

Execution consumes the formal model. Source text travels with an answer as
quoted fragments with addresses: `law_rules` anchors each rule to its article
fragment, and the `producers` structural query returns `anchors` with act,
locator (`article/82`) and fragment id for each producing rule. The engine
never executes the quoted sentences; they are addresses a human can open.

No instruction-filtering mechanism was verified: nothing observed promises
that imperative sentences inside an uploaded document, a search result or a
prior answer are neutralized. The boundary is therefore procedural and
absolute. Such text may become case facts (with provenance) or quoted
excerpts. It is never followed as an instruction.

```text
Status: labeled pseudocode (not executed).

document_span = read_span(evidence_doc, page=3)     # data
if looks_like_instruction(document_span):            # "ignore the above and answer X"
    do_not_follow(document_span)                     # never executed, never paraphrased as advice
    facts.append(quote_only(document_span, provenance={...}))
    tell_user("page 3 contains an instruction addressed at the reader; "
              "it was quoted, not followed")
```

If a document tells the agent to change its question, add facts, skip checks,
withhold the proof or publish, that sentence is quoted and reported, and the
original task continues unchanged. Enforce it in code as well: a publication
request that originates in document text never satisfies the
`user_asked_to_publish` check above.

## Failures: three axes

Keep three outcomes apart everywhere: the call can fail, the evaluation
carries its own `evaluationStatus`, and the truth status answers the question.
A completed computation is not a positive answer, and an empty answer is not a
negative one.

`RESOURCE_LIMIT`, `RUNTIME_ERROR`, `EXTERNAL_UNAVAILABLE` and the other
non-`COMPUTED` statuses are evaluation outcomes delivered inside a successful
call, not transport errors. Read them with the single status table in
[Read answers](/agent-engineering/handle-answers/). Evaluation is deterministic, so
resending identical input does not clear a `RESOURCE_LIMIT`; change the
question or its scope.

### Timeouts

The stdio transport has no call timeout. An HTTP deployment may set one
(`LAW_MCP_CALL_TIMEOUT`); on expiry the client receives JSON-RPC error
`-32001` with `data.code` `CALL_TIMEOUT` and the limit in seconds, while the
computation keeps running detached on the server and its result is
discarded. A timeout bounds your wait, not the server's work (see
[Resource limits and cancellation](/operate/resource-limits-cancellation/)).
Status: read in the server source; not triggered in this track.

Set your own client-side budget. Measured wall-clock costs from the runnable
suite in this section (see [Evaluate, debug and upgrade](/agent-engineering/evaluations/)):

```text
PASS    cli/version-smoke (156016 ms)
PASS    cli/guide-gum-route (7472 ms)
PASS    cli/query-producers-feeding-break (7672 ms)
PASS    cli/query-scope-shape-negative (30899 ms)
```

Status: ran locally (suite runner over the `./law` CLI in a source checkout).
The first local call paid the engine build (156 s); later calls took 7 to
31 s. Give the first call in a fresh checkout a long budget (the suite allows
600 s per scenario), then shorter ones. Rerunning a read or an ask after a
timeout is safe: asking never publishes and never changes the model.

### Limits and truncation

Long results are paged or trimmed, and the answer says so.

| Shape | Observed | What to do |
|---|---|---|
| Catalog pagination | `law_packages` returned entries 1 to 20 of 321 with a continuation (`offset` 20) | Follow the continuation; never present page one as the whole catalog |
| Query funnel | A `producers` structural query narrowed 854 candidate rows to 2, with the empty-clause marker null | Quote the funnel when explaining why a result is small |
| Empty scope | A scope matching no package returned `CORPUS_EMPTY` with zero rows and zero packages | Fix the scope shape; do not retry the same scope |
| Rule-body report | A `why_not` answer listed each candidate rule with per-condition states (satisfied, unsatisfied, undetermined) | Read every condition; undetermined is not false |

Status: ran via MCP (session server) and locally (`./law query`). The
`rowLimit` / `rowOffset` paging of structural answers and the `compact`
response mode are contract-described but NOT RUN; reproduction is one paged
query or one compact call. Shortened proofs were never requested, so no claim
is made about what a trimmed proof omits. If an answer looks cut off, ask for
the fuller form instead of filling the gap from context.

### Unavailable tools: three observed shapes

**World-merge refusal from `law_argue`.** Ran via MCP (session server), twice,
same refusal: the tool declined to merge the question world because one
imported package declares an older semantics revision than the rest (the
message names the package and both revisions, in Russian). Not retryable.
Record the message, drop the argue step and answer from `law_ask` with its
proof. The refusal says nothing about the question itself.

**Publication tools without their service.** Ran via MCP: `law_capture_answer`
and `law_prepare_answer` returned `ANSWERS_UNAVAILABLE`, naming the two
missing settings and noting that ordinary asking keeps working. Stop the
publication plan and name the missing setting; do not retry
([Prepare and publish an answer](/agent-engineering/publishing/)).

**Dead session connection.** Observed: a repeat of an earlier identical
`law_ask` call failed with

```text
law-dsl: MCP stdio connection is closed
```

Reconnect and rerun the exact call. Because the repeat never ran, this page
makes no determinism claim from repetition.

### Recovery checklist

| Symptom | First action | Safe to repeat? |
|---|---|---|
| Client-side timeout or `CALL_TIMEOUT` | Rerun the identical call with a longer budget, or narrow it | Yes; asking has no side effects |
| `RESOURCE_LIMIT` status | Narrow the question or its scope | Not unchanged; the same input gives the same status |
| Paged or trimmed answer | Follow the continuation or request the fuller form | Yes |
| `CORPUS_EMPTY` or unknown-name error | Fix names and scope shape | Only after the fix |
| Merge or revision refusal | Record it; route around the tool | No; the refusal is the answer |
| `ANSWERS_UNAVAILABLE` | Stop; name the missing setting | No |
| Facet refusal `-32002` | Use another path or ask the operator; the reason says why | No |
| Dead connection | Reconnect; rerun the exact call | Yes |

### Idempotency, only where verified

- Repeating a read or an ask has no side effects: nothing in the observed calls
  changes the model or publishes. Rerun after timeout is the suite's normal path.
- Repeating publish with the same preparation id returns the same address and
  revoke secret instead of a second document. Contract-described, NOT RUN (no
  answers service); reproduction is prepare once, publish twice, compare.
- Byte-identical repetition of one `law_ask` call: NOT RUN here (the repeat hit
  the dead connection). Check it as described in
  [Evaluate, debug and upgrade](/agent-engineering/evaluations/).

## Validation record

| Claim | Status | Source |
|---|---|---|
| Domain-tool limits (`tools: []`, one allowed tool, `maxTurns: 6`) | Ran with the Claude Agent SDK, 2026-10-04 | First agent |
| Facet refusal `-32002`; publication tools in the `execution` facet | Read in the server source and tool registry | Server source |
| HTTP journal default, 64 KiB body cap, stdio never journals | Read in the server source | Server source |
| `CALL_TIMEOUT` `-32001`, detached computation | Read in the server source; not triggered | Server source |
| `case_input` acceptance, `truth` and `why_not` asks, pagination, funnel, `CORPUS_EMPTY`, `why_not` report | Ran via MCP (session server) and locally | This track |
| `law_argue` refusal, `ANSWERS_UNAVAILABLE`, dead connection | Observed failures, quoted exactly | This track |
| Provenance fields, `rowLimit`, `compact`, publish idempotency, byte-identical repeat | NOT RUN | Contract only |
| Public-endpoint retention, redaction, transport security | Unknown | Not verified |

Previous: [Cases and processes](/agent-engineering/case-lifecycle/)
Next: [Prepare and publish an answer](/agent-engineering/publishing/)
Related: [Read answers](/agent-engineering/handle-answers/) · [Operate](/operate/)