Markdown for LLMs

L4: the charities case bank

The source Markdown for this article. Copy it into your assistant or download it as a text file.

# L4: the charities case bank

**Status:** the protocol is prepared and frozen; the sixteen cases and the
L4 model files are pinned with hashes. Expected outcomes below are
predictions from the statute text and from expectations recorded as
comments in the L4 tests — not results. The comparative run has **not**
been performed: neither the L4 engine nor the Arxo side was executed on
this bank.

This page is the detailed case bank behind the [L4](/comparisons/l4/)
comparison. Its twelve base cases also feed the five-system scenario on
[One charities text, five systems](/comparisons/jersey-charities/).

## The question

The **Charities (Jersey) Law 2014** has been formalized twice,
independently: once as a published L4 model by an outside author, once as
an Arxo package. The experiment asks a narrow question: where the text
uses an evaluative term — public benefit, an analogous purpose, a
connection with Jersey, an empty list of purposes — does the L4 model
compute a value while Arxo returns a request for the deciding body's
judgment, and do the signs and grounds diverge only where that difference
predicts?

The comparison is between two **models** of the Law, not between two
languages. Shortcuts in the published model — a non-empty string standing
for an approved statement, a substring standing for a legal connection —
are choices of that model; the L4 language has other ways to encode them.

## The two arbiters

- **Edition A — the statute alone.** The consolidation "from 16 October
  2025 to Current", published by the Jersey Legal Information Board at
  [jerseylaw.je/laws/current/l_41_2014](https://www.jerseylaw.je/laws/current/l_41_2014),
  pinned as one plain-text extraction by hash and size. All twelve base
  cases use edition A.
- **Edition B — the statute plus a cited secondary order.** The L4 model
  draws on several Jersey orders made under the Law, kept in its own
  reference folder. Edition B adds exactly the order a case cites:
  R&O 144/2019 on the timing of annual returns
  ([jerseylaw.je/laws/current/ro_144_2019](https://www.jerseylaw.je/laws/current/ro_144_2019)),
  R&O 27/2025 on reportable matters, or R&O 60/2018 on the restricted
  section. The order PDFs are recorded by hash only, because the
  publisher's reuse terms were not checked.

## What each side produces

- **L4** — the engine at a pinned commit evaluates the model's test
  queries (the model's tests hold 94 evaluation lines, 93 of them with a
  recorded expected value). Output is the evaluated value with its trace.
  Recorded expectations are reading material: an L4 test run exits
  successfully even when an assertion fails, so only the executed values
  count.
- **Arxo** — the existing Jersey package, version 0.1.0, answers the same
  questions through its own scenarios: a status (true, false, requires
  judgment, or no answer under an unfixed reading), a request for judgment
  naming the deciding body where the text names one, and a proof graph.

The L4 model and its tests are pinned with the experiment under the
model's MIT licence. Two of its test queries, with their recorded
expectations, as they appear there:

```text
#EVAL `all purposes are charitable` completeCharity
-- Expected: TRUE

#EVAL `has Jersey connection` incompleteCharity
-- Expected: FALSE
```

## The base cases

| Case | Situation | Recorded L4 expectation | What the text says | Predicted Arxo answer | Comparability |
|---|---|---|---|---|---|
| C01 | Complete applicant: meets the charity test? | true | the test of Article 5(1) | requires judgment on public benefit; true once the body has answered | comparable |
| C02 | Empty purposes, invalid constitution | false | fails under Article 5(2) | false, on the Article 5(2) ground | comparable; the ground is checked against 5(2), not the empty list |
| C03 | Public benefit on the complete applicant | computed true | to be determined by the body named in Article 7(1) | requires judgment, body named | comparable as a pair: computation against request |
| C04 | Valid public benefit statement | non-empty string, true | needs the Commissioner's approval (Article 8(3)(f)) | boundary | not comparable: string check against approval |
| C05 | All purposes charitable, on an empty list | vacuously true | open reading of Article 5(1)(a) | three answers under three declared readings | comparable once the reading is fixed |
| C06 | Advancement of education | true | listed purpose (Article 6(1)(b)) | true | comparable |
| C07 | An analogous purpose | computed true | "may reasonably be regarded as analogous" (Article 6(1)(p)) | requires judgment | comparable as a pair |
| C08 | Constitution allows government control, plus an exempting order | boolean field | needs "acting in that capacity", and Article 5(3) lets an order disapply it | answer with both conditions | comparable; the ground is checked against the article |
| C09 | Connection with Jersey | true or false by address substring | a legal connection, or the Commissioner's opinion (Articles 11(4)(c), 2(3)) | connection by Article 2(3) or the Commissioner's opinion | not comparable: address string against legal connection |
| C10 | Appeal deadline, timely or not (28 and 56 days) | computed dates and flags | the Law leaves the deadline to an order (Article 36(2)(a)) | boundary: delegated to another instrument | not comparable: numbers with no support in the text |
| C11 | Governor with a spent conviction under this Law | false, under an assumption about spent convictions | reportable "whether or not spent" (Article 19(1)(f)) | reportable | comparable; the text decides |
| C12 | Risk level, "compliant" | computed strings and flags | no such concepts in the Law | refusal: no such question | not comparable: question outside the Law |

Three outcome predictions follow. C01, C02, and C06 are where the two
sides are expected to agree. C03 and C07 are where a computed value meets
a request for judgment — a recorded pair, not a defect. C04, C08, C09,
and C11 are where the model's reading and the text can part ways, so the
article decides.

## The edit cases

Each edit case moves from edition A to edition B and asks how each side
learns which answers changed.

| Case | What changes | Under edition A | Under edition B | How each side learns |
|---|---|---|---|---|
| R1 | Annual-return timing: the Timing Order sets "the period of 2 months following the end of the year" | a number with no support: not comparable | comparable; the model's 60-day window approximates two calendar months | L4: the rule is already written against the order. Arxo: its package states the annual-return duty and declares the deadline delegated to the order; covering it means adding the order and a two-month rule |
| R2 | Reportable matters widen: the 2025 order adds convictions involving vulnerable persons and any unspent conviction | only Article 19(1)(a)–(g) applies | the field widens; a spent conviction involving a vulnerable person becomes reportable by the order | L4: reading the reportable-matter rule against the order. Arxo: seven kinds from the Law, the ordered kind declared as delegated |
| R3 | Solicitation defined: the 2018 order gives a four-limbed definition | the content is left to the order | the model's substring test now differs from the order's definition | L4: reading the solicitation rule. Arxo: declared boundary until the order is added |
| R4 | Financial year end: anniversary of registration or an agreed date | as R1 | the same charity is due or overdue depending on the date chosen | L4: a different fixture parameter. Arxo: the same boundary as R1 |

None of the edit cases touches the evaluative-term pairs of C03 and C07:
the orders do not speak to public benefit or analogy.

## Not comparable, and on what ground

1. An address string is not a legal connection (Article 11(4)(c)).
2. A number with no support in edition A: 28 and 56 days, two months,
   7 and 30 days, risk levels, "compliant".
3. A string feature against an offence with a bearer, a measure, and an
   intent (prohibited words, solicitation).
4. A question outside the Law (risk, compliance).
5. A case comparable only under edition B: under A it is marked not
   comparable, naming the missing order — never scored as a win for
   either side.

## When answers count as the same

Signs and statuses must be strictly equal. There is no numeric tolerance:
the Law itself computes no amounts, and numbers appear only in the
not-comparable controls. A missing fact on the Arxo side and an empty
list or string in the L4 model are recorded explicitly as different
things. A computed value against a request for judgment is a pair to
settle against the article. Explanations — the Arxo proof graph and the
L4 trace — are compared qualitatively, not byte for byte.

## The machine-author axis

The same case bank is also prepared for the machine-author side of the
program: a model writes an L4 model and an Arxo package from the Law's
text alone, with an equal documentation budget, never seeing these cases.
What is measured is the cost of accepting and maintaining the result —
silent errors after convergence, what each system's own check catches,
the cost of the R1–R4 edits — not first-try correctness, which is kept
for calibration only. Because the L4 Jersey model is public, any
resemblance in names or structure is examined as possible recall, and the
Arxo author's copy excludes the existing Jersey package. This axis is
prepared, not run.

## Out of scope

- The full set of 94 test lines; the bank is a twelve-case subset.
- Secondary orders as norms in the base cases (they enter only through
  R1–R4).
- Event-driven simulation of duties over time, beyond single points.
- Performance measurement.
- Changes to the Arxo package or its migration.

Materials: experiments/comparisons/l4/ in the project repository. See
also [L4](/comparisons/l4/), [One charities text, five
systems](/comparisons/jersey-charities/), and the
[methodology](/comparisons/methodology/).