Evaluation

How an answer is built, and how three ways of answering compare on the same questions.

The full write-up is on GitHub →

Terms used on this page

Provision: An addressable piece of the law: an article, a numbered paragraph, or an annex.
Original / Amended 27 Jul 2026: The Act as first published, or as the Digital Omnibus changed it on 27 July 2026, moving two of the three compliance deadlines.
Reference traversal: Following the Act's own citations outward from what search found, to reach provisions a similarity match would miss.
High risk: A classification carrying a chain of duties that other systems do not take on.
General-purpose AI model: A model trained for a broad range of tasks rather than one narrow purpose, carrying its own obligations under the Act regardless of what is built on it.
Prohibited practice: A use of AI the Act bans outright, such as manipulation that causes harm, rather than one it regulates.

How an answer is built

Five stages, in the order a question passes through them.

Intake

a description becomes a structured query

Retrieve

search finds candidate passages by meaning

Traverse

the Act's own citations expand outward

Synthesize

an answer is drafted, citing what it read

Verify

every claim is checked against the text

The comparison switches these two on and off.

The three-arm comparison

The Act fits inside a current context window, so a model can read the whole document and answer from it. That makes retrieval something to justify rather than assume, so the same twelve questions were answered three ways: reading the whole Act, search alone, and search with reference traversal, then scored the same way on what each arm read and what it cost.

Arm	Text	Recall	Faithful	Nodes	Refused
Full context	Amended	1.00	0.97	133.0	2/3
Full context	Original	1.00	0.98	306.0	0/3
Search only	Amended	0.71	0.93	11.9	1/3
Search only	Original	0.61	0.91	12.0	1/3
Search + traversal	Amended	0.81	0.92	26.0	1/3
Search + traversal	Original	0.83	0.91	22.8	0/3

Faithfulness is matched against what an arm was supplied, so the baseline is compared against its own document-wide vocabulary rather than the dozen or so nodes the retrieval arms read. Refused is scored over three questions the text does not settle, per arm and text: too few to rank the arms on, and read here as a direction rather than a rate.

Run on one RTX 5090. The baseline generates with annex-longctx; search and traversal generate with annex-qwen3-27b over an index built with snowflake-arctic-embed2. Nothing here was paid for, so every cost figure is a projection at a stated hosted rate rather than a bill.