A stronger boundary is not an observed gain.

Our previous paired evaluation included quotes that failed the exact-substring and length checks. We added an alternative to free-form quote generation, then compared both modes on a fresh set of 40 fictional requests.

The new indexed mode matched 35/40 decisions. Existing quote mode matched 36/40. Neither mode had an evidence-contract rejection. All seven failed calls returned HTTP 503. On the 33 cases where both modes produced valid output, their decisions were identical, including one request that should have gone to review.

This run does not demonstrate improved observed live evidence-contract reliability or semantic accuracy. It also does not demonstrate a semantic regression: the one-point all-case score difference comes from different failed calls. The prior set’s quote failures and this new set are not a controlled before-and-after comparison.

Select a source reference, then validate locally.

Quoted mode asks the model to return a category and an unchanged 4–120-character passage from the subject or body. Indexed mode instead supplies request-local excerpt references and asks the model to return a category and an evidence ID.

  1. The adapter partitions each input field into consecutive exact passages, normally up to 96 JavaScript UTF-16 code units. A short remainder may extend a passage to 99 units; surrogate pairs are not split.
  2. The provider schema restricts the evidence ID to an enum of references available for that request. The complete request and its excerpts remain untrusted data.
  3. The adapter checks output shape, reference membership and category locally. It resolves the chosen ID to source text and checks the same exact-quote contract. Successful records retain the source field and offsets.

The adapter does not trim, normalize, truncate or repair a generated quote. Both comparison modes now reject a quote that spans the synthetic newline between subject and body; evidence must belong to one original field. Indexed mode is explicit opt-in. Quoted mode remains the compatibility default, and the clarified routing policy and rules baseline are unchanged.

The provider schema does not replace local validation. An unknown reference still fails closed. If neither input field supplies at least four code units, indexed mode stops before making a provider call rather than inventing evidence. Fixed excerpts can cut meaningful clauses or include hostile text. Exact extraction does not establish that the passage supports the category, contains enough context or is the best explanation.

A fresh set, with separate AI label review.

A separate AI author created 40 new fictional English support requests using the written routing policy, without inspecting the adapter, earlier cases or provider results. A fresh-context AI reviewer independently saved labels before opening the author’s proposed labels. Initial agreement was 40/40, with no unresolved disagreements or replacements.

This was AI review, not human adjudication. The implementer froze the adapter and runner before inspecting the fresh cases. Hash-bound originals preserve the author’s inputs and labels, the reviewer’s initial labels and the final decisions. Checks found no exact body reuse from four older sets. They do not establish conceptual novelty or absence of analogous provider-training material.

There are five cases in each of eight strata: signup, account closure, ordinary access, billing, technical, security, injection and contextual requests. The reviewed expectations contain 22 routable cases and 18 review cases.

Both model modes used Gemini 3.5 Flash Lite on the same 40 requests. Call order alternated by case. Labels, strata and rationales were withheld from model requests. Each mode made 40 calls, one attempt per call, without retries and with 6.5-second spacing. The existing unbilled project and repository secret were reused without a billing upgrade. Zero business actions were executed.

Indexed mode changes evidence instructions, schema and the supplied excerpt options together. This tests two integration designs rather than isolating one prompt sentence. The original run completed all 80 calls. GitHub marks its job failed because provider or contract errors deliberately produce a nonzero exit; the saved report is complete. No second evaluation was submitted.

Every failed call stays in the denominator.

Decisions on the same 40 fictional requests
MeasureRulesQuotedIndexedGuarded indexed
Exact matches / 4022 (55%)36 (90%)35 (87.5%)28 (70%)
Valid decisions / 4040373636
Provider errors / 400344
Local decision-contract rejections0000
Expected review caught / 1810151617
Unnecessary valid review8008
Matching route proposals / proposals12/2221/2219/2011/11

Failed calls receive no correctness credit, including when operational fallback sends them to review. The valid-output counts are 37/40 for quoted mode and 36/40 for indexed mode; neither is a 40-case record of successful classification.

Both modes made identical decisions on all 33 common-valid cases, with 32/33 matches. That conditional subset helps distinguish routing behavior from availability, but excludes seven failures and must not replace the all-40 scores. Three cases had a quoted failure and an indexed success; four had a quoted success and an indexed failure. Those differences do not establish semantic corrections or regressions.

The guarded mode preserves rules abstention and sends disagreements to review. Its 11 proposals all matched the labels, but it unnecessarily reviewed eight routable cases. Proposal precision alone hides that loss of coverage.

All seven failed calls, and the shared miss.

Every quoted or indexed failure or routing mismatch; IDs start with v5-
CaseExpectedQuotedIndexed
011accessaccessHTTP 503
015accessaccessHTTP 503
021technicalHTTP 503technical
024technicaltechnicalHTTP 503
029reviewHTTP 503review
033reviewHTTP 503review
036reviewreviewHTTP 503
038reviewtechnicaltechnical

Quoted mode had HTTP 503 failures on v5-021, v5-029 and v5-033. Indexed mode had them on v5-011, v5-015, v5-024 and v5-036. Each stopped after its first attempt. There were no local decision-contract rejections, model-JSON failures or fabricated-reference rejections in either mode in this run. The report preserves bounded HTTP diagnostics and attempt timing for every failed call.

Case v5-038 exposes a different failure:

Something is wrong with the service. Can someone sort it out?

Both AI labelers assigned review because the request did not establish one specific supported intent. Both model modes proposed technical. Quoted mode selected “Something is wrong with the service.” Indexed mode selected the full body. Both passages satisfied the mechanical contract; neither decision followed the declared review policy.

The guard caught this miss by preserving the baseline abstention. That does not erase its eight unnecessary review decisions elsewhere. The full report also lists every rules-baseline and guarded mismatch, not only the model failures shown here.

What the result can—and cannot—support.

The design removes freely generated evidence text from indexed responses and verifies request-local provenance. The live comparison did not show fewer contract errors, because both arms had zero. A zero observed count in 36 or 37 valid responses is not proof of a zero failure rate.

Excerpt relevance and completeness were not independently scored. The reviewed labels cover routing decisions, not whether a selected quote is sufficient support. This is not an evidence-quality benchmark. The shared review miss demonstrates why a valid source reference cannot substitute for a correct decision.

Separate AI contexts and a label-blind first pass reduce direct anchoring. They do not eliminate correlated blind spots from shared model architecture, training or interpretations of the policy. Forty-of-forty label agreement does not establish ground truth. Human adjudication remains missing.

This is one small, policy-specific synthetic set, with five cases per stratum. It cannot estimate representative customer error rates, service reliability, production readiness or general resistance to injection. The returned alias was gemini-3.5-flash-lite in both arms; an alias is not an immutable backend snapshot. Different failures at different call times are not a controlled provider-availability experiment.

The set is now seen. Tuning against it and rerunning would not create untouched evidence. Earlier articles use different datasets and cannot be combined into an accuracy trend. Indexed mode remains opt-in; this result does not justify promoting it to the default.

Extra structure has an input cost.

Provider-reported usage and attempt latency
MeasureQuotedIndexed
Known total tokens19,92524,976
Attempts with usage / 4037/4036/40
Median attempt latency798 ms875.5 ms
Nearest-rank p952,512 ms1,640 ms

Indexed mode supplies both the original request and excerpt options. Its known token total was higher here, despite one fewer attempt with usage metadata. Missing HTTP-failure usage is unknown, not zero; these totals are not a complete cost or billing statement.

Attempt timings exclude deliberate spacing and include failures. One run does not establish a latency advantage or a service guarantee. We would need repeated measurements and an appropriate cost model before making operational claims.

Inspect the frozen record.

All 44 software tests passed locally and in Node 22/24 validation. Saved metrics were recomputed offline, and each successful indexed reference was checked against its source field, offsets and exact excerpt. These checks validate implementation and provenance; they do not establish semantic accuracy.

The next useful evidence would include human-adjudicated routing and excerpt-support labels on a further unseen set, plus repeated availability measurements. The finding today remains narrower: bounded extraction is implemented, but this run did not demonstrate a live contract-reliability gain.

Publication note: this article was AI-drafted and checked against the frozen experiment artifacts. Evaluation labels were AI-authored and separately AI-reviewed as described above; no human adjudication is claimed.

Next experiment: vague requests, three routing corrections and the remaining review error.

← Back to the journal