Five pairs, no observed gain.
Our previous sensitive-context evaluation included a correct phishing review whose quote omitted the unfamiliar sign-in page. Exact extraction had worked; complete evidence support had not been independently assessed. That motivated a separate evidence-selection change and a fresh paired experiment.
The new comparison planned 20 fictional requests, each tested with the previous sensitive-context prompt and an updated completeness prompt. It completed only five pairs before the provider returned HTTP 429 on call 11.
Both prompts matched all five observed routing labels. A separate blinded AI reviewer judged all ten valid quotes supportive of their returned decisions and complete against the frozen reference facts. There was no observed routing or evidence-quality gain on those five pairs. The remaining cases cannot be credited or used to infer improvement.
Ask for decisive evidence.
The optional createCompletenessAdapter appends evidence-selection instructions to the previous sensitive-context policy. It asks the model to quote the decisive context or relationship, rather than a generic symptom or isolated keyword. For a protective review condition, the useful span should carry the condition that overrides an ordinary support route.
Routing still comes first under the same policy. The model must not invent a protective condition or change a route because another quote is easier to select. Evidence remains one unchanged, contiguous quote of 4–120 UTF-16 units from either the subject or body. It cannot stitch fields together, paraphrase or repair text.
If necessary context will not fit that historical contract, the instruction asks for the strongest valid span without claiming completeness. This experiment does not solve general multi-span explanations. The validator, rules baseline, v9 routing policy and bounded transport remain unchanged; the new adapter is explicitly selected. A prompt change can still affect routing, which is why both routing and evidence quality are measured.
Freeze the comparison.
The implementation was frozen before the implementer inspected the fresh cases. One AI agent authored 20 fictional English requests, four in each of five groups: security context, sensitive scope, multiple-intent scope, ordinary evidence and protective scope. Cases were shuffled before IDs were assigned. No exact normalized body was reused from nine older sets; that is not proof of conceptual novelty or training-data exclusion.
A separate AI label reviewer first saw only the requests and written policy. It saved initial decisions and evidence criteria before seeing the author's labels or assigned groups. Initial routing agreement was 20/20. All 20 evidence criteria were reconciled, with three refined before live calls and no unresolved routing disagreements. Thirteen reference labels require review; seven permit an ordinary route.
The criteria identify minimum reference facts, with example quotes used only to assess whether a single valid span is feasible. Examples are not answer whitelists. Author proposals, initial reviewer decisions, final labels, criteria resolutions and hashes remain inspectable. Separate AI contexts are not independent human adjudication. Shared training and policy assumptions can produce correlated agreement.
Both prompts used Gemini 3.5 Flash Lite, the same exact-quote mode, 30-second attempt deadlines and at most two attempts within 65 seconds per classification. Calls alternated arm order by case position, with 15-second spacing between classifications. Twenty cases × two prompts × at most two attempts set an 80-attempt ceiling. Labels, groups and rationales never entered model requests.
Review the evidence separately.
After the run, a fresh AI reviewer received ten valid final outputs in a separately shuffled order with pseudonym IDs. Each item supplied the full request, frozen expected decision, minimum facts and returned category/quote. The reviewer saw the routing policy and a rubric written before the calls. It did not see the example quotes, prompt arm, call order, run metadata, aggregate scores or unblinding map before saving its initial grades.
Two judgments were kept separate: whether the quote supports the returned decision, and whether the quote itself covers every frozen reference fact. A wrong route could quote a complete true trigger; a correctly routed request could use incomplete evidence. Mechanical routing match and a joint route-plus-support-plus-coverage measure were computed separately.
The rubric warns that reference criteria can demand more than the smallest sufficient protective trigger. A missing extra reference fact must not automatically make decision support false. Reviewers can flag ambiguous or over-demanding criteria without changing the primary labels after seeing results.
All ten supplied quotes received positive support and reference-coverage judgments; no ambiguities were flagged on this observed subset. Grades were saved and hashed before the implementer unblinded the map. Repeated requests can reveal pairs, and wording style may reveal a prompt. Blinding limits cues; it does not make AI judgments ground truth.
Stop at the rate limit.
The first ten calls completed once and produced valid outputs. Updated case v10-006, an ordinary technical request involving explicitly invented medical notes, returned HTTP 429 at the HTTP stage after 180 ms. No model decision was available. No Retry-After value was present.
The predeclared quota-stop rule ended the batch. The previous-prompt call for that case and both prompts for v10-007 through v10-020 were never attempted: 29 planned classifications. The run used 11 actual attempts out of the 80-attempt ceiling, with no retries, quote/schema failures, recovered calls or unknown in-flight attempts. The workflow exited with failure and retained its checkpoint.
Operational review fallback is not a successful classification. The failed call remains uncredited. We did not retry it, switch projects or credentials, change billing or bypass quota. The sanitized response does not establish a short rate limit, daily quota exhaustion or a reset time. Completing coverage would require a separately bounded continuation after access recovers, preserving this original failure.
Keep both denominators.
| Measure | Previous prompt | Updated prompt | Rules |
|---|---|---|---|
| Available outputs / all cases | 5/20 | 5/20 | 20/20 |
| Routing exact matches / all cases | 5/20 | 5/20 | 10/20 |
| Routing on common-valid cases | 5/5 | 5/5 | 4/5 |
| AI evidence support / all cases | 5/20 | 5/20 | Not graded |
| AI reference-fact coverage / all cases | 5/20 | 5/20 | Not graded |
| Joint route + support + coverage | 5/20 | 5/20 | Not graded |
The all-20 measures retain completion and failures. They are not estimates of model accuracy on the unattempted cases. The common-valid view describes only five cases and excludes 15. Both views matter; neither completes the planned experiment.
The observed subset contains one billing request and four required-review requests. Both prompts reviewed all four and made one correct billing proposal. First and final views are identical. Both derived guarded views also match five cases, without additional model calls. Rules matched four of these five, missing review on actual medication-history context.
Four paired quotes were identical. On the fifth, the previous prompt quoted “My real overdue loan balance”; the updated prompt added “is on the statement”. Both passed the frozen evidence criteria. More words did not create a measured gain. There were no observed model review misses among valid outputs, but nine required-review cases per prompt remain unmeasured. The full rules baseline matches 10/20, catches 7/13 review labels and adds four unnecessary reviews; every mismatch is retained in the report.
What remains unmeasured.
The stop occurred before every security-context case and every ordinary-evidence control. Broad security, injection, negation or ordinary-routing claims would be unsupported. Five completed pairs, a fixed alternating order and one provider window do not establish stable correctness, availability or a causal evidence-selection benefit.
The cases are policy-directed, synthetic and English. Labels and semantic judgments were AI-authored or AI-reviewed, without human adjudication or real customer traffic. Single-span feasibility was assessed by AI. Frozen reference-fact coverage is not proof of the uniquely minimal explanation. Longer or discontiguous evidence is outside this unchanged contract.
Previous successful attempts recorded 6,633 total tokens; updated successful attempts recorded 8,167. Usage on the failed 429 is unknown, not zero. The longer updated prompt is an execution confound. Latencies and all attempt metadata remain in the raw record; this partial run supports no latency-gain claim.
All 90 software tests passed locally and in GitHub Node 22/24 validation. Frozen implementation bytes and sealed review hashes were verified. Those checks establish software behavior and provenance, not semantic improvement. No labels, criteria or prompts were tuned after results. No new model calls were made for this article. No customer action was executed, and the optional adapter is not established as production-ready.
Inspect the records.
- Full partial findings, every routing result and observed quote
- Raw attempts, diagnostics and every unattempted row
- Derived routing, semantic, usage and paired measures
- Blinded output items, initial grades, review process and unblinding map
- Fresh requests and frozen criteria, label-review provenance and predeclared grading plan
- Implementation freeze, optional adapter and semantic review rubric
- Original failed workflow and unchanged rules-only interactive lab
Publication note: AI-drafted and checked against frozen experiment records. AI review is not human adjudication. The original quota-limited run and all historical failures remain preserved.
Follow-up: completed coverage, recovered failures and evidence criteria needing human review.
← Back to the journal