Complete the untouched slots.
Our evidence-completeness evaluation stopped at HTTP 429 after ten valid outputs across five paired cases. One updated-prompt call failed, leaving 29 planned classifications unattempted. The original partial result showed no observed gain on its five completed pairs.
A separate continuation has now tested only those 29 untouched slots: 15 previous-prompt and 14 updated-prompt classifications. It did not rerun the original failed updated v10-006 or replace any original valid output. These are classifications from the same 20-case set, not 29 new cases.
All 29 final routes matched the frozen labels, versus 12/29 for rules and 22/29 for the derived guarded view. The continuation used 31 attempts. Across both windows, all 40 planned slots have been attempted, 39 have valid final outputs, and 42 of the permitted 80 attempts were used. The original unavailable output still earns no routing or evidence credit.
The prompts, model, routing policy, labels, evidence criteria, exact-quote validator and bounded transport were unchanged. Calls used Gemini 3.5 Flash Lite, 30-second attempt limits and at most two attempts within 65 seconds per classification. Spacing increased from 15 to 30 seconds. The first selected call checked access without an additional probe. The provider accepted calls again, but later timing and slower spacing do not isolate what caused or cleared the earlier rate limit.
Recover a call, retain its failure.
Two first attempts in the previous-prompt arm returned HTTP 503. On v10-010, the response failed after 499 ms; a single retry after 4,067 ms recovered a correct billing route and a quote judged supportive and reference-complete. On v10-013, the response failed after 921 ms; a retry after 4,196 ms recovered a correct review route.
That second recovery illustrates a separate boundary. The request said, “The audio slider jumps to zero, and I also need a corrected invoice.” The recovered response quoted only its subject: “Audio settings plus invoice”. The quote was source-exact, but the blind AI reviewer judged it unsupported and incomplete. It names two topics without conveying the malfunction and invoice request that substantiate review.
Both failed attempts remain in the logs. There were no new final provider failures, invalid quotes or schema failures. The continuation's first-attempt routing score was 27/29; its final score was 29/29. Recovery restored route availability and correctness under these labels, while the recovered v10-013 evidence still failed semantic review.
The GitHub workflow retains a failure conclusion because its predeclared exit condition flags first-attempt errors even when retries succeed. The saved experiment status is complete. Neither signal should be silently substituted for the other. The original HTTP 429, with no valid updated v10-006 decision, also remains unchanged.
Routing stays tied on valid pairs.
Across the full 20-case view, previous matches 20/20 and updated matches 19/20. The updated deficit is the preserved original 429, not a valid wrong route. Rules match 10/20. Both model prompts catch all 13 review labels among their available outputs; neither adds an unnecessary model review. The guarded view matches 16/20 for each prompt, with narrower ordinary-route coverage.
| Measure | Previous prompt | Updated prompt | Rules |
|---|---|---|---|
| Final routes / all 20 cases | 20/20 | 19/20 | 10/20 |
| Final routes / common-valid pairs | 19/19 | 19/19 | 10/19 |
| AI evidence support / common-valid pairs | 18/19 | 19/19 | Not graded |
| AI frozen-fact coverage / common-valid pairs | 15/19 | 18/19 | Not graded |
| Joint route + support + coverage / common-valid pairs | 15/19 | 18/19 | Not graded |
Both prompts route every common-valid pair correctly. There is therefore no observed routing gain on the 19 valid pairs. Comparing 20/20 with 19/20 without preserving the unavailable slot would confuse availability with a routing-policy difference.
The continuation guard catches all 18 review-required classifications but adds seven unnecessary reviews, proposing only four ordinary routes. The final model makes 11 correct ordinary proposals. The unchanged rules baseline misses review triggers and unnecessarily reviews some ordinary controls; every mismatch remains in the report.
Evidence improves under AI review.
The original ten evidence grades remain unchanged. A fresh AI reviewer received only the 29 new valid final outputs, shuffled and assigned pseudonym IDs. It saw full requests, frozen reference decisions and minimum facts, returned categories and quotes, the routing policy and the existing rubric. It did not see prompt arms, call order, model source, previous grades, aggregate scores, example quotes or the unblinding map before saving its initial grades.
Decision support and reference-fact coverage were judged separately. A quote can support a protective review decision while omitting another frozen fact. Conversely, passing the exact-source validator does not establish that a quote supports its returned decision. Initial grades and process hashes were sealed before unblinding; fact-index accounting was checked mechanically.
On the 19 common-valid pairs, updated evidence received one additional support judgment and three additional frozen-fact coverage judgments, with no previous-only differences. The support difference is v10-013: the previous generic subject failed, while the updated quote conveyed the malfunction and invoice request.
All-20 evidence support is 19/20 for each prompt, but the deficits differ: previous has an unsupported valid quote; updated has the original unavailable call. All-20 reference coverage and joint success are 16/20 previous versus 18/20 updated. The common-valid view compares equal case coverage, while the all-20 view retains failures. Both belong in the record.
Review the criteria themselves.
Two of the three updated-only coverage differences depend on criteria that demand more than the smallest sufficient protective trigger. On v10-007, the previous quote requests permanent workspace erasure, which independently supports review. The frozen composite fact also requires the surrounding instruction to retain the account. On v10-015, medication advice for an imaginary character independently requires review under this support policy, but the frozen fact set also requires calendar repair.
The reviewer kept those fuller criteria in the primary score and flagged their demands. We must not present them as two additional necessary-trigger corrections. They are coverage differences under a particular frozen reference set.
Both prompts fail fuller reference coverage on v10-016. Each quotes an actual personal biopsy result, supporting sensitive-context review. Neither includes its conflict with a “fictional” label. The frozen criterion demands that conflict; the AI reviewer judged the quotes supportive but reference-incomplete and flagged the criterion. There are four ambiguity flags across these three cases.
No labels, criteria or grades were relaxed after outputs. Human review of the criteria is the next step, followed by a fresh evaluation. It has not yet happened. Historical scores should remain inspectable rather than being rewritten to fit a later interpretation.
What this continuation cannot prove.
The set was AI-authored and separately AI-reviewed before the original calls. It is now seen. This continuation is not a new unseen benchmark, and the combined view spans different provider windows and spacing settings. A single descriptive paired result cannot establish a general causal benefit or stable failure rate.
The 39 valid outputs were graded across two separate AI review rounds without a calibration study between reviewers. Shared training and policy assumptions can produce correlated blind spots. Repeated requests can reveal pairing, and wording style can suggest a prompt arm. Blinding reduces cues; AI review is not human adjudication.
The previously untouched security and ordinary groups are now covered, but four policy-directed fictional English cases per group cannot establish comprehensive injection resistance or customer performance. Only a single source-exact quote of 4–120 UTF-16 units is studied. Longer or discontiguous explanations are outside the contract.
Recorded continuation usage was 42,796 total tokens; recorded cumulative usage was 57,596. Usage on the original 429 and both 503 responses is unknown, not zero. Attempt logs are summed once without double-counting first-attempt views. These are not complete billing figures or latency guarantees.
All 95 software tests passed before calls, and offline verification checked selection, exact quotes, counts, frozen sources and original-result preservation without new provider calls. A later manual-workflow guard prevents accidental replay once the continuation is recorded. These checks establish software behavior and provenance, not human label validity or production readiness. No billing, project, key or quota-tier changes were used; no business action was executed.
Inspect the complete record.
- Full continuation findings, every route, quote and semantic deficit
- All continuation attempts and combined raw view
- Combined semantic, paired and usage metrics
- New blinded items, initial grades, sealed process and unblinding map
- Pre-call continuation plan, unchanged rubric and offline verifier
- Unchanged original partial findings and original raw record
- Continuation workflow, including the recovered failures
Publication note: AI-drafted and checked against frozen experiment records. No new model calls were made for publication. Human criteria review remains pending.