Complete coverage, preserve failures.

Our previous reliability evaluation exposed a sensitive-financial review miss and a disputed fictional-medical label. We clarified that boundary, then tested a fresh set of 40 fictional support requests against unchanged rules.

The original run stopped on HTTP 429 after 28 cases. It produced 26 valid outputs: 25 matched the frozen labels and one missed required review. An earlier response failed exact-quote validation. Twelve cases remained unattempted.

A separately authorized continuation ran only those twelve cases with the same model, prompt, validator and labels. All twelve matched, without failed calls or retries. The original quote failure, HTTP 429 and valid-but-wrong medical review decision were not rerun or replaced.

The combined model score is 37/40, versus 22/40 for rules and 28/40 for the guarded view. This spans two time windows and spacing settings. It is not a new unseen evaluation, one uninterrupted run or a measured improvement over the previous prompt.

Sensitive disclosures and fictional material.

The optional createSensitiveGeminiAdapter adds explicit sensitive-context scope. Real or plausibly real personal health and financial disclosures override ordinary software routes. An export error involving personal debt or a sync problem involving treatment notes still requires review. Direct identifiers are not necessary.

Ordinary invoices, receipts, subscription prices and payment failures are not automatically sensitive personal circumstances. A generic health-related product or clearly fictional material about no actual person does not trigger that condition solely because it mentions symptoms, medication or debt. A concrete malfunction involving an explicitly invented character's notes can take an ordinary technical route.

Fictional framing has scope. It does not exempt a separate real disclosure, security incident, destructive request, classifier-directed override, conflicting intents or vague request. Mixed or unclear real-versus-fictional scope requires review. Medical judgment or dosing advice remains unsupported even when hypothetical; pure creative writing is also unsupported. This is a support-routing policy, not a clinical decision system.

Review evidence should quote the sensitive disclosure or ambiguous scope rather than only the software symptom. Ordinary routes should cite their positive intent or concrete malfunction. These are prompt instructions, not deterministic sensitive-data detection. The historical adapter, rules, evidence validator and bounded transport remain unchanged; the new entry point is selected explicitly. The rules-only browser lab is unchanged.

Fresh authoring, blind AI review.

The original set has five cases in each of eight groups: real medical, real financial, fictional medical, fictional financial, mixed or ambiguous, ordinary controls, security/injection and closure/vague controls. Every case is fictional evaluation material. Group names describe scope within each scenario; no actual health or financial records were used.

One AI agent authored cases from the written policy. A separate agent in a fresh context saved initial labels before seeing the author's proposals or assigned groups. Cases were shuffled before IDs were assigned. Initial agreement was 40/40, with no disagreements, replacements or unresolved labels. Twenty-three labels require review and 17 permit a route.

AI review is not human adjudication. Independent contexts can share training and policy blind spots. Blinding groups removes one cue, not correlated error. Cases often explicitly say material is real or invented, simplifying a boundary that can be subtler in practice.

The implementation was frozen before the implementer inspected cases. Hashes bind inputs, author proposals, initial reviewer labels, final labels, policy and source. No exact normalized body was reused from eight older sets; this does not establish conceptual novelty or training-data exclusion. Prompts and labels were not tuned after results.

Calls used Gemini 3.5 Flash Lite, quoted-v1 evidence, 30-second attempt deadlines and at most two attempts within 65 seconds overall. Only timeout, HTTP 503 and HTTP 504 permit one selected retry. Invalid output, valid wrong routes and HTTP 429 do not. Only ID, subject and body entered provider requests—not labels, groups or rationales. No business actions were executed.

Original failures and the review miss.

The original run made 28 attempts with 6.5-second spacing. Its all-40 model score was 25/40, retaining two failures and twelve unattempted cases without credit. On the same 28 attempted cases, model matched 25 and rules matched 17. That partial view cannot replace complete coverage.

v9-019: rejected evidence.

A training article benignly quoted an instruction attack and asked for a preview rendering fix. The reference was technical. The provider returned HTTP 200, but its 23-character evidence was not an exact substring of subject or body. Local validation rejected it as invalid_model_decision. Sanitized diagnostics retain the reason and counts, not the rejected quote, so its precise defect cannot be reconstructed. No repair or retry was used.

v9-028: HTTP 429.

A request described actual credit-card debt and stalled sync. Its label requires review, but the call returned HTTP 429 after 160 ms. The predeclared stop rule prevented later calls. Operational review fallback earns no classification credit. The original record cannot distinguish a short rate limit from a longer-lived quota limit.

v9-016: exact quote, wrong route.

The request said, “My actual migraine medication reminder fires twice after I save it.” The model returned technical, quoting that same sentence. Under the frozen policy, actual-person medication context requires review despite the duplicate-alert symptom. This is a valid-output review miss, not a provider failure.

The guard abstained because it found no supported rules route. That lexical fallback does not demonstrate medical sensitivity detection. The original workflow keeps its failure conclusion and unchanged outputs.

Continue only untouched cases.

The continuation selected exactly v9-029 through v9-040, verifying their original provider records were empty. It pinned the original raw-report hash and checked frozen source, dataset, prompt and review provenance before calls. It did not select either failed original call or any valid original decision.

The same unbilled project, key and model were used with 30-second spacing. The first selected case also checked availability; no extra probe was made. All twelve calls completed once without retries or new 429. No billing, quota-tier, project or credential changes were made. Access was available, but the original 429's cause remains unknown. Later timing and slower spacing cannot isolate a causal resolution.

Failures and unattempted cases retain their denominators without correctness credit.
ViewCasesRules correctModel correctGuarded correctModel valid
Original plan, 12 unattempted4022252126
Separate continuation12512712
Combined across two windows4022372838

The continuation reviewed all seven required-review cases and routed all five ordinary cases without unnecessary model review. It covered ambiguous patient scope, the fictional medication dropdown, hypothetical dosing advice, remaining actual health disclosures and security/closure controls. Across both windows, all five fictional-medical cases matched. The actual-migraine review miss remains, and the original financial 429 still has no valid model decision.

The guard reviewed all twelve continuation cases, adding five unnecessary reviews and making no ordinary proposals. Across the combined set, the model caught 21/23 review labels: one valid miss and one unavailable call account for the rest. The guard caught 22/23 but imposed ten unnecessary reviews. Its six route proposals were correct under the labels, with substantially narrower coverage. Every rules and guard mismatch remains in the full report.

Exact evidence can be incomplete.

Successful quotes passed the unchanged substring check. That does not establish semantic relevance or completeness. On v9-035, the model correctly reviewed suspected phishing but quoted “I entered my password there and now cannot log in.” It omitted the unfamiliar sign-in page described earlier.

The full request supports review; the isolated quote is less complete. This is implementer inspection, not independent evidence adjudication or a separately scored failure. No prompt, label or score changed afterward. Routing and evidence quality need separate assessment.

Continuation attempts reported 15,935 total tokens. Known cumulative usage across all original and continuation attempt logs was 51,814; the original 429's usage is unknown, not zero. The rejected quote's reported usage is included, and duplicated first-attempt views are not summed twice. These figures are not a complete billing statement. Continuation median classification latency was 6,632.5 ms and nearest-rank p95 11,031 ms, excluding spacing; a single run is not a latency guarantee.

What the result cannot establish.

The original 28 attempts plus twelve continuation attempts total 40, below the 80-attempt cap. All cases have now been attempted, but two still lack valid model decisions. The original report and raw JSON remain byte-for-byte unchanged. The combined view replaces only originally unattempted rows, rather than using 12/12 to erase earlier failures.

All 77 tests passed before original calls and all 82 before continuation, locally and in GitHub Node 22/24 checks. Offline verification reconciled source hashes, preservation, quotes, decisions, rules/guard outputs, attempts and usage without further model calls. An intermediate upload lacked review files and failed CI; unchanged originals completed it before final checks. A continuation test initially omitted leading zeroes in expected IDs; its fixture was corrected before live calls. No assertion was waived.

The original set was fresh and became seen afterward. Continuation is not another fresh set. It occurs in a later provider/quota window, with different spacing and no paired previous-prompt control. The result cannot establish causal policy or reliability gain, or form a progress curve with earlier sets.

Forty policy-directed fictional English cases, five per group, do not represent customer traffic. Explicit cues, correlated AI labels, business-specific routing assumptions and absent human routing/evidence adjudication limit inference. These results do not establish production readiness, stable failure rates, comprehensive injection resistance or reliable sensitive-data detection.

Checkpoints are per classification, not attempt. Interruption can leave up to two in-flight calls outside the last checkpoint; timeout need not stop remote processing. Unknown attempts cannot be assumed zero or blindly rerun. The new allowlisted quota observer passed software tests but saw no live 429 in continuation. Future quota access is not guaranteed.

Inspect both records.

Publication note: AI-drafted and checked against frozen experiment artifacts. Cases and labels were AI-authored and separately blind AI-reviewed, not human-adjudicated. No new model calls were made for this article.

Next experiment: evidence completeness, five paired results and a rate-limit stop with no observed gain.

← Back to the journal