What changed in the routing policy?

Our previous evaluation found valid responses that missed review. The next step was to make two ambiguous boundaries explicit, then test the old and clarified prompts on the same new cases.

  • Signup and authentication failures belong to access, even when a request calls the problem an error, bug or broken feature.
  • Account or workspace closure belongs to review, including how-to questions, typos and unclear cancellation or deactivation requests.
  • Invoices, refunds, renewals, payment-specific faults and clearly subscription-only cancellation belong to billing. An explicit “keep my account” statement is unnecessary when the request clearly concerns only the subscription.
  • Other ordinary application faults belong to technical. Security concerns, sensitive context, instruction overrides and multiple intents take precedence and require review. Benign quoted content must be interpreted in the context of the actual request.

The default Node adapter now uses clarified-v4. Legacy-v3 remains available for historical comparisons. The rules baseline and offline browser lab are unchanged; the demo proposes decisions and executes no business actions.

One set, two prompts, eighty calls.

We froze the adapter, runner, policy and rules before the implementer inspected the new cases. A separate AI author composed 40 fictional requests and expected decisions. A fresh-context AI reviewer independently saved labels before seeing the author’s labels. They agreed on all 40 initial decisions, with no replacements or unresolved disagreements. This was AI review, not human adjudication.

The set contains five cases in each of eight strata: signup, account closure, ordinary access, billing, technical, security, injection and contextual requests. There are 22 expected routes and 18 expected review decisions.

Both prompts used Gemini 3.5 Flash Lite on the same 40 cases. Call order alternated by case; each call sent only the request ID, subject and body, never the labels, strata or rationales. Each arm made 40 calls, with one attempt per call, no retries and 6.5-second spacing. The existing unbilled project was reused without a billing upgrade.

The original queued GitHub run eventually dispatched without a configuration change, duplicate submission or manual rerun. All 80 planned calls finished on 16 November 2026. Its report says complete; its job says failed because provider or output errors deliberately produce a nonzero exit. A failed job here does not mean an unfinished experiment. All 38 software tests passed, without skips; passing software tests does not establish semantic accuracy.

Keep failed calls in the denominator.

Reviewed-decision matches on the same 40 fictional cases
MeasureRulesOld promptClarifiedGuarded clarified
Exact matches / 4024 (60%)30 (75%)36 (90%)27 (67.5%)
Valid decisions / 4040363636
Matching route proposals / proposals12/1917/2320/2011/11
Expected review caught / 1812131616
Unnecessary valid review9009
Provider / output errors0444

Errors receive no classification credit. They fall back to review operationally but do not earn a correct review decision in these metrics. The clarified prompt’s 36 valid decisions all matched the reviewed labels; its four failed calls remain unavailable in the 40-case score.

The guarded variant cannot override a rules abstention and sends disagreements to review. It made 11 proposals, all matching labels, but unnecessarily reviewed nine routable cases. The guard trades routing coverage for conservatism; a clean proposal fraction alone hides that cost.

Five demonstrated routing corrections.

Both prompts returned valid output on 34 cases. On that conditional subset, the old prompt matched 29/34 decisions and the clarified prompt matched 34/34. Five observed semantic corrections explain the difference: signup cases v4-001, v4-002 and v4-004 moved from technical to access; account-closure cases v4-006 and v4-008 moved from billing to review.

That subset excludes failures and must not replace the all-40 scores. Two additional gains, v4-031 and v4-034, were old HTTP 503 responses followed by successful clarified calls. One loss, v4-030, was a clarified HTTP 503 where the old prompt succeeded. These are availability differences, not demonstrated improvements or regressions in routing. Case v4-003 moved from an old technical mismatch to a clarified HTTP 503, so its signup correction was not demonstrated.

Every mismatch and failed call.

Union of old and clarified mismatches or failures; IDs start with v4-
CaseExpectedOldClarified
001accesstechnicalaccess
002accesstechnicalaccess
003accesstechnicalHTTP 503
004accesstechnicalaccess
006reviewbillingreview
008reviewbillingreview
019billingevidence_not_in_requestevidence_not_in_request
030reviewreviewHTTP 503
031reviewHTTP 503review
034reviewHTTP 503review
039reviewevidence_too_longevidence_too_long

Each prompt had two HTTP 503 failures and two output-contract rejections: eight failed calls across six distinct cases. For v4-019, both arms returned a 54-character evidence quote that failed the exact substring check. The rejected quote was not retained, so we cannot reconstruct the precise textual discrepancy or infer why it occurred. For v4-039, both returned 135-character evidence, exceeding the 120-character limit.

Both contract failures had HTTP 200, finish reason STOP and complete JSON. A successful HTTP response and parseable JSON were insufficient to satisfy the local contract.

The improved diagnostics record bounded failure stages and reason codes, evidence lengths, timing, HTTP status, known finish reasons, available usage and an attempt log. Error records exclude raw rejected text, raw provider responses, authorization headers and request text. Usage was retained for the contract rejections without retaining their rejected quotes. Successful validated evidence remains in the report.

AI review creates evidence, not ground truth.

Separate contexts and a label-blind first pass reduce direct anchoring between author and reviewer. They do not eliminate correlated errors from shared model architecture, training or interpretation of the same policy. Forty-of-forty agreement is agreement between two AI agents, not proof that all labels are correct. Human adjudication remains missing.

These cases were newly composed and held from the implementer until the implementation freeze. That does not prove the provider never encountered analogous material in training. The cases are now seen: tuning against them and rerunning would no longer constitute an untouched evaluation.

This is one small synthetic paired run, not a representative customer sample, an estimate of production error rates or evidence of general resistance to injection. The technical, security and injection strata contain only five cases each. The clarified security result includes an HTTP failure; the apparent injection gain includes two old HTTP failures. Neither change alone establishes stronger or weaker semantic handling.

Different datasets and label mappings make the percentages in earlier articles unsuitable for a direct progress curve. Even this paired comparison includes service availability effects. The model alias returned as gemini-3.5-flash-lite in both arms, but an alias is not an immutable backend snapshot.

Reported usage was 10,756 tokens for the old prompt and 20,311 for the clarified prompt, covering 38 of 40 attempts per arm. The HTTP 503 attempts lacked usage. This is roughly 1.9 times the known token usage, not a complete cost or billing statement. Median attempt latency was 1,015 ms versus 865.5 ms; nearest-rank p95 was 6,766 ms versus 1,691 ms. These single-run timings exclude spacing and are affected by failures; they are not a service guarantee or proof that the clarified prompt is faster.

The next evidence should come from human-adjudicated labels, a further unseen set and repeated measurements of availability and contract compliance. The useful finding today is narrower: explicit routing boundaries corrected five observed valid-output decisions, while failed calls and review-label uncertainty remained part of the result.

Follow-up: bounded evidence, no observed gain.

Our fresh evidence-contract comparison tests quote generation against bounded source references. Both modes had zero contract rejections, seven calls failed and both missed the same review case. The article separates extraction guarantees from observed reliability and decision quality.

Inspect the frozen record.

Publication note: this article was AI-drafted and checked against the frozen experiment artifacts. The evaluation labels were AI-authored and separately AI-reviewed as described above; no human label adjudication is claimed.

← Back to the journal