Three observed routing corrections.

“Please fix it” can express urgency without identifying what needs fixing. A triage system that automatically turns that into a technical route is adding information the requester did not provide. We added an explicit intent-sufficiency instruction to our optional Gemini adapter, then tested it against the previous prompt and the unchanged rules baseline.

On 40 new fictional cases, the updated prompt matched 38/40 reference decisions. The previous prompt matched 35/40; rules matched 28/40. Each model arm had one failed call or rejected output. Those failures remain in the denominator without correctness credit.

On the 38 cases where both prompts returned valid decisions, the updated prompt corrected three vague-request mistakes, with no observed regressions. It reviewed all five deliberately vague requests and correctly routed all five clear, brief controls. These are observations on this set, not reliable operating-rate estimates.

One semantic mistake remained in both prompts: unnecessary review of a concrete chart-loading problem that explicitly rejected account deletion. Exact source evidence did not prevent the wrong interpretation.

Ask whether an intent is established.

The default Node adapter now uses intent-sufficiency-v6. Before proposing access, billing or technical routing, the prompt asks whether the request establishes a supported intent or a concrete symptom. Generic dissatisfaction, unexplained “it” or “this”, and an unspecified fix or help request require review rather than a guessed technical fallback.

Brevity alone does not make a request vague. A password-reset request or invoice request can be clear without account identifiers or diagnostic detail. A concrete application crash or unavailable service can justify technical routing without a named feature. Security, destructive intent and conflicting substantive requests still take precedence.

The change is a prompt instruction, not a deterministic detector. It does not guarantee abstention on every vague request. The previous clarified-v4 prompt remains unchanged and selectable. Both paired arms use the same quoted-evidence contract; the optional indexed-evidence mode from our previous experiment was not used. Rules and the browser demo were unchanged.

Fresh cases, separately AI-reviewed labels.

The set contains 40 fictional English requests: five each in vague, terse-clear, access, billing, technical, account-closure, security/injection and contextual groups. Seventeen reference decisions require review; 23 permit a route proposal.

One AI agent wrote the cases from the written routing policy. A separate agent in a fresh context independently saved its labels before seeing the author’s proposals. Initial agreement was 40/40, with no disagreements, unresolved labels or replacements. This was AI review, not human adjudication. Agreement is a process observation, not proof that the labels are correct.

The implementation was frozen before the implementer inspected the cases. Hashes bind the original inputs, initial labels, final labels, written policy and frozen implementation. No exact normalized body was reused from five earlier sets. That check cannot establish conceptual novelty or exclude analogous provider-training material.

We compared both prompts with Gemini 3.5 Flash Lite on the same requests, alternating arm order by case and spacing calls by 6.5 seconds. Each classification had one attempt, with no retries and an 80-attempt cap. Only request ID, subject and body entered provider requests—not labels, groups or review rationales. A separate rules-guarded arm was derived from the updated responses without additional calls. No business actions were executed.

Keep failures in the denominator.

All 40 cases, including failed calls and invalid outputs without credit.
MeasureRulesPrevious v4Updated v6Guarded v6
Exact matches / 4028353830
Valid decisions / 4040393939
Required review caught / 1714131717
Unnecessary valid reviews9119
Correct route proposals / proposals14/1722/2521/2113/13
Vague cases matched / 55255
Terse-clear controls matched / 53553

The valid-output corrections were v6-002, v6-004 and v6-005. The previous prompt routed each vague request to technical; the updated prompt requested review. On the 38 common-valid cases, the previous prompt matched 34/38 and the updated prompt 37/38. This conditional subset excludes failures and does not replace the all-40 scores.

Two other case differences involved availability or output validity. On v6-040, the previous output was rejected while the updated decision matched. On v6-036, the previous decision matched while the updated call failed. These differences do not establish semantic improvements or regressions.

The rules-guarded arm caught all 17 required-review cases, but imposed nine unnecessary reviews and matched only 30/40 overall. Rules alone had three security/injection review misses and nine unnecessary reviews. The full record lists every mismatch in every arm. A guard can reduce route proposals while losing useful coverage; its name is not a safety guarantee.

Two failed calls and a shared review error.

Updated v6, v6-036: HTTP 503. The expected decision was billing: an actual receipt request accompanied by a benign quotation of account-closure wording. The provider returned no valid decision. We did not retry it or credit fallback review as a correct classification.

Previous v4, v6-040: evidence too long. The request combined an invoice request and a search crash, so the reference decision required review. The provider returned HTTP 200, but its evidence was 141 characters, exceeding the unchanged 120-character limit. Local validation rejected the output as invalid_model_decision. Transport success is not contract success. We did not truncate, repair or retry the quote.

Both prompts, v6-038: unnecessary review. The request says, “I do not want my account deleted or my plan changed,” then describes a signed-in dashboard chart that stays blank while its loading animation runs continuously. The reference decision is technical. Both models requested review instead.

The updated prompt cited the subject “No account deletion”; the previous prompt cited “I do not want my account deleted”. These are exact source passages, yet they do not support treating the negated closure as an active request. This is the remaining review-decision miss: an unnecessary escalation, not a missed required review. Routing correctness and evidence relevance need separate evaluation. The frozen label was retained; no post-result tuning or relabelling was used to improve the score.

A queued job is not a model result.

The original GitHub run remained queued with zero jobs after submission at 16:42:27 UTC. Normal cancellation failed. A single guarded recovery passed 54 software tests but refused to enter its model-call step because the original was not confirmed cancelled with zero jobs. It made no model calls.

The original eventually began the evaluation at 17:49:53 UTC and finished all 80 attempts at 17:59:37 UTC. GitHub marked the job failed because the evaluation intentionally returns a nonzero exit when either arm contains call or output errors. The saved experiment status is complete. There were no unattempted cases, retries or duplicate provider comparisons. The queue cause remains unknown; we cannot attribute its recovery to our replacement attempt.

The existing authorized unbilled evaluation project was reused without billing or access changes. The previous arm reported 21,155 total tokens across 40 attempts with usage metadata; the updated arm reported 28,181 across 39. Usage for the failed HTTP call is unknown, not zero. These totals are not a complete cost or billing statement. Median attempt latency was 748.5 ms versus 765 ms; nearest-rank p95 was 1,389 ms versus 1,349 ms. Timings exclude deliberate spacing and include failures. One run cannot establish a latency advantage or a service guarantee.

What this evidence cannot establish.

Separate AI agents can share training, model-architecture and policy-interpretation blind spots. Forty-of-forty agreement does not make their labels ground truth. There was no human adjudication, and excerpt relevance or completeness was not independently scored.

The cases were composed from the policy being evaluated. They are small, synthetic and English-only, with possible wording or conceptual overlap with the intended behavior. Five vague cases and five clear, brief controls are too few to estimate broad vagueness handling or unnecessary-review rates. Other business policies could legitimately draw different boundaries.

This is one paired run, not a causal guarantee, representative customer traffic, production-readiness evidence or comprehensive injection testing. Model aliases and live generation are not deterministic. The 21/21 correct v6 proposals and 17/17 required reviews caught are sample observations, not assurances about future requests. Earlier evaluations used different case mixes and cannot be combined into an accuracy progress curve.

The set is now seen. Later changes must be tested on further unseen cases and described as a new experiment. The next useful evidence would include human-adjudicated routing and evidence-support labels, repeated availability measurements and wider coverage of negation and vague-versus-specific boundaries. A route proposal still authorizes no business action.

Inspect the frozen record.

All 51 implementation and provenance tests passed locally and in GitHub Node 22/24 checks before the live evaluation. The saved scores, all 80 attempts, successful exact quotes, unchanged rules outputs, guarded decisions and provenance hashes were independently rechecked offline without more provider calls. Passing software tests does not establish semantic correctness.

Publication note: this article was AI-drafted and checked against the frozen experiment artifacts. Evaluation cases and labels were AI-authored and separately AI-reviewed as described above; no human adjudication is claimed. No new model calls were made to prepare this article.

Next experiment: negated closure, no observed routing gain and the unmeasured correction.

← Back to the journal