No observed routing gain.
In our previous experiment, both model prompts unnecessarily reviewed a concrete chart-loading problem that explicitly ruled out account deletion. We added a more explicit instruction for account-closure polarity and clause scope, then tested it on a fresh set of 40 fictional support requests.
The updated prompt matched 36/40 reference decisions. The previous prompt matched 38/40; the unchanged rules baseline matched 26/40. Those all-case scores retain failed calls and invalid outputs without correctness credit.
On all 35 cases with valid output from both prompts, their category and review decisions were identical, and all 35 matched the frozen labels. There were no common-valid routing corrections or regressions. This run does not demonstrate a semantic routing gain.
The lower updated score came with four failed calls, versus one invalid output in the previous arm. It does not establish a semantic regression either. On the fresh case where the previous prompt still unnecessarily reviewed negated closure wording, the updated call returned HTTP 503. The correction we most wanted to observe remains unmeasured.
Resolve polarity and scope before review.
The Node adapter now offers its new default policy, closure-negation-v7, by appending explicit scope instructions to the unchanged intent-sufficiency-v6 prompt. The instruction asks whether closure is actually requested, suspected to have happened, or explicitly ruled out. A negated closure statement should not itself become a destructive request, a second substantive intent or a reason to review an otherwise clear ordinary request.
If a requester rules out deletion and asks only for a receipt, the intended ordinary route is billing. If they rule out account closure but describe a specific application malfunction, the intended ordinary route can be technical. Evidence for that route should come from the positive request or concrete symptom, rather than only the negated phrase.
Negation is scoped: denying one action does not erase another affirmative destructive request. Active or conditional closure, how-to closure questions, suspected accidental deletion, contradictory or unclear closure scope, security incidents, classifier-directed overrides and conflicting substantive intents retain protective review. A negated closure statement alone also cannot make an otherwise vague request specific.
This is a prompt instruction, not a deterministic negation detector or an assurance of model behavior. The prior prompt remains selectable. Both live arms used the same quoted-evidence contract; indexed evidence was not part of this comparison. The rules baseline and interactive browser lab were unchanged.
A fresh set, with AI-review limits.
The set has five cases in each of eight groups: negated technical, negated access, negated billing, active closure, ambiguous closure, security/injection, vague controls and contextual requests. Twenty-two frozen labels require review; 18 permit a route proposal.
One AI agent authored the cases from the written policy. A separate agent in a fresh context saved independent initial labels before reading the author’s proposals. Initial agreement was 40/40, with no disagreements, unresolved labels or replacements. The reviewer knew the assigned groups, which can influence interpretation. This was AI review, not human adjudication.
The implementation was frozen before the implementer inspected the fresh cases. Hashes bind the original inputs, author proposals, reviewer initial labels, resolutions, final labels, written policy and implementation. No exact normalized request body was reused from six earlier sets. That check does not establish conceptual novelty or exclude analogous model-training material.
The paired comparison used Gemini 3.5 Flash Lite, quoted evidence, alternating prompt order by case and 6.5-second spacing. Each classification had one attempt, with no retries and a maximum of 80 API attempts. Only ID, subject and body entered provider requests—not labels, groups or rationales. Guarded v7 was derived from the same updated responses without extra calls. No business actions were executed.
Availability remains in the score.
| Measure | Rules | Previous v6 | Updated v7 | Guarded v7 |
|---|---|---|---|---|
| Exact matches / 40 | 26 | 38 | 36 | 29 |
| Valid decisions / 40 | 40 | 39 | 36 | 36 |
| Unavailable or invalid / 40 | 0 | 1 | 4 | 4 |
| Required review caught / 22 | 16 | 22 | 22 | 22 |
| Unnecessary valid reviews | 8 | 1 | 0 | 7 |
| Correct proposals / proposals | 10/16 | 16/16 | 14/14 | 7/7 |
Both model prompts matched all 35 common-valid reference decisions. That subset excludes failures; it cannot replace the all-40 results or turn the updated arm’s 36/36 valid matches into evidence of improvement.
All 22 required-review cases had valid, correct review decisions in both arms on this set, including the active/ambiguous closure and security/injection controls. The four updated failures happened on routable cases. Zero observed valid-output unnecessary reviews therefore does not establish that unnecessary review was fixed.
The guarded arm matched 29/40 and added seven unnecessary reviews. Rules alone had six required-review misses and eight unnecessary reviews. These are material coverage tradeoffs, not hidden implementation details. The complete record lists every mismatch in every arm and all per-group results. No route proposal is authorization to execute a business action.
Every failed call and the unmeasured correction.
| Arm and case | Failure | Reference decision |
|---|---|---|
| Previous v6 / v7-004 | Evidence not in request | Technical |
| Updated v7 / v7-005 | 15-second timeout | Technical |
| Updated v7 / v7-011 | 15-second timeout | Billing |
| Updated v7 / v7-012 | HTTP 503 | Billing |
| Updated v7 / v7-037 | HTTP 503 | Access |
On v7-004, the previous prompt returned HTTP 200, but its 107-character evidence failed the unchanged exact-substring check. Local validation rejected it as invalid_model_decision. Sanitized diagnostics retain the reason and counts, not the rejected raw response, so the precise quote defect cannot be reconstructed from this record. We did not normalize or repair it.
The updated timeouts occurred on a blank-dashboard request (v7-005) and a subscription-only cancellation request (v7-011). The HTTP failures occurred on a receipt request (v7-012) and a brief password-recovery request (v7-037). Operational fallback requests review, but that is not a valid model classification and earns no correctness credit.
The unmeasured correction is v7-012. Its body says, “I do not want our workspace closed,” then asks for the receipt for the most recent subscription payment. The reference route is billing. Previous v6 requested review and cited the negated closure sentence; updated v7 returned HTTP 503. A failed call cannot establish whether the new instruction resolves that remaining negation error.
All 80 planned classifications were attempted once, with no unattempted cases. The workflow intentionally exited with failure because it contained call or output errors; the saved experiment status is complete. No second provider run was used to remove those failures.
An exact quote can support the wrong explanation.
On v7-031, both prompts correctly chose review under the frozen label because the ordinary request was vague. But the previous prompt cited “Please do not erase the account.” Updated v7 instead cited “This is awful and I need someone to fix it.” The first quote’s presence in the request does not establish active account-deletion intent.
This inspection illustrates a distinction the routing score does not measure: a correct category with a source-exact quote can still have a weak explanation. It is not an independently adjudicated evidence-support score, and we do not count the excerpt change as a measured semantic gain. Routing, evidence relevance and evidence completeness need separate assessment.
Known provider-reported total tokens were 29,334 for previous v6 across 40 attempts with usage metadata, and 34,831 for v7 across 36. Usage on the four failed updated calls is unknown, not zero; those totals are not a complete cost or billing statement. Median attempt latency was 1,108.5 ms versus 970.5 ms; nearest-rank p95 was 8,528 ms versus 8,475 ms. Timings include known failed-attempt timings and exclude deliberate spacing. A single run does not establish a latency or reliability advantage.
What this run cannot establish.
Author and reviewer can share training, model-architecture and policy-interpretation blind spots. Forty-of-forty agreement is not ground truth. The cases were authored from the policy, and assigned groups can cue expected behavior. There was no human routing adjudication or independent excerpt-support adjudication.
Forty synthetic English cases, with five per group, do not represent customer traffic or establish stable operating rates. Negation scope can be legitimately disputed under different business policies. No exact body reuse cannot establish broad novelty or provider-training exclusion. This set is now seen; further tuning needs new evidence.
Cases ran in fixed dataset/group order, with alternating prompt order rather than randomized repeated trials. Temporal provider variation can correlate with case position. Model aliases and generation are not deterministic. The added instruction also lengthens the prompt, so this run cannot isolate each sentence’s effect or explain the availability difference causally.
The result supports neither a semantic routing gain nor a semantic regression on common-valid cases. It does not establish production readiness, comprehensive injection resistance or a general reliability conclusion. Earlier sets have different mixes and cannot be combined into an accuracy progress curve. The narrow finding remains: scope handling was made explicit, but the intended remaining correction was not measured successfully.
Inspect the frozen record.
All 56 implementation and provenance tests passed locally without skips and in GitHub Node 22/24 checks before live calls. Counts, scores, successful exact quotes, unchanged rules outputs, derived guarded decisions and provenance hashes were rechecked offline without further provider calls. These checks establish mechanics and provenance, not semantic improvement.
- Full findings, every mismatch, all failed calls and limitations
- Unchanged machine-readable record of all 80 attempts
- Fresh cases and frozen reference labels
- AI-review manifest and implementation freeze
- Written routing policy and frozen adapter
- Original completed run, with its intentional failure conclusion
- Unchanged rules-only interactive lab
Publication note: this article was AI-drafted and checked against the frozen experiment artifacts. Cases and reference labels were AI-authored and separately AI-reviewed as described above; no human adjudication is claimed. No new model calls were made to prepare this article.
Next experiment: bounded call reliability, a recovered-but-wrong response and unchanged correctness.
← Back to the journal