Our first triage experiment showed how a keyword baseline can miss paraphrases, mistake quoted context and overlook requests that need review. We have now added a real Gemini adapter and compared it with the unchanged rules on a new set.
The model matched more expected decisions. The more useful finding is how its errors differed from the baseline’s, and how constraining it changed the balance between useful routing and human review. This is a public engineering experiment with fictional requests, not a customer deployment.
1. Freeze the question before calling the model.
The new evaluation contains 40 hand-authored fictional support requests. Each has an expected route—access, billing or technical—or an expected human-review decision. The set includes ordinary requests, paraphrases, multiple intents, negation, security concerns, sensitive context, destructive requests, instruction-like text, benign quotations and an unsupported language.
We committed the dataset and prompt before the first model call. The adapter sends only the request ID, subject and body to Google. It does not send the expected decision or the case’s evaluation category.
We used gemini-3.5-flash-lite in a separate unbilled evaluation project. Its available free quota covered all 40 cases. Requests were sequential and spaced by 13 seconds. The run made 40 API attempts, with no retries and no business actions.
The cases are new relative to the original evaluation, but they were authored by the implementation author. They are deliberately diagnostic, neither blind nor representative of customer traffic. “Fresh” describes their first use with this adapter; it does not establish an independent benchmark.
2. Compare three approaches on the same requests.
Rules is the existing keyword baseline. Gemini uses the model’s validated category, including its option to request review. Guarded Gemini reconciles the same model output with the existing baseline: it preserves baseline abstentions and requires review on disagreement or when the model requests it.
| Measure | Rules | Gemini | Guarded Gemini |
|---|---|---|---|
| Exact decision matches / 40 | 20 (50%) | 38 (95%) | 29 (72.5%) |
| Valid decisions / 40 | 40 | 39 | 39 |
| Correct route proposals / proposals | 8 / 17 | 18 / 19 | 8 / 8 |
| Expected review cases caught / 21 | 12 | 20 | 21 |
| Unnecessary review decisions | 11 | 0 | 10 |
| Invalid outputs / provider errors | 0 | 1 | 1 |
An exact match requires both the route and the review flag to match the expected decision. The denominator remains all 40 cases, including the invalid output. Errors fall back to review operationally but receive no correctness credit.
Gemini made 19 route proposals; 18 were correct, for 94.7% proposal precision. It correctly requested review on 20 of the 21 expected review cases. It made no unnecessary valid review decisions, but one ordinary request failed output validation and also went to review as an error.
The guarded approach caught all 21 expected review cases and made eight route proposals, all correct on this set. Its cost was coverage: ten ordinary requests received unnecessary review decisions, with the invalid output reported separately. Preserving baseline abstentions prevents the model from recovering many routable paraphrases.
3. Keep both model issues visible.
v2-009 / An invalid output is rejected.
The request said the fictional user was locked out of the workspace after several attempts. The expected route was access. The adapter recorded invalid_model_decision and fell back to review.
This error means the response did not satisfy the strict category-and-evidence contract. The rejected raw response was intentionally not retained, so the report cannot establish which specific field or evidence check failed. We should not invent a more precise diagnosis. This is an observability limitation to address before the next experiment.
v2-023 / A vague request receives a confident-looking route.
The subject was “Help please.” The body said, “Something is different in the fictional workspace. Can someone assist?” The expected decision was review because the request did not identify a supported issue. Gemini proposed technical support and quoted the first sentence as evidence.
The quotation was real and the schema was valid. The route was still wrong. This is a concrete example of why evidence matching does not establish semantic correctness. The guarded approach prevented this proposal because the baseline found no supported route and abstained.
The live workflow exited with failure because one output was invalid. All 40 rows were retained. We did not rerun the experiment to seek a cleaner headline or tune the policy against these observed cases.
4. Separate output validation from permission to act.
The server-side Node.js adapter uses a fixed Gemini HTTPS endpoint, header authentication, a 15-second timeout and at most two attempts per case. Only HTTP 429 and 503 responses qualify for a retry. Output must be complete, bounded JSON with exactly a supported category and a 4–120 character evidence quote present in the request.
These checks bound the integration and reject malformed responses. They cannot prove the category is correct, detect every security issue or authorize a business action. There is no action executor in any of the three approaches.
The public browser lab remains an offline rules demonstration. It does not expose the API key or make model calls. The model comparison runs separately with a GitHub Actions secret. Trying the browser lab does not reproduce the live model experiment.
The 22 automated tests passed before the live run, and validation also passed on Node 22 and 24. Those tests verify software behavior, transport limits and failure handling. They do not erase either model issue or establish performance on real support traffic.
5. Publish the limits alongside the scores.
The full per-case report records expected decisions, all three approaches, validated evidence, errors, attempts, returned model version and available usage metadata. The original run preserves the execution record. Its source is pinned to revision 3ef1f28; the report was later committed separately.
For the 39 validated responses, recorded latency had a median of 1,007 ms and a nearest-rank p95 of 7,705 ms. This excludes the invalid response and the deliberate 13-second spacing. It is not an end-to-end service latency guarantee.
Those validated responses reported 10,546 total tokens. The invalid response’s usage was not retained, so this is incomplete usage evidence and cannot establish total cost. No billing upgrade was performed. A free-quota run does not predict the cost of a production workload.
The report includes SHA-256 hashes for the dataset and prompt. Live generation is not claimed deterministic, and a model alias may change over time. The set is now seen; using it to tune a prompt and then reporting it as untouched would undermine the comparison.
Our original 25/30 result belongs to a different set and pinned revision. Comparing that 83.3% headline directly with this model’s 95% would mix a changed case distribution with a changed classifier. Here, the meaningful comparison is 20/40 versus 38/40 on the same 40 requests.
Follow-up experiment: We improved failure diagnostics and evaluated 40 independently authored inputs, including the review misses and label-mapping limits. The results above remain tied to their original set and revision.
6. Use the result to design the next test.
This run supports a narrow conclusion: Gemini interpreted more of these fictional requests correctly than the unchanged rules, while the guarded approach traded routing coverage for review recall and proposal precision.
The next evaluation should use separately authored, unseen cases, explicit workflow boundaries and repeated runs. It should measure wrong routes, missed review, unnecessary review, invalid outputs, service failures, latency and complete usage separately. Better sanitized diagnostics should identify output-contract failures without retaining credentials or sensitive inputs.
Only after stronger evidence would a constrained customer pilot make sense, with approved data access, named review owners, audit records and rollback. Those are proposed next steps, not capabilities proved by this experiment.