1. Ask two questions, measure them separately.

Our first Gemini comparison recorded one rejected output and one valid but wrong route proposal. The rejected response did not include enough diagnostic detail to explain which output-contract check failed.

For the next experiment, we improved failure diagnostics and changed the source of evaluation inputs. We kept the routing prompt, categories, rules baseline and evidence constraints fixed. This lets us examine the integration without claiming that a new score proves the diagnostics made the classifier better.

The important distinction is between a response that satisfies the output contract and a decision that fits the workflow. All 40 responses in this run were valid. Seven differed from the predeclared mapped expectations, including three requests assigned to human review.

2. Independent authorship has a precise meaning here.

The input instructions were authored by Bitext Innovations, independently of this adapter implementation. From its public synthetic support dataset, a script selected five instructions from each of eight predefined intents by a fixed SHA-256 ranking. Selection used no model output or baseline prediction. Original instruction bodies were preserved; we added neutral subjects, IDs and provenance.

We mapped password recovery and registration problems to access; invoice queries, invoice requests, payment methods and payment issues to billing; account deletion and requests for a human agent to review. This produced 10 access, 20 billing and 10 review expectations. The source-only import made no model calls. We committed the sample and its label audit before the live run.

Input authorship was independent. Workflow-label mapping was ours. These labels were not independently adjudicated against our workflow policy. The pre-run audit flagged a request about a forgotten key: its source intent was password recovery, but the words alone are ambiguous enough that review could be defensible. Signup problems also sit near the access/technical boundary.

Public synthetic training examples may already have appeared in a provider’s training data. This is a reproducible external-input sample, with a documented mapping, rather than evidence of blind generalization to customer traffic.

3. More label matches came with missed review.

We compared the unchanged rules baseline, validated Gemini 3.5 Flash Lite output, and guarded Gemini, which preserves baseline abstentions and requires review on disagreement or when the model requests it. All approaches received the same 40 requests.

First run / 40 external synthetic inputs / fixed workflow mapping
MeasureRulesGeminiGuarded Gemini
Mapped-decision matches / 4022 (55%)33 (82.5%)22 (55%)
Valid responses or decisions / 40404040
Matching route proposals / proposals12 / 1226 / 3312 / 12
Mapped review cases caught / 1010710
Unnecessary valid review decisions18018
Invalid outputs / provider errors000

A match requires both route and review flag to agree with the fixed expectation. All 40 cases remain in the denominator. Failures would receive no correctness credit, even though operational fallback sends them to review.

Four signup-routing disagreements.

Cases ext-006, ext-007, ext-008 and ext-010 describe registration problems. Our predeclared mapping assigns them to access. Gemini proposed technical support. These are disagreements at a policy boundary; the source intent does not independently settle which queue a real service should use.

Three misses against the review requirement.

Cases ext-032, ext-034 and ext-035 concern account closure or cancellation. Our conservative mapping requires review for the source’s account-deletion intent. Gemini proposed billing. Its recall against the ten mapped review cases was therefore 7/10, or 70%.

All seven mismatches are retained in the per-case report. We did not repair the labels after seeing predictions or rerun the experiment to seek a cleaner result. The account-closure findings concern the stated review policy; they are not a demonstration that any real account was changed. There is no business-action executor.

The guarded version retained the baseline’s coverage limit.

Rules and guarded Gemini caught all ten mapped review cases and made 12 route proposals, all matching the mapping. Both also unnecessarily reviewed 18 routable requests. Guarded Gemini did not improve on the baseline in this run: it preserved the same abstentions and proposed the same routes.

The model matched more labels overall while missing review that the mapping required. The guarded approach preserved review recall while limiting useful routing. A single accuracy headline would conceal that tradeoff.

4. Make integration failures explainable without retaining rejected text.

The adapter now records the stage at which a failure occurs: transport, HTTP, provider JSON, candidate handling, model JSON or decision validation. Decision validation distinguishes six reasons: object shape, unsupported category, evidence type, evidence too short, evidence too long and evidence absent from the request.

For example, a fabricated quote produces invalid_model_decision with validationReason: evidence_not_in_request. That diagnostic identifies the failed check without copying the rejected quote into the error record.

Failures and successes retain at most two attempt records, bounded timing, HTTP status, known finish reason, candidate count and sanitized usage metadata. A response rejected after decoding can still retain its reported token counts. Missing usage stays unknown; it is not treated as zero cost.

Diagnostics omit raw rejected candidate text, provider messages, headers and request text. Successful decisions still retain the validated evidence excerpt in the evaluation report. This boundary improves observability while limiting what failure records retain.

The diagnostic error paths were demonstrated by injected tests. All 31 tests passed in Linux CI before the run; checks passed on Node 22 and 24. Tests cover malformed output, invalid evidence, hostile metadata, incomplete candidates, retries, HTTP failures, timeouts, oversized responses and propagation into the comparison report.

No invalid output or provider failure occurred in this live run. That observation does not show that the earlier validation failure was eliminated, because the cases and live generation differ. Its rejected raw output was not retained, so the new implementation cannot retrospectively identify the old failure’s exact cause. Diagnostics also cannot explain away a schema-valid policy disagreement.

5. Keep provenance, usage and missing coverage visible.

The adapter and sample were frozen at revision d3664b3. The dataset records the pinned Bitext revision, raw-file hash, CSV line, original intent, instruction hash and selection rank. The report records dataset and prompt hashes; the prompt hash is unchanged from the first Gemini experiment.

The first live run made 40 attempts with no retries and executed zero business actions. It used the existing unbilled evaluation project. No billing change was made.

Provider-reported total usage was 10,531 tokens, with total-token metadata present for all 40 attempts. This is usage evidence, not a billing statement. Recorded attempt latency had a median of 894.5 ms and a nearest-rank p95 of 1,420 ms, excluding the deliberate 13-second spacing. These figures are not an end-to-end service guarantee.

The slice contains no cases expected to take the technical route and no dedicated security, injection, sensitive-context or multi-intent strata. Its small size, public training-data provenance and non-independent workflow mapping limit the conclusions. One run does not measure service reliability, and no result here establishes production readiness.

The earlier hand-authored set and this source-derived set have different case mixes. Their headline percentages cannot establish an improvement or regression. Both sets are now seen; tuning on them and presenting the same measurements as untouched evidence would weaken the evaluation.

Original synthetic instructions and intent labels are attributed to Bitext Innovations. Our selected dataset and the report’s source excerpts retain CDLA-Sharing-1.0; the linked records include attribution and a modification notice. The article summarizes the experiment; inspect the licensed records for the underlying examples.

Follow-up: clearer routing, visible failures.

Our new paired evaluation tests the old and clarified prompts on 40 newly authored, separately AI-reviewed cases. It separates five observed routing corrections from service availability differences and reports every failed call.

6. Improve the next question before optimizing the score.

A stronger next evaluation needs independently adjudicated workflow labels, with a clear policy for signup faults and account cancellation. It also needs unseen cases covering the missing routes and risk strata, plus repeated runs that separate semantic decisions, output-contract failures and provider availability.

That is proposed work. The current evidence supports a narrower lesson: better diagnostics make contract failures easier to investigate, while evaluation must still expose valid outputs that miss the workflow’s review requirements.