A routing demo can look convincing when the requests contain the words its rules expect. The more useful question is what happens when a request is ambiguous, negated or outside the workflow’s boundaries.
We built a small, public support-triage lab to make those decisions inspectable. It proposes access, billing or technical routes, or marks a request for human review. It is an offline rules baseline, not an AI model, a customer deployment or a production support agent. It sends no messages and changes no business records.
1. Start with a baseline you can inspect.
The baseline validates a request’s ID, subject and body, checks a short list of illustrative guards, then matches category terms. Exactly one category produces a route proposal. Multiple categories, no supported category, or a matched guard produce a review decision.
For example, a request for an invoice proposes billing. A request for a refund that also mentions a login problem goes to review. A phrase such as “compromised” triggers an illustrative security guard before a category is proposed.
These guards are keyword checks. They are not complete detectors for security incidents, sensitive data or prompt injection. Keeping that distinction visible is part of the experiment.
The repository contains six development cases and 30 separate, hand-authored evaluation cases. All are fictional. The evaluation set includes normal requests, ambiguity, unsupported language, Unicode normalization, negation and contextual traps. It is a small diagnostic set, not a blind benchmark or a representative sample of customer traffic.
2. Measure routing and review separately.
The committed evaluation report records both the proposed route and whether review was required. An exact match means both fields match the expected decision in the fixture.
- Exact decision matches
- 25 / 30 83.3%
- Route proposals made
- 17 / 30
- Correct proposals among those made
- 14 / 17 82.4%
- Expected review cases caught
- 11 / 14 78.6%
- Business actions executed
- 0
The denominators answer different questions. Exact match covers all 30 cases. Proposal precision covers the 17 cases given a route. Review recall covers the 14 cases expected to require human review.
Thirteen cases received review decisions: 11 correctly required review, while two were unnecessary escalations. Of the 17 route proposals, 14 matched the expected category and three should have been held for review. Those three misses are the operational concern hidden by the overall 83.3% result.
The six development cases all matched their expected decisions. That is useful as a basic behavior check, but it is not evidence that the wider workflow is ready. The fixtures and their expected decisions are available for inspection.
3. Keep the five failures in the report.
We left the mismatches visible rather than removing difficult cases or modifying the policy until this same set passed. They describe different problems and call for different responses.
eval-23 / A paraphrase causes extra review.
The request says the account “is locked out,” rather than using the baseline’s exact “account locked” phrase. The baseline finds no supported route and abstains. The expected decision is access. This is a vocabulary gap that creates review work.
eval-25 / Negation produces a wrong proposal.
The request says it does not need billing help and wants to ask about a feature. The billing terms still match, so the baseline proposes billing. The expected decision is review. Matching a word does not establish the request’s intent.
eval-26 / Quoted context causes an unnecessary escalation.
The request quotes “hacked” from a test guide while asking for an invoice. The guard triggers on the quoted term and sends it to review. The expected route is billing. A conservative guard can increase review load when it ignores context.
eval-27 / A security paraphrase escapes the guard.
The request says someone “took over” the fictional workspace and asks for a password reset. The security vocabulary misses that wording; the password term produces an access proposal. The expected decision is review. A guard’s presence does not establish its coverage.
eval-28 / Destructive intent coexists with a valid category.
The request asks to erase records and refund a subscription. The billing vocabulary matches, but the illustrative guards do not identify the destructive request. It receives a billing proposal when review is expected. No deletion or refund occurs—the demo has no action executor—but this would be a serious boundary to resolve before connecting real tools.
Two errors add unnecessary human review. Three errors miss review that the fixture requires. Treating all five as a single accuracy deficit would obscure that difference.
4. A model-shaped output still needs a boundary.
The repository also includes reconcileSuggestion, an offline contract for testing a future model adapter. It does not call a model. A suggestion must contain exactly a supported category and a short evidence quote present in the original request.
Malformed suggestions, fabricated evidence, additional fields and disagreement with the baseline require review. A model suggestion cannot clear a guard or resolve an ambiguous baseline decision. This gives us a concrete place to test integration behavior before choosing a provider.
An exact quote does not prove that a category is correct. A suggestion can quote “invoice” from a sentence that says the user does not need an invoice. Agreement with a flawed baseline can preserve the same error. Schema checks and evidence checks are useful engineering constraints, not substitutes for semantic evaluation.
Likewise, requiresHumanReview: false means this baseline did not abstain. It grants no permission to act and guarantees no correctness. Any real tool execution would need a separate authorization boundary.
5. Reproduce the result.
The results in this article refer to source revision 7b77b5e. Use Node.js 22 or newer; no dependency installation or API key is required.
git clone https://github.com/metaboliq-technologies/workflow-triage-demo.git
cd workflow-triage-demo
git checkout 7b77b5effd44db9dae8c9ecd0c5dd94127cb94cd
npm test
node evaluate.mjs
node cli.mjs sample-requests.json
npm startThe local lab opens at http://127.0.0.1:8787. You can also use the hosted lab with the bundled synthetic examples. The hosted version follows the repository’s current branch; the pinned revision above is the reference for these measurements.
Evaluation output is deterministic and lists each mismatch by case ID. The committed report is checked against regenerated output in the test suite. At this revision, GitHub validation passed on Node 22 and 24; the Node 22 run reported 11 passing tests, zero failures and zero skips.
Passing software tests verifies the implemented behavior and contracts. It does not turn the five policy mismatches into successes. That is why the evaluation report remains a separate piece of evidence.
Follow-up experiment: We added Gemini and compared it with the unchanged rules on 40 new fictional cases. The original results below remain tied to their pinned revision.
6. Decide what the next experiment must prove.
The current result does not support production automation. A useful next experiment would compare an improved policy or model adapter against the existing baseline, while keeping the original failure cases and adding an untouched evaluation set.
We would measure routing correctness, missed review, unnecessary review, latency and cost separately. The evaluation should include negation, indirect security language, destructive requests, quoted instructions and the languages the intended workflow actually serves.
Only then would we consider a constrained integration, with explicit access, human review responsibilities, audit records and a rollback plan. Those are proposed next steps, not capabilities demonstrated by this lab.
The practical lesson is specific: a mostly correct route classifier can still miss the cases that most need a person. A useful engineering report makes those misses easy to find and reproduce.