THE SHORT-CALL BENCHMARK
One million routing decisions.
A very different bill.
Estimated cost at the request lengths measured in our test, before any confidence-based handoff.
See the full comparison93.0% accuracy
95.6% accuracy
Zuse 9B: 98.8% lower cost, with accuracy 2.6 percentage points lower on these 900 calls.
Python examples, benchmark code and saved results.
A ticket arrives. Where should it go?
Before anyone writes a reply, a support desk has to make a decision: billing, delivery, account access, or another team. At scale, that small choice becomes a workload of its own.
A general-purpose language model can make that choice. Zuse is built specifically for decisions: give it the message and a set of options, and it returns a choice with a probability for each option. It doesn’t generate a written reply.
We wanted to measure the difference on a concrete task. So we asked Zuse 9B, Zuse 27B and Claude Sonnet 5.5 to classify the same customer messages.
01 / THE RESULTS
Lower cost. A measurable accuracy gap.
We tested short banking calls from MINDS-14 and longer complaints from the CFPB’s public database. Each model received the same texts within each task.
Short bank calls
900 calls · 9 languages · 14 intents
| Model | Accuracy | Cost / 1M |
|---|---|---|
| Zuse 9B | 93.0% | $16 |
| Zuse 27B | 92.8% | $98 |
| Claude Sonnet 5.5 | 95.6% | $1,322 |
Longer bank complaints
270 complaints · classified by product
| Model | Accuracy | Cost / 1M |
|---|---|---|
| Zuse 9B | 75.2% | $49 |
| Zuse 27B | 78.5% | $304 |
| Claude Sonnet 5.5 | 80.4% | $2,046 |
Runs recorded 15 November 2026. Costs are extrapolated to one million decisions from the measured requests; longer inputs increase cost. Sonnet ran on Amazon Bedrock with thinking disabled and structured output. The calculation uses the list prices recorded for the test, without batch discounts.
Sonnet was more accurate on both tasks and answered faster in these runs. These results describe two classification tests, not every customer service workflow. Inspect the code and saved results ↗
02 / HANDLING UNCERTAINTY
You don’t have to automate every ticket.
A wrong route can cost more than the model call. Zuse’s probabilities let you set a threshold: accept a decision above it, and send the remaining cases for review.
On the bank-call test, keeping only Zuse 9B decisions with a reported probability of at least 90% covered roughly four in five calls. Accuracy on that accepted subset was 98.9%.
A workflow with a review step
Probability ≥ 90%
Accept the route
About 80% of calls · 98.9% accuracy on this subset
Probability < 90%
Send for review
About 20% of calls · a person or another model decides
The threshold result is for bank calls only. Validate it on your own tickets; a probability is not a guarantee. Review costs are additional to the model costs above.
Why the model bill is smaller
The difference comes from both the model’s price and the work it does. In the complaints test, Zuse used 405 input tokens per complaint compared with 940 for Sonnet. Zuse also has no generated output tokens.
The test used $0.12 per million input tokens for Zuse 9B, versus $2 for input and $10 for output for Sonnet. Those are token prices; the tables above translate them into the cost of actual decisions.
Zuse 27B improved on 9B for the longer complaints, but not for the short calls. A larger model wasn’t automatically the better choice. The right starting point is the task you need to solve.
03 / TRY IT YOURSELF
Start with one decision in your queue.
The repository includes five Python examples you can adapt. Only routing and classification are compared with Sonnet here; the other examples are starting points for your own evaluation.
- 01
Route a ticket
Choose the team responsible for the customer’s request.
- 02
Escalate a conversation
Flag requests for a person or cases that need closer attention.
- 03
Check a reply
Screen an agent’s draft against rules before it is sent.
- 04
Apply a refund policy
Evaluate a request against the policy you provide.
- 05
Organise feedback
Group survey comments into topics your team can act on.
A message in. A team out.
Define the teams and send a customer message. This request asks Zuse to choose between billing, delivery and other.
curl https://api.bineric.com/v1/zuse \
-H "Authorization: Bearer $BINERIC_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "zuse-9b",
"state": "I was charged twice for the same order.",
"questions": {
"team": {
"type": "choice",
"instructions": "Which team should handle this ticket?",
"criteria": {
"billing": "Payments and refunds",
"delivery": "Shipping",
"other": null
}
}
}
}'Keep a language model where you need writing.
Zuse handles choices from a defined set. For drafting replies, summarising conversations or extracting names and amounts, use a language model. A support workflow can use both: one to route the message, another to help answer it.
What does your queue cost to route?
Run the examples on your own tickets. Compare accuracy, cost and the cases that need review.