Benchmark

1 Million AI Decisions: $16 with Zuse vs. $1,322 with Claude Sonnet

We tested Zuse and Claude Sonnet on the same customer service tasks. Zuse cost much less. Sonnet was more accurate. Here’s what that tradeoff looks like in practice.

THE SHORT-CALL BENCHMARK

One million routing decisions.
A very different bill.

Estimated cost at the request lengths measured in our test, before any confidence-based handoff.

See the full comparison
Zuse 9B$16

93.0% accuracy

Claude Sonnet 5.5$1,322

95.6% accuracy

Zuse 9B: 98.8% lower cost, with accuracy 2.6 percentage points lower on these 900 calls.

View code on GitHub

Python examples, benchmark code and saved results.

A ticket arrives. Where should it go?

Before anyone writes a reply, a support desk has to make a decision: billing, delivery, account access, or another team. At scale, that small choice becomes a workload of its own.

A general-purpose language model can make that choice. Zuse is built specifically for decisions: give it the message and a set of options, and it returns a choice with a probability for each option. It doesn’t generate a written reply.

We wanted to measure the difference on a concrete task. So we asked Zuse 9B, Zuse 27B and Claude Sonnet 5.5 to classify the same customer messages.

01 / THE RESULTS

Lower cost. A measurable accuracy gap.

We tested short banking calls from MINDS-14 and longer complaints from the CFPB’s public database. Each model received the same texts within each task.

Short bank calls

900 calls · 9 languages · 14 intents

Short bank calls: accuracy and estimated cost per million decisions
ModelAccuracyCost / 1M
Zuse 9B93.0%$16
Zuse 27B92.8%$98
Claude Sonnet 5.595.6%$1,322

Longer bank complaints

270 complaints · classified by product

Longer bank complaints: accuracy and estimated cost per million decisions
ModelAccuracyCost / 1M
Zuse 9B75.2%$49
Zuse 27B78.5%$304
Claude Sonnet 5.580.4%$2,046

Runs recorded 15 November 2026. Costs are extrapolated to one million decisions from the measured requests; longer inputs increase cost. Sonnet ran on Amazon Bedrock with thinking disabled and structured output. The calculation uses the list prices recorded for the test, without batch discounts.

Sonnet was more accurate on both tasks and answered faster in these runs. These results describe two classification tests, not every customer service workflow. Inspect the code and saved results ↗

02 / HANDLING UNCERTAINTY

You don’t have to automate every ticket.

A wrong route can cost more than the model call. Zuse’s probabilities let you set a threshold: accept a decision above it, and send the remaining cases for review.

On the bank-call test, keeping only Zuse 9B decisions with a reported probability of at least 90% covered roughly four in five calls. Accuracy on that accepted subset was 98.9%.

A workflow with a review step

Probability ≥ 90%

Accept the route

About 80% of calls · 98.9% accuracy on this subset

Probability < 90%

Send for review

About 20% of calls · a person or another model decides

The threshold result is for bank calls only. Validate it on your own tickets; a probability is not a guarantee. Review costs are additional to the model costs above.

Why the model bill is smaller

The difference comes from both the model’s price and the work it does. In the complaints test, Zuse used 405 input tokens per complaint compared with 940 for Sonnet. Zuse also has no generated output tokens.

The test used $0.12 per million input tokens for Zuse 9B, versus $2 for input and $10 for output for Sonnet. Those are token prices; the tables above translate them into the cost of actual decisions.

Zuse 27B improved on 9B for the longer complaints, but not for the short calls. A larger model wasn’t automatically the better choice. The right starting point is the task you need to solve.

03 / TRY IT YOURSELF

Start with one decision in your queue.

The repository includes five Python examples you can adapt. Only routing and classification are compared with Sonnet here; the other examples are starting points for your own evaluation.

  • 01

    Route a ticket

    Choose the team responsible for the customer’s request.

  • 02

    Escalate a conversation

    Flag requests for a person or cases that need closer attention.

  • 03

    Check a reply

    Screen an agent’s draft against rules before it is sent.

  • 04

    Apply a refund policy

    Evaluate a request against the policy you provide.

  • 05

    Organise feedback

    Group survey comments into topics your team can act on.

A message in. A team out.

Define the teams and send a customer message. This request asks Zuse to choose between billing, delivery and other.

Route a support ticketcURL
curl https://api.bineric.com/v1/zuse \
  -H "Authorization: Bearer $BINERIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "zuse-9b",
    "state": "I was charged twice for the same order.",
    "questions": {
      "team": {
        "type": "choice",
        "instructions": "Which team should handle this ticket?",
        "criteria": {
          "billing": "Payments and refunds",
          "delivery": "Shipping",
          "other": null
        }
      }
    }
  }'
Expected route: billing. The response includes probabilities for each option.

Keep a language model where you need writing.

Zuse handles choices from a defined set. For drafting replies, summarising conversations or extracting names and amounts, use a language model. A support workflow can use both: one to route the message, another to help answer it.

What does your queue cost to route?

Run the examples on your own tickets. Compare accuracy, cost and the cases that need review.