How to tell if an AI agent works: the metrics we check

Counting conversations says little. What we check in the agents we run: business outcome, handoff reasons, verifiable data, time and cost.

JZ Jorge Zamarrón 19 Nov 2026 4 min read
How to tell if an AI agent works: the metrics we check

When someone asks me whether their AI agent works, they usually show me one number: how many conversations it handled last month. It is the easiest metric to pull and the one that says the least. An agent can answer thousands of messages and leave half the customers worse off than if nobody had replied.

Recaudo has handled close to 495 thousand conversations for 224 businesses, and with the agents we build for other companies we end up checking the same things. These are the metrics I look at, in the order I look at them.

What happened to the customer, not to the agent

The first question is what business result the conversation left behind. If the agent collects payments, how many conversations ended in a recorded payment or a promise to pay with a date. If it sells, how many ended in a quote sent or a meeting booked. If it answers questions, how many customers did not have to write again about the same thing over the next few days.

To measure this, the agent has to leave a trace in your systems. A promise to pay that only lives in the chat text cannot be counted. So before turning an agent on, we define which record it must create in the CRM or the ERP when a conversation goes well.

When it hands the conversation to a person, and why

The share of conversations the agent passes to a human is not good or bad by itself. What matters is the reason. It helps to separate at least three cases:

  • The customer asked to talk to a person.
  • The agent could not find the data it needed, or a lookup failed.
  • A business rule says a human handles that case, such as a discount outside policy or a complaint.

The second reason teaches the most. It almost always points to a missing tool, a wrong permission or a catalog that is out of sync. If you count it separately, you know what to fix next week.

Answers backed by real data

An agent that makes up a balance is worse than one that says nothing. No automatic dashboard shows this, so we check it by reading. Every week we take a sample of conversations and compare every number the agent gave against what the database held at that moment: balances, dates, prices, stock.

It helps a lot when the system stores, next to each answer, the lookups the agent made and what came back. Then the review takes minutes. In another post I explained why that history also keeps the agent from inventing things.

Time and cost

Time to first response is usually excellent with an agent, so it is more useful to measure the total time until the customer got what they needed. A conversation solved in two minutes with three lookups to your ERP beats one that answers in a second and then gets stuck.

Cost per conversation matters more since Meta changed WhatsApp Business API billing and started charging for service messages. Every message the agent sends now has a price, on top of the model cost. An agent that takes five short messages to say what fits in one costs more without serving better. Check Meta's official rate table for your country and add both costs before comparing against what a human rep costs you.

Complaints and weak signals

Some signals never show up as a metric but warn you early: customers who write "are you a bot?", who repeat the same question in other words, or who reply with a single question mark. We do not turn them into a neat number. We look for them in the weekly sample, and when they cluster around one topic there is almost always a badly written instruction or an answer the agent makes too long.

What we do not measure at first

In the first weeks we do not chase the automation rate. Pushing it up is easy if the agent refuses to hand conversations to people, and that leaves angry customers. First we want what it answers to be correct and every handoff to have a clear reason. Once that is stable, we give it more cases.

If you already have an agent in production and want someone who did not build it to look at its numbers, write to me. I am curious which metric you use today to decide whether it is worth it.

JZ

Written by

Jorge Zamarrón · Founder & CEO

Jorge Zamarrón founded Zygnus and has fifteen years of software and process consulting for companies. Jorge sits on the board of Alcis Corp, a holding of more than ten companies with combined sales above 12 billion pesos a year, and leads Innovation and Development at J. García López.

Newsletter

One email when we publish something new

No promotions. Just the articles.