# How do you know if an AI agent is actually working?

Read 4 numbers every week. Tasks it finished without a person. Handovers it sent to a person. Mistakes caught after the fact. And cost per task. Tasks finished also shows use: an agent can score well and still sit unused. Set a pass mark for each number before launch, so the result is not argued later.

By Surinder Thakur, AI Product Designer and Builder. Published 17 Nov 2026. Topic: [Getting your team to use it](https://www.surinder.design/work?type=getting-your-team-to-use-it).

## Why not measure accuracy alone?

Because accuracy hides 2 problems. An agent can be right and slow, or right and unused.

Business numbers catch both. Tasks finished shows use. Cost per task shows whether it pays.

## What do handovers tell you?

Where the rules are thin. A rising handover rate means the agent meets cases nobody planned for.

Read the handed-over cases weekly. Most turn into 1 new rule, and the rate falls.

## What does a pass mark look like?

A number with a direction, set before launch. In the Ordering agent design: a median of 5 minutes or less from first message to order.

And a stop rule. If wrong-part returns double against the sales reps, the pilot stops at once.

## Who reads the numbers?

The owner of the work, every week, in 10 minutes. Not a data team once a quarter.

A weekly read catches drift early, while the fix is still small.

## In 1 picture

A live agent is judged on tasks finished, handovers, mistakes and cost per task.

Weekly agent report. The agent fills finished, handovers, mistakes and cost. Finished, handovers and cost check out. Mistakes is flagged.

Above the pass mark: the owner reviews this week.

4 numbers, read every week.

## Next questions

- [How do you measure the ROI of an AI agent?](https://www.surinder.design/work/how-to-measure-the-roi-of-an-ai-agent): Count 4 numbers. The hours the task takes today, priced at what those hours cost.
- [How do you test an AI agent before customers use it?](https://www.surinder.design/work/how-to-test-an-ai-agent-before-launch): In 3 stages. Replay past cases your team already decided and compare answers.
- [Why do users stop using AI features after the first week?](https://www.surinder.design/work/why-users-stop-using-ai-features): Because the first week runs on novelty, and novelty is not a job.
- [Can a small team run an AI agent without an AI engineer?](https://www.surinder.design/work/can-a-small-team-run-an-ai-agent): Yes, if 2 jobs are covered. 1 person on your team owns the rules and decides the exceptions.

## Questions that lead here

- [Why do AI apps lose subscribers faster than normal apps?](https://www.surinder.design/work/why-ai-apps-lose-subscribers-faster): Because the wow moment is not a habit. RevenueCat's 2026 data puts AI app annual retention at 21.1%, against 30.7% for everything else.

Page: https://www.surinder.design/work/how-to-know-if-an-ai-agent-is-working
