Agentic AI Companies: How to Evaluate One Before You Sign

The phrase agentic AI companies now covers everyone from serious engineering firms to agencies that added the word to a landing page last quarter. The output looks similar in a demo. It is not similar in production, and the difference is checkable before you sign.

What an agent actually has to have

An agent that works in production, as opposed to one that impressed a room, has five things a demo does not need. An evaluation set built from your real cases, scored before launch, so there is a number to improve. Tool permissions scoped per function, so the agent can read the CRM but not delete from it. A human boundary at a defined confidence threshold, with the context travelling to the person who takes over. Tracing on every run: inputs, tool calls, cost, latency. And a plan for what happens when the model provider changes a version underneath it.

Agentic AI companies that build these routinely will show them to you without being asked. The ones that do not will talk about the model.

Nine questions

1. How will you evaluate it, and against what? Real conversations or cases from your history; retrieval and reasoning scored separately.
2. What can the agent do, and what can it not do? Ask for the permission model per tool.
3. Where does a human take over, and what do they receive? A threshold and a handoff payload, not "it escalates."
4. What is logged per run? If the answer is "the conversation," there is no observability.
5. What happens when the model version changes? The honest answer involves the evaluation set again.
6. What will inference cost at our volume? Routing simple steps to a cheap model typically halves it; a company that cannot estimate has not run one long.
7. Who owns the repository, the cloud account and the prompts? You, from day one.
8. Is the price fixed against a written scope? Hourly agent work rewards the clock.
9. Give me a reference I can phone. Then phone them.

Reading the answers

You are not looking for perfect answers to all nine. You are listening for whether the company has thought about each one, because the questions they have not thought about are the failure modes that surface in month three. Our guide to vetting an AI agent partner goes further on the evaluation itself, and our enterprise agent governance framework covers what the boundaries look like at scale.

A demo we sat through, and what it hid

Last spring a prospect forwarded us a recording of another vendor's demo. The agent answered a support question, looked up an order, issued a refund. Slick. The prospect wanted to know why our quote was higher. So we asked for three things: the evaluation set, the tool permissions, and the log from that run. None existed. The refund tool had a general API key with delete rights on the customer table. The order lookup was a hard-coded record. The answer came from the model's training, not the company's documents, which is why it was fluent and why it would have been wrong the first time a policy changed. That vendor was not dishonest. They had built a demo, and demos are a different product from systems. The prospect went with them anyway, and came back to us seven months later with a support queue that had grown rather than shrunk. I tell this story not to score a point but because it is the most common shape of failure we see, and it is invisible in the sales meeting.

What the contract should say

Most of the risk in an agent engagement can be moved into the paperwork before anyone writes code. Four clauses matter. First, an acceptance test defined as a number: the agent resolves at least this share of a named set of real cases without a human, and the set is agreed before the build. Second, ownership: repository, prompts, evaluation data and cloud accounts in your name from the first commit, with no licence back to the vendor. Third, a change process: what happens when the model provider ships a new version, who re-runs the evaluation, and who pays if scores drop. Fourth, running costs: an inference estimate at your expected volume, written into the proposal, with a routing plan for keeping it there. A company that resists any of these is telling you how the project will go. A company that offers them unprompted has done this before.

Signals that are not as useful as they look

Some things buyers weigh heavily predict very little. Headcount is one; several of the best agent deployments we have seen came from teams of five or six, and several of the worst from firms with hundreds of engineers who had never run one past a pilot. Model partnerships are another. A logo on a partner page means a commercial arrangement, not competence, and every serious team works with all three major model providers anyway. Case studies with no numbers and no named client are marketing copy. Speed of proposal is the last. A quote that arrives the same day, without a scoping call, is a template with your name pasted in. The signal that does hold up is boring: can the person who will actually build your agent explain, on the first call, how they would evaluate it and what it would cost to run. If that person is not on the call, ask why.

How the good ones price

Pricing tells you a lot about how a company thinks. The agentic AI companies we respect quote a fixed price against a written scope, and the scope names the workflow, the tools, the integrations and the acceptance number. That is not because fixed price is generous; it is because a company that has built a dozen of these can predict the work. Hourly quotes, or day rates with an estimate attached, mean the vendor is pricing their own uncertainty into your invoice. Watch also for what is excluded. Retrieval, evaluation, hosting setup and the first month of tuning are the four items most often left out of a low quote and billed later as change requests. A focused single-workflow agent from a competent team now lands between about $4,000 and $10,000 and a multi-step agent with retrieval and several integrations between $13,500 and $41,000. Quotes far below those usually describe a prompt on a general model, and quotes far above them usually describe an enterprise programme rather than an agent.

If you only have time for one question

Ask to see the evaluation set from their last deployment. Not a summary of it. The actual cases, the scores, and the date it was last run. Every other question in this article is a proxy for that one. A company that has an evaluation set has thought about what correct means for their client's workflow, has measured it, and has a way of noticing when a model update breaks it. A company that does not have one has been shipping on feel. They may be talented, and their demo may be better than anyone else's, but they cannot tell you whether the agent works, because they have never defined it. When we lose a deal, it is usually to a company that could not answer this question, and when the client calls back a few months later, this is the question they wish they had asked.

What it costs

A focused single-workflow agent, one job, one or two integrations, human review on exceptions, typically runs $4,000 to $10,000 and ships in four to six weeks. A multi-step agent with retrieval over your own content and several integrations runs $13,500 to $41,000 over eight to fourteen weeks. Enterprise deployments with private hosting, SSO and audit logging scale above that. Our Saudi Arabia agent cost guide shows the same bands in a regulated market.

We build AI agents on exactly the five requirements above, and we would rather lose a project on question 1 than win it on a demo. If the question is whether you need an agent at all, our AI agent vs chatbot comparison will tell you.