How to evaluate an AI agent or automation vendor

Ask an automation or agent vendor whose accounts the system runs in, which operating-system user each agent runs as, what happens on the day you stop paying, how an agent gets stopped at two in the morning, and who reads the logs. A demo shows the happy path. Those answers show the system you would be running a year from now.

Use the list in the first call. It takes the vendor ten minutes to answer and tells you more than a slide deck.

What should you ask about ownership?

The answers decide whether the proposal sells you a working system or a subscription.

• Whose accounts do the workflows run in, named one by one?
• Do the workflow source, configuration and credentials come with the build, or stay with the vendor?
• What stops working on the day we cancel, in writing?
• Is support optional, or is there a mandatory retainer?

What should you ask about the agent itself?

Only the operating system can answer these. A policy document is not an isolation boundary.

• Which operating-system user does each agent run as, and what else shares it?
• Where do the agent's credentials live, and can each one be revoked on its own?
• What tools can the agent call, and which of those calls cannot be undone?
• Which steps need a named human to approve before anything leaves the company?
• How is an agent stopped, and how is an action reverted?

What should you ask about the model?

These tell you whether the recommendation is an engineering judgement or a partnership commitment.

• Which model does each agent use, and who picked it?
• Can the agent be pointed at a different model, cheaper or self-hosted, without a rebuild?
• Is model choice tied to a partnership or commission the vendor holds?

What should you ask about operating it?

These decide what your first week after handover looks like, and whether you need the vendor again.

• What does the audit trail contain, and where does it live?
• What alerts exist, and who receives them?
• What documentation do we get, and can our own staff change a rule without you?
• What is the runbook for a failure at two in the morning?

What are the red flags that should end the call?

A shared service account, or one master credential across several agents. No audit trail. An agent with write access to production and no approval gate. A demo that only runs in the vendor's environment. Per-seat pricing on a system you supposedly own. A fixed price quoted before anybody asked about the process.

What does a good answer sound like?

Specific, boring and checkable. Named accounts, named users, a link to the log, a documented stop procedure, a model name and version per agent, and a plain "we would not automate that step" when the process is not worth it.

Those are the answers we build to. See AI agents: integration and management and agent isolation. A vendor who will not put them in writing has answered already.