How Companies Are Using Old-School Software to Grade AI Agents

Measuring the quality of an AI model’s work can be tricky, especially when it comes to AI agents that can take a broad range of actions to autonomously solve a task. As businesses turn to agents to automate longer and more important tasks, they are getting savvier about how to evaluate the performance of those agents.

“Enterprises are getting more sophisticated,” said Brad Kenstler, director of general agents at Scale AI, where he focuses on developing evaluations for Scale’s enterprise customers. “They’re looking to deploy agents into much more business-critical operations.”

One surprisingly important way that enterprises are improving evaluations is by grading agents using software programs, rather than relying on another AI to serve as a judge. Scale is discussing developing these so-called “programmatic verifiers” with all its customers, including Mayo Clinic, real-estate and finance firm Howard Hughes and educational publishing company Cengage.