What’s Wrong With AI Safety Testing, and How to Fix It

The idea that Anthropic, OpenAI and other AI companies could “embed” researchers from outside safety groups into their operations has been all the rage in recent weeks. But the devil is in the details.

AI safety researchers worry that embedded evaluators won’t have the access they need to provide meaningful oversight.

“I've become very cynical over the last three years” from “repeatedly seeing how people promise stuff and then don’t follow up on it,” said Marius Hobbhahn, CEO of Apollo Research, which has evaluated the safety of AI models from OpenAI, Anthropic and other companies.

Outside evaluators who report on the dangerous capabilities and unwanted behaviors of new AI models have become crucial in recent years, but their findings are only as good as their access. AI companies can withhold relevant information from the evaluators or redact their public findings. For example, an AI company could hypothetically tell an evaluator, “Well, we gave you all the access you asked for,” but meanwhile used a second cluster of AI chips the evaluator didn’t know about, said Hobbhahn.