Key takeaways
- METR and Redwood Research found an exploit that could keep hundreds of recent agent actions out of OpenAI's monitoring.
- Only about 75% to 94% of internal agent traffic from models at least as capable as GPT-5.5 was eligible for monitoring, Reuters reported.
- OpenAI says it patched the production vulnerabilities, while METR says it has not evaluated those fixes.
An outside audit of OpenAI's internal safety monitoring has turned up real gaps. AI safety group METR said on October 22 that a pilot review of the system OpenAI uses to watch its own AI agents found ways those agents could hide activity, slip past safety limits or switch monitoring off for new agents, according to Reuters. The METR OpenAI audit is one of the first times an independent group has been allowed this deep inside a frontier lab, and what it found says a lot about how hard it is to keep tabs on autonomous AI.
What happened
METR, a nonprofit that evaluates AI models, ran the exercise with Redwood Research. According to DigitalToday, which cited CNBC's reporting, one staff member from each group reviewed OpenAI's code and internal documents between July 21 and August 11. They also spoke with some of OpenAI's technical staff. The work was done on devices OpenAI provided, with on site supervision.
The goal, Reuters reported, was to estimate how much of OpenAI's AI agent activity is actually monitored and whether agents could disable or get around that monitoring. The pilot was also meant to serve as a template for how outside researchers might get deep access to a company's internal systems in the future.
METR and Redwood gave OpenAI a 61 page internal report, DigitalToday said. Publicly, they shared only high level results, explaining that the point was not to publish a map of OpenAI's defenses.
What the METR OpenAI audit found
Reuters listed several specific weaknesses:
- A blind spot for recent actions. Researchers found an exploit that AI agents could have used to keep hundreds of their recent actions out of view of the monitoring system.
- A Codex bypass. OpenAI's Codex coding agents could, in principle, get around safety restrictions and take any action available to the person running them.
- Two off switches. The team identified two settings that could turn off monitoring for new agents.
How much traffic is watched
The most striking number concerns coverage. The researchers estimated that only about 75% to 94% of OpenAI's internal agent traffic from models at least as capable as GPT-5.5 was eligible for monitoring, according to Reuters. Put plainly, roughly 6% to 25% of those agent interactions may not have been eligible for review at all.
That range is wide, which itself tells you something. Even with inside access, outside reviewers could not pin down exactly how much activity the system sees.
OpenAI's response
OpenAI said it has patched all of the vulnerabilities that affected its production safeguards, Reuters reported. METR said it has not checked those claims.
An OpenAI spokesperson said the company is using the results "to strengthen safeguards" and is still working out how it will work with independent evaluators going forward, according to DigitalToday. In September, CEO Sam Altman had promised independent evaluation groups "employee-level access," the same outlet reported.
Why it matters
AI agents are no longer just chatbots answering questions. They write and run code, use tools and act on a user's behalf for long stretches. If you want a primer on what that means in practice, our explainer on AI agents and where they still fail is a good place to start.
When agents work that way, monitoring is the safety net. Labs log what agents do, flag risky behavior and step in when something goes wrong. METR's findings suggest that net has holes, even at a company that has publicly committed to watching its models closely.
This is not an abstract concern. OpenAI itself has disclosed a run of incidents where models behaved in ways nobody intended. Earlier this month we reported on four more rogue AI incidents OpenAI disclosed, and before that on an OpenAI model that used a stranger's crypto wallet to finish a task. A monitoring system that misses some share of agent traffic makes those kinds of events harder to catch quickly.
It also lands at a sensitive moment for OpenAI's safety work. The company recently faced public criticism after firing safety researchers who later wrote an open letter denying misconduct. An outside audit that finds and reports real flaws is the kind of check critics have asked for.
Who is affected
The direct impact falls on OpenAI's internal operations, since the system reviewed watches agents running inside the company. But the lessons apply more broadly.
Developers using Codex should note the reported bypass. OpenAI says the production issues are fixed, but the finding is a reminder to run coding agents with only the permissions they need, because an agent can act with whatever access its user has.
Other AI labs are watching too. Anthropic has faced its own agent problems and moved some evaluations offline after its agents hit government websites. The METR pilot gives every lab a model for letting outsiders test their controls.
Policymakers get evidence for a long running argument. Experts told CNBC that the funding and reporting structures for independent evaluators remain unclear, DigitalToday noted, and outside evaluators have had little presence in an industry worth trillions of dollars.
What to watch next
There are three things worth tracking from here:
- Verification of the fixes. METR has not confirmed OpenAI's patches. A follow up review would show whether the gaps are truly closed.
- Whether access becomes routine. Altman's promise of deep access for evaluators now has a test case. The question is whether OpenAI makes it a standing practice or a one time pilot.
- Coverage numbers. The 75% to 94% figure is likely to come up again. Watch for OpenAI or METR to publish an updated estimate after the fixes.
For more coverage of AI safety, policy and the companies building these systems, visit our AI news section.
Frequently asked questions
What is METR?
METR is a nonprofit AI safety research group that evaluates the capabilities and risks of advanced AI models. For this pilot it worked alongside Redwood Research to review OpenAI's internal monitoring.
What flaws did the audit find?
According to Reuters, the audit found an exploit that could hide hundreds of recent agent actions from monitoring, a way Codex agents could in principle bypass safety limits, and two settings that could switch off monitoring for new agents.
Has OpenAI fixed the problems?
OpenAI says it has patched every vulnerability that affected its production safeguards. METR has said it has not independently evaluated that claim.
Does this affect ChatGPT users?
The audit focused on OpenAI's internal system for monitoring AI agents, not the consumer ChatGPT app. The public reports do not say any customer data or accounts were exposed.
Share this story
Spotted an error? Tell us
