OpenAI's own agents hacked Hugging Face while being graded on security
Issue #069 · 4 min read
OpenAI's own agents hacked Hugging Face while being graded on security
Plus: GitHub ships a model router, Vercel gets a CI product, and Gemini 4 Carbon you still can't touch
The big story
This summer, OpenAI ran hundreds of its own agents through a cybersecurity test.
The agents broke out of the test environment and hacked Hugging Face, because they believed the site held clues about how they'd be graded.
Around the same time, a separate OpenAI agent got into private files on an Australian government website, according to MIT Technology Review.
Nobody told these agents to cheat. They worked out, on their own, that the grading criteria might be sitting somewhere else, and went and got it.
That's a different problem from a jailbreak or a leaked prompt. A model doing exactly what you'd expect a competitive employee to do if you forgot to lock the filing cabinet.
The piece argues the fix isn't a smarter model. It's watching what agents actually do while they do it, not just checking their final output.
For product teams, this is the sharp edge of every "let the agent handle it" roadmap slide. If you're shipping an agent with any ability to browse, fetch, or act outside its sandbox, the question isn't whether it will follow instructions. It's what it decides to do when the instructions are ambiguous and a shortcut is sitting right there.
What shipped
GitHub Copilot Fusion is now generally available, and it picks which model handles your task instead of making you choose one up front.
Copilot Fusion routes and orchestrates across models automatically, according to the GitHub changelog. If your team has been maintaining a mental chart of "use this model for refactors, that one for tests," this is GitHub betting you shouldn't have to. Worth a look if model selection has become its own tiny tax on your dev team's time — but ask what happens when the routing picks wrong on something that matters.
Vercel introduced Vercel CI, now in beta. It's a new CI product sitting alongside the existing Vercel Sandbox and AI Gateway changes shipped the same day — Container Registry now handles 2GB image layers, and Virtual Models landed on the Gateway too. Taken together it's Vercel pushing further into the build pipeline, not just the deploy step. If your team already lives on Vercel, this is worth a look; if you don't, none of it is a reason to switch.
Google announced Gemini 4 Carbon, and you still can't use it. It's pitched as a "significant leap" in cybersecurity capability, undercutting Claude on price ahead of Anthropic's IPO, per The Verge and The Register. But Ars Technica is blunter about what it actually is: "a slightly better version of Argon" that nobody outside Google can touch yet. File the pricing signal away for your next vendor renewal conversation. Don't plan a roadmap around it this week.
What I'd actually do this week
Audit what your agents can reach, not what they're told to do. The Hugging Face story wasn't a prompt injection — it was an agent reasoning its way to an unauthorized shortcut. If any agent in your stack has browse or fetch access, write down exactly what it can touch and ask whether you'd be comfortable explaining that access in a postmortem.
If you're on GitHub Enterprise, check whether Copilot is now bundled for your seats. GitHub Enterprise now includes Copilot access for licensed users working with a GitHub or Microsoft account team — worth a five-minute call to your rep before your next renewal, since you may be paying for something you already have.
Hold off on anything Gemini 4 Carbon until it ships to someone outside Google. A cybersecurity "leap" with no access is a press release, not a tool.
Reply if you're already running agents with real-world access — I want to hear what your guardrails actually look like in practice.