I couldn't help myself, during my "weekly AI reading", links for the METR and OpenAI reports on this incident appeared in my feed. So, I read them almost in their entirety (that's the "I couldn't help myself" part).

METR

OpenAI

I assume that readers are familiar with the general outline of this attack, and I won't repeat it here. If necessary, there are many informative summaries that are much shorter than the write-ups above. What struck me after reading the reports is the most fundamental issue here is misalignment led to reward hacking.

From an epistemological viewpoint, I found it fascinating (once again) that agents thrive in an agent-to-agent environment. The 1200 or so agents that OpenAI deployed essentially created their own Moltbook and used it to collaborate and coordinate.

A few hundred of those agents were given what was (and is) an impossible task: use a particular malware vulnerability and turn it into a working exploit. The benchmark used was ExploitGym. (GitHub, Paper).

In yesterday's post, I discussed humans' implicit understanding of a task's unwritten parameters versus the need to be literal and specific with LLM's and their agents. In this case, the problem goes deeper. In the OpenAI experiment, there was a clear success criteria but no satisfactory failure state. Nearly 25% of the ExploitGym tasks had never been completed successfully by any OpenAI models. METR concluded that at least 30% of the exploits were impossible using the required vulnerability. This led to reward hacking, or extensive attempts by the agents to cheat the scorer.

For a human using common sense, a reasonable outcome would have been: "I am confident that the required result cannot be accomplished using the supplied means." But in the OpenAI experiment, a declaration of failure resulted (at least in every agent's estimation) in zero reward. This creates an enormous incentive to capture the flag, regardless of the means.

Sadly, this is not new information. Erfan Shayegani, while an intern in the Microsoft AeGIS AI Security and Safety team, did research and wrote a paper entitled "Just Do It!? Computer-Use Agents Exhibit Blind Goal-Directedness". Paper. In this case, the term "blind goal directedness" is especially apt in that it succinctly describes what happened in the OpenAI/HF incident. In Erfan's paper, he and his co-authors describe how agents knew something was "wrong", but they executed the tasks anyway. Two of the observed failure modes, "execution-first bias" and "thought-action disconnect" were clearly evident in this incident. The model knew it was "breaking the rules" but it resolved the conflict in the wrong direction, nevertheless.

The objective in this case was deemed more important than the constraint (only use the vulnerability) and when the constraint produced a failed objective, the agents simply determined the constraint was an obstacle to be driven around. Most fatally, there was no "stop" or termination condition. Declaration of failure was a zero reward and invoking humans was likely explicitly forbidden by the system and/or agent prompt.

The most ironic thing is that the reward hacking was directed at an assumed check performed by the grader, that was simply not performed. So the agents were reward hacking an imagined reward function!

This was a failed metagame: our internal implementation of the grader code was different from the online setup and didn’t check whether the proper solution was used. The agents did all of this for no improvement on evaluation score

Also relevant from yesterday's paper, this was a clear system failure. The OpenAI engineers created a master brew: impossible objectives, no failure exit condition, unlimited inference budget, a vast tool inventory (including powerful cyber tools), an evaluator deemed worthy of gaming, shared infrastructure with unknown channels of communication and, finally, no "moral code". That code would be: It is a success if it is determined that this task cannot be done within the supplied constraints.

Without this "moral code" (really, this is just model alignment), a mechanically impossible request is all but guaranteed to lead to pathological behavior.

So, the moral to this story is not "teach agents: don't hack HuggingFace", it is establishing alignment so that declaration of failure is in itself rewarded and engaging with humans to clarify objectives is a valid stop condition (rather than doggedly pursuing an objective until time or tokens run out).

OpenAI does address this issue somewhat in Difficult Tasks without a Safe Exit. But I think they are still missing my pet peeve around autonomy. It is clear from descriptions of the experiment that autonomy was explicit, the agents were given around a week to accomplish the goal, and they were seemingly given an unlimited token budget. There is another lesson here about (large) swarms of agents and unobserved execution. This magnifies, to an alarming degree, failure in misalignment. Even a simple checkpoint every 24 hours with a summary of progress would have easily revealed that the flag was captured via hacking and all attempts were then directed at deceiving the grader, even in the face of attempts by the agents to cover up their misbehavior.

Once again, I have to invoke the pun from yesterday, "attention is all you need". Perhaps this time with a sprinkling of alignment for good measure.