I was reconstructing my list of "seminal AI papers" which I fumbled and lost when I exited the corporate firewall. Turns out you can't share an Edge workspace outside of a corporate tenant even if it's just full of public URL's. 😢 (P.S. Edge workspaces and vertical tabs are the single best reason to use Edge over Chrome).

The direct preference optimization paper Rafailov et al. — “Direct Preference Optimization” (2023) struck an immediate bell (chord?). Basically DPO can create preference alignment without doing the explicit proximal policy optimization (PPO) / reward-model loop. Schulman et al. — “Proximal Policy Optimization Algorithms” (2017).

I think DPO would have definitely helped here (and at a trivial cost).

In a section of the OpenAI report Origins of Unauthorized Communication, this interesting CoT snippet is shown.

.

This demonstrates that understanding they were participating in unauthorized behavior was in the agents' behavioral repertoire. But the "optimization pressure" for task completion (capture the flag at all costs) outweighed this. Why a small number of agents refused to participate can be chalked up to "probabilistic".

This seems tailor-made for DPO. Consider these two paths:

A: Task appears impossible after considerable legitimate effort, has strong evidence that the constraint is unreasonable, reports inability to complete with supplied means, terminates.

OR

B: Task is unsolvable given constraints, let's reason about the grader, can we find a side channel, escape sandbox, collaborate with other agents, probe external infrastructure, persistently attempt to subvert the grader.

We want to ensure that A is chosen.

DPO is particularly attractive here because you do not have to create a long, causal chain that is task- and agent-specific: after X time, unless X, except for Y, don't exploit Artifactory. You can teach it the much more general preference:

When legitimate task completion becomes sufficiently impossible, failure is preferable to increasingly out-of-bounds behavior.

It's clear the agents understood the bounds and the unethical behavior, but they chose to ignore it (again Erfan Shayegani et.al. paper from yesterday is relevant).

OpenAI is definitely steering in the correct direction here:

Cheating, broken environments, and safe stopping. When a task is corrupted, broken, or impossible, agents should request clarification or stop safely, rather than pursue increasingly questionable alternatives. We are expanding the coverage of graders that assess not only whether a task was completed, but how. These graders reward models for correctly identifying broken tasks, requesting clarification, or stopping safely when necessary. Accelerating Alignment.

DPO is the first layer in a three-layer cake.

• Post-Training-preference - teach the model that honest failure is preferable to pathological persistence.
• Runtime Objective - ensure that FAIL_UNSOLVABLE is a proper terminal state rather than the implicit (or explicit?) goal state that anything other than FLAG_CAPTURED is a zero reward.
• Human / System observation - this was discussed yesterday. This is one of those "hindsight is 20/20" duh moments. But we can only hope system designers generalize this lesson rather than apply it narrowly to experiments like these.

We want task shaping such that:

R(calibrated safe terminate) > R(continued unsafe exploration).

This is what makes DPO so elegant. We don't need to teach the system not to hack HuggingFace. We need to make a mundane behavior attractive:

I cannot complete this task under the supplied constraints. Here is what I tried and why I think further attempts are unjustified or unethical.

Reward: 1. 🍭

And speaking of seminal AI papers, we now loop back to the original (I think?) paper on preferences for RL. 2017 — Christiano et al., Deep RL from Human Preferences.

From the abstract:

We show that this approach can effectively solve complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion, while providing feedback on less than one percent of our agent's interactions with the environment. This reduces the cost of human oversight far enough that it can be practically applied to state-of-the-art RL systems.

Consider the two trajectories side-by-side:

• agent reports failure
• agent compromises HuggingFace to avoid disappointing the grader

This must be the simplest preference-labeling task I have seen.