I read with great interest James Mickens' paper The Implications of Linguistic Illegibility for LLM Security. It makes some important points that correlate well with the Models and Systems thesis I just posted earlier today. But rather than just link a paper that makes some similar points to me, I wanted to call out a couple of interesting items.

James compares taint tracking and information flow control (IFC), correctly treating IFC as the stronger control, but then constrains the comparison to architectures capable of whole-program IFC. He points to IFC languages such as Jif or privilege-separated designs such as Asbestos. What the FIDES team and I have been advocating is more accurately described as boundary IFC: information-flow policy enforced at concrete mediation points, without requiring complete visibility into the computation on either side.

You could argue that universal noninterference is superior, but this creates a situation akin to "perfect is the enemy of good". Boundary IFC is vastly more deployable across all reasonable programming languages and system architectures.

The strongest points in the paper, however, are the myriad examples of linguistic illegibility. These are key supporting arguments for FIDES and the (as yet unpublished) Safe Agent Framework (SAF) which state that:

The model may be epistemically illegible, but the system must still remain operationally legible.

An IFC trace that supports the mediation point shows:

• What entered the system?
• What state was visible?
• What did the model propose?
• What authority did it possess?
• What did deterministic policy decide?
• What actually executed?
• What state changed as a consequence?

The conclusion from our experience with IFC, FIDES and SAF supports the logical conclusion: it is not necessary to understand the model to mediate the system.

The "pullout quote" that stood out for me in the paper is:

Prior research on feature superpositioning suggests that a model’s internal representation space contains more usable features than the size of the token vocabulary space [9]. Thus, we are confronted with a fundamental mismatch of capacity: there are more unique features than unique tokens, meaning that there must be some features which defy a reasonable single-token description. If we could automatically enumerate such features, we could at least empirically observe which kinds of linguistic inputs most strongly trigger those features. However, the number of possible feature directions in even a moderately-sized LLM is vast [24, 27]. The specter of linguistic illegibility therefore seems to be inescapable, at least with our current understanding of internal model dynamics.

This seems both intuitively obvious and completely damning of the notion that "we will monitor CoT (chain of thought) and ensure safety." On the monitorability front, people have been wringing their hands, anticipating the day when models and their agents will invent "neuralese" that evades human comprehension. The present-day reality makes these worries a bit prosaic. The observation of "more features than vocabulary tokens" does not by itself prove illegibility. The model can quite well be emitting faithful English language CoT even while the material supporting computation has no direct representation.

But the danger is the opposite. The human observable CoT reasoning does not need to faithfully represent the actual underlying computation.

It's easy to interpret this sequence as effectively: English thought -> English thought -> English thought but it in no way must be faithful in that sequence. This leads to a false sense of security:

The CoT is beautifully legible, but it is not the execution trace.

I assert that from a security standpoint, we are already in the "era of neuralese." The internal state involved in inference is enormous, high-dimensional, and continuously-valued; the CoT is a narrow sequence of discrete linguistic tokens. The idea that we can simply train a model to serialize all security-relevant causal state into the CoT assumes an extraordinary property for which current architectures provide no guarantee. A security boundary cannot depend upon that assumption.

This means that the CoT (and monitoring of it) can never be a security boundary. Yes, we can and should monitor the hell out of the CoT and we should try to make reasoning faithful, but we cannot rely on it for security decisions.

This has quite grave implications for incident response. When reading CoT logs, there might not even exist an English account inside the model corresponding one-to-one with the consequential action(s) that are under investigation.

This is why I like James' linguistic illegibility better than "neuralese". The fact that there may not be a "hidden sentence" to recover is much more chilling. The causal object is the computation itself.

The implications here re: models vs. systems should be clear to even the most casual of observers.