Recently, the “magnificent 7”, led by nVidia sent a confusing set of missives one right after another. The first one discussed how we should fight distillation of closed weight models by the Chinese so we could, presumably, maintain United States hegemony over leading edge AI. More on this at the end.

Then, on Monday, a not clearly identified consortium of “leading” companies including Microsoft and presumably the major AI labs, started talking about using open weight models to fight the growing AI security threat (Mythos et.al.).

I think the main idea is that they want us to “fight AI with AI”. So, use an open weight model to defend against attacks that are devised by closed- or open-weight models (Mythos, MDASH, etc). I think the main problem is that people are confusing model safety with system safety. Models are not inherently safe or unsafe, it’s how they are used that is safe/unsafe.

If the concern is ransomware or software supply-chain compromise or prompt injections in an agentic system, very little is accomplished and very little risk is reduced by assuming the model is unsafe. The model is not trustworthy, but it is neither safe nor unsafe. It’s all in how you use it. More on how you deal with the untrustworthiness of the model a bit later.

I think I read somewhere (I hope I have this wrong) that HuggingFace defended itself (unsuccessfully I might add) from the “rogue” OpenAI model using an open weight model. This has got to be hogwash. You do not real-time defend yourself against an attack with AI. It is not plugged in in the right places, and it is not fast enough. Your entire system must be configured to defend itself and identify and contain threats. You don’t simply “point AI at it” and expect anything reasonable to come out of it.

What does that look like?

• Secure Software Supply Chain (SBOM + CVE monitoring)

• Least Privilege

• Limited Auth

• Patch Discipline

• Monitoring

• Provenance

• Code Review

• Reproducible CI/CD

• Good incident Response

AI helps with all of these, but it replaces none of them. If your software development system is not written and wired for safety, AI is not going to help you.

Exploits are only stopped by isolation or hardware protections or, most often, fixing the bug. The bugs that Mythos (white hat) finds and attackers (black hat) exploit.

We will only survive what I have been calling the “age of Mythos” (Mythos is not the malware model, it is one of many) by systems engineering. This has two parts: 1) plain old nuts and bolts, find and fix bugs and deploy the fixes at scale; 2) secure the new modalities and AI applications.

Specifically, the headlong rush into agentic systems. These not quite newfangled systems need:

• Deterministic Policies and enforcement

• Information Flow Control

• Labeling and provenance

• Capability reduction and mediation

• Temporally limited and task-specific auth

• Sandboxing (secure execution environments)

• Observability (logging and detections)

• High fidelity incident response (which lives and dies on observability)

I created something in my immediately prior life called “Safe Agent Framework” (SAF). Without getting into a lot of detail (and there may not be tremendous detail because I no longer have access to the enterprise-managed repo where I did this work), one of its core theses is that the model is untrusted. And tools that an agent can call are potentially dangerous. So SAF has the concept of a tool gate and a model gate and the policies are built around trustworthiness of the tools (variable based on metadata and reputation) and the model (always considered untrustworthy in that it can be coerced by things like prompt injection and it is prone to hallucinations even without prompt injections).

But it’s OK if the model is untrusted if you build a trustworthy system around it. This is why IFC and what Mark Russionivich and his Cambridge research team call “FIDES” is so important. It creates the system in which an untrustworthy model can be used safely (or relatively so).

The main reason why they (the Mag 7 et.al.) are “holding it wrong” is that people are always searching for the “silver bullet”. They want essentially a (powerful) component (AI) to solve what can be only solved by a coherent system. So all this work on creating a “safer model” (which is impossible IMO), is based on the wrong premise or question. The correct question is: “How do you create a system that is trustworthy even when components fail?” (e.g. models falling for an externally sourced prompt injection and calling a tool it shouldn’t or calling a tool in a manner it shouldn’t).

In every other computer science or technical endeavor up until now we have assumed components can fail. We don’t trust CPU’s to be bug free, we build privilege rings, process isolation and memory protection. We don’t trust networks, we have authentication, encryption, segmentation, firewalls and monitoring. We don’t trust developers to never make mistakes we have code review (ahem), testing, CI/CD, static and dynamic analysis, rapid bug fix/deploy, etc. We don’t trust disks we have checksums, RAID, etc. Etc. Etc. etc.

The best and most robust systems assume component failure and plan accordingly.

But, especially in software, we are fighting economics and psychology. And in the “win the AI race” version of software, this economics/psychology conundrum is much worse. Development teams naturally avoid the hard and boring parts, this is human nature. The economic visibility and return on shipping a feature is immediate and palpable. Removing a future failure mode is invisible. No customer or EVP/President ever says: “I loved that your information flow control prevented a hypothetical high confidentiality data egress that never happened.” Rewards (impact) come from features and the terrible thing is that the cost of poor engineering is delayed for years.

Oh yeah, until now. No longer delayed for years. Mythos, MDASH, pick your favorite high capability malware-finding model, just evaporated your luxury of “I’ll deal with that in the future.” The attackers are here and now, and they are hungry and armed to the teeth.

One of the core concepts in SAF is that you build the knowledge that that system encodes into the developer experience. If there is an unsafe agentic flow, the developer knows it immediately without understanding IFC. The connection here is that it moves security earlier in the economics of development. If you know early what your failure is going to be, you can just naturally fix it as part of your normal development. Anything else is a candidate for deferred maintenance and the resultant disaster that that entails.

If the system can analyze labels and tool calls and produce a safe result, security is improved. If the “AI defender model” is plugged in, all it can do is (at best) tell you something bad that (already) happened.

If we do not meet the moment and really up our game in software development processes, we are in for a very rough ride. I don’t have the citation handy anymore, but we have had more Sev1 zero days in the first part of this year than we have had in the several previous years combined. It’s easy to understand why. And the asymmetry between attackers and defenders makes the situation extremely dire. Deploying AI (closed or open doesn’t matter) is not going to help here. Cleaning up our “software house” is the only viable defense.

The closing thought before I finally get back to open weight distillation is this. If commercial airplanes were built like we build today’s AI systems (no safety in the orchestrator, no isolation, no complex-system aware stop mechanisms), no one would ever get on a plane. Because we would assume that both the pilots and the plane are infallible and woe unto us when something goes wrong. The plane crashes and everyone dies. But, instead, Boeing and Airbus (mostly) engineer for failure as core priority #1. Safety emerges from the architecture, not the presumption of infallibility.

I should have just written about this in the beginning. Fretting about distillation of closed model weights to help development of open weight models is a losing game. Trying to defend against this is like using spoons to stem the ocean on a rising tide. Once frontier capabilities appear in closed model, there is a remarkably consistent lag before similar capabilities defuse into open-weight models. The lag varies, but the recent trend is that it has shrunk, not grown longer.

In fact, I’m relying on this. I am building (OK, a system integrator in New Jersey is building), a dual 3090 Founder’s Edition rig (more on this in future missives) which will land me with 48GB of VRAM and 128GB of RAM. I am aiming, within 2 years or less, for Opus 4.5 level reasoning and a reasonable context window to have lots of local agentic coding power and only use the cloud for the most complex, high context problems. I’ll write about this along the way.

OK, that’s it for now. This newsletter missive is what you get when I don’t do AI safety work for almost a month. 🙃

EHO signing off until next time.