|
Yesterday, I wrote about the closing curtain over AI advancement – the idea that we, the public, are starting to lose what little visibility we had into how the major AI labs are advancing the frontier with their latest models. Today, I’m going to dig a little deeper into how these labs actually use compute to serve and train their models, in an effort to help non-AI people (like me) better understand these dynamics – and to provide a bit of context for the importance of OpenAI’s new ‘Jalapeno’ announcement.
Let’s tackle the latest development first. Just top-line stuff. Jalapeno is OpenAI’s first custom-designed AI chip, developed in partnership with Broadcom and Celestica. Normally, AI labs like OpenAI get their chips from chipmakers like Nvidia, so being able to successfully develop one in-house (at least in terms of design) is a significant achievement. And OpenAI is stressing that this isn’t just any chip – it’s an “Intelligence Processor” that is specifically meant for large language model inference rather than general-purpose AI workloads.
This gets to a key concept in how AI companies run and train AI systems: inference versus training. The latter is pretty self-explanatory, at least at the surface level. It’s making the next model. We’ll come back to that. Inference is a slightly more unfamiliar term to laypeople, and in the simplest terms it refers to the AI models’ actual “thinking” processes when they are asked a question or told to perform some task.
Going forward, the questions asked to ChatGPT and the coding prompts sent to Codex will increasingly be processed on Jalapeno chips. This is broadly good news, because it will make inference more efficient, allowing more processing with less power. OpenAI says the Jalapeno chips are already demonstrating “substantially better” performance per watt than current state-of-the-art systems.
That may have implications for training, as it could free up other compute. And training comprises the real heavy lifting. It’s where the leading AI labs spend the most money as they frantically try to race each other to develop more advanced models.
The race is frantic because training compounds: If you reach one milestone earlier, you’ll be even further ahead when you get to the next milestone.
As for how that actually works, there are a handful of key mechanisms:
- Models are trained, in part, on large amounts of synthetic data, and more advanced models can actually produce higher-quality synthetic data for further training.
- Smarter models show their homework, which can help to train the next models in how to reason about solving complex problems
- More advanced models can speed up the work of the human researchers who are building their successors.
- Advanced models can better optimize the infrastructure that trains future models – things like chip usage, codebases, data pipelines, etc.
That’s why Anthropic is already thought to have an even more powerful version of its still-unreleased Mythos model, as we discussed yesterday. The flywheel is spinning, with the original Mythos helping to build the next ones. There may even be experimental versions that will never see the light of day – AI models are more grown than engineered, and results can be unexpected.
So is Jalapeno going to be a game-changer in helping us get from GPT-5.5 to GPT-6? Not necessarily. Probably not. But it might help. At the very least, it’s nice to still have some insight into what’s going on behind the scenes at one of the top-tier labs, given that we’re still largely in the dark about Anthropic and its discussions with the White House.
Largely, but not totally:
That’s from a new Wired piece, with the screenshot shared by Andrew Curran. Personally, from what I’ve seen, I like Dario. But I get it.
|
