New posts from Sean Goedecke
Thank you for subscribing! Bear with me as I figure out how to send these emails. Here are the blog posts I wrote this week:
Why can't language models come up with new ideas?
Why can’t large language models come up with new ideas? Human polymaths routinely come up with interesting new ideas by combining insights from different areas. But language models have not come up with any notable new ideas, despite being familiar with a far broader range of knowledge than any human. Is that surprising? What’s going on here?
AI skeptics like to say that the inability to generate new ideas is inherent to the nature of language models. The idea here is that language models work by estimating the next most likely token based on all the tokens they’ve seen in their training data, so structurally they’re just built to only output things that they’ve already seen before during training. Coming up with genuinely new ideas is thus impossible, because they’re bounded by their training data. At best they can reshuffle or recombine ideas that humans have already had. I don’t buy this argument even a little bit. First…
The importance of virtue in software engineering
Software engineers care a lot about being smart and knowledgeable. Conversations about how to become a better software engineer often center around learning more facts: programming language syntax, design patterns, details of how particular technologies work, and so on. It’s also undeniable that having a strong working memory is really useful, if only so you can fit more of a codebase in your head. However, there’s a whole other dimension to being a strong software engineer that doesn’t get talked about as much: character.
What do I mean by character? I mean the kind of person you are: the quality of your internal self. For want of a better word, your “virtue” or “moral fiber”. In my experience, people with good character are dramatically more effective software engineers. There are approximately one million books about learning software engineering skills and approximately zero books about software engineering character. It’s a bit unfashionable today to talk about character, for some…
How projects fail at large tech companies
How do projects fail at large tech companies? As I’ve said many times, failure means executives aren’t happy with how the project turned out. At healthy companies, that typically means that a sensible engineer wouldn’t be happy either, because the project didn’t work or users hated it. But what actually causes the projects to fail? I’ve seen a lot of projects go wrong - both up close and at a distance - in the last ten years. Here are the main reasons why.
Lots of projects fail because there’s no way they could possibly have succeeded. In American law, some cases get dismissed at “summary judgment”: even if the plaintiff succeeds in proving everything they aim to prove, it still wouldn’t add up to demonstrating enough illegal activity to win their case. At tech companies, some projects are like that: even if the plan goes off without a hitch, the project is still doomed to fail. Some doomed projects begin with over-ambitious plans. For instance, an executive…