April 25, 2022

Technology changes fast

As a teen I regularly read The Daily WTF, a comedy site about bad programming and bad software jobs. Two articles have stuck with me. The first is this fantastic article about an incredibly wtf radio jammer. The second is No, We Need a Neural Network, which opens with this:

M.A. is one of the world's foremost experts on neural networks. His undergraduate specialty was artificial intelligence, his master's thesis was about genetic algorithms, and his doctoral dissertation covered evolutionary programming. Such an extensive computer science education opened up a wide range of career options, ranging from a professor at a university to ... a professor at another university.

The article was published in December 2006. Within six months NVIDIA released CUDA, by 2009 Raina et al were training neural networks on GPUs, and in 2012 the GPU-trained AlexNet crushed every single image recognition benchmark. It took just over five years for neural networks to go from punchline to multibillion dollar industry.

I think about that a lot.

I don't know if anyone, outside of very niche academic fields, could have rationally predicted the sudden rise of neural networks.² We needed several technological advances to make it commercially viable. Things also needed to align socially. You could argue that even if neural networks were viable, they wouldn't have gone commercial so quickly if image recognition wasn't such a lucrative problem. But image recognition meant that NNs could be short term profitable, meaning a lot more money is poured into it, bankrolling the long term research that gets us where we are today.

(To be clear, I'm much shakier on social and commercial history than technical history, so I don't know how true that is. I'd also speculate that the SaaS also mattered a lot too, as is significantly increases the base of vendors willing to put money in.)

The other fascinating thing about NNs is that it's AI, but the wrong AI. If you asked an AI researcher what language they used in 2006, they'd probably say prolog or a lisp. Some quick ACM Digital library searches show that just under 7% of AI papers between 1980 and 2009 reference one of lisp or prolog, but that number drops to 1.5% of papers published after 2010. It's a crude measure, sure, but it doe agree with the theory that the majority of AI research was in symbolic approaches and expert systems that ultimately didn't have nearly the impact on the industry that neural networks did.

So why did NNs "succeed" while Prolog and LISP were """dead ends"""? Honestly, couldn't tell you. Any answer I give would be post-hoc and not be insightful. My main takeaway from this is that predicting "winners" is really hard. There's no way for someone in 2006 to predict which AI approach would catch on, or even that any would at all! There's no guarantee that there'd be any payoff at all to AI research.

I'd venture more strongly that most research, CS or otherwise, doesn't "recoup the investment" in time and effort. Which paradoxically, means investing in lots of research is more important, because you want to cast a very wide net to see what works. It could make more sense to invest 10k in 100 ideas than 100k in 10 ideas.

(I often think about this in the context of marketing. When I walk through a grocery store and see 100 flavors for potato chips, I wonder if the Kellogg is expecting most of their flavors to go nowhere, and they're instead hunting for the next sour cream and onion.)

(Okay this feels very "first day business student thinks he's solved everything" to me.)

Revisiting Historical Tech

Let's pull back to a topic where I'm not talking out of my ass. While we can't systematize predicting changes of technology, maybe we can systematize reevaluating old tech.

A lot of old ideas were resource-limited. It's a bit cliche to talk about just how fast modern computers are, but it's true! That's one reason we're seeing more interest these days in formal methods. A specification is more useful if you can model check it quickly. Getting accurate comparisons is hard, but based on 5-10 minutes of googling UPPAAL benchmarks, it seems like a great computer could check ~4000 states a second? Whereas TLC on my desktop can crunch 500,000 states a second, and on a significantly more complex language than the timed automata that UPPAAL supports.³

More subtle changes: new practices, new industries. Does anything become more useful in the context of TDD, CI/CD, cloud services, containerization? I guess that's why Elixir is doing well. Does any technology become more useful now that we have memory-safe systems languages? Is there any tech we should reevaluate in the context of a new business opportunities? Is Prolog the perfect language for IoT programming, augmented reality?

(Obviously I don't have answers for any of these. I'm just spitballing what the technique of reevaluation would look like. Structurally, it's an uphill battle. As Bret Devereaux puts it, studying modern history is like trying to sip from a firehose. That includes modern tech history: how do we pull out the "essence" of forgotten technology and what the structural limitations were that have changed? Better "cataloging" would go a long way to making this easier, but that's not something academia or industry does. It's the kind of work I'd do in a heartbeat if I could pay rent with it.)

Even if we do reevaluate old tech, it still might not pan out, because most things don't. But it's a lot cheaper to rediscover forgotten things than it is to invent entirely new things, since a lot of the work is already done.

For the record, I'm always interested in hearing about obscure or exotic technology y'all've stumbled into. I've learned about a lot of cool things from my readers!

If you're reading this on the web, you can subscribe here. Updates are once a week. My main website is here.

My new book, Logic for Programmers, is now in early access! Get it here.