August 24, 2020

Please read the paper before you comment

Yesterday the main character of Twitter was this new paper by Thompson et. al: "Cultural influences on word meanings revealed through large-scale semantic alignment". They used machine learning techniques to show that the meanings of words in different languages are affected by history and geography. This was roundly mocked on Twitter as a standard techno-chauvinism story where a bunch of tech bros think they've discovered a field that has already been around for several centuries, in this case anthropology.

The only problem: just one of the three authors is a "techbro" by any stretch of the word. The second author is a linguistics professor attached to an anthropology department. The third is a psychology professor who specializes in the evolution of language. They also do a literature review of existing work in word meanings, making in very clear they were building on the existing work in anthropology and linguistics. This is an interdisciplinary team that knew exactly what they were doing and how it fit into the broader body of knowledge.

Now, a caveat. I am not an ML expert or an anthropologist or a linguist. I have about as much ignorance of these domains as all the people mocking the paper on Twitter. But there's one big difference between me and them: I actually read the paper. All of this information is on the first page. Even if you don't know about sci-hub, you could still read the abstract and check out the researcher bios! Nobody read the paper before dunking on it. Nobody read the abstract either. About half the people yelling at this paper had only read the press release. The other half hadn't even done that. They had only read tweets about the press release.

Rule of thumb: the more steps you get away from the primary source the more corrupted everything gets. The article is good. The press release is bad. The tweet about the press release is worse. The tweets dunking on the tweet about the press release are terrible. Each one loses more of the critical context we need to actually understand what's going on. Only the paper itself is a primary source. If you want to know what the paper's about, you gotta read the paper!

I'd be less frustrated about this if it was a one-time thing. But it's the same story every single time. Every time a press release gets popular on Twitter or Hacker News or Reddit or whatever, nobody actually reads the primary source. Many of them don't even get past the title of the press release.

And I'd be less frustrated if this was just academic research. But people make claims and judgments on material without having read the material. I ranted a while back about Donald Knuth and the literate programming challenge. These days everybody hails it as a slamdunk victory of the Unix philosophy over literate programming, but that has nothing whatsoever to do with the original article. People are repeating hearsay and tertiary sources when the primary source is right there.

And I'd be less frustrated if this was just essays. But hell, even quotes are like this too. People will offhand use famous quotes to support their arguments, when the context the quote was given in has nothing to do with it. Or even says the opposite! Like "Premature optimization is the root of all evil", which is regularly cited to justify pushing optimization to the very end of development. It's yet another misunderstanding of Knuth, wasn't arguing this at all! The full quote is

We should forget about small efficiencies, say about 97% of the time: Premature optimization is the root of all evil

You see that "small efficiencies"? You see that "97% of the time"? Knuth goes on to say that we should care about efficiency and profiling, and that compilers should include performance metrics by default to help people identify hotspots in their code. Oh, also this is in a section of the paper where he talks about a microoptimization getting a 12% speedup and how then laments how software engineers dismiss a 12% speedup as insignificant. This context is completely missed by the people who bring up this quote, context they miss because they didn't read the primary source.

I don't think I'm demanding anything radical here, I'm just asking people to read the paper before they make fun of it. Isn't it better to know what you're making fun of? Won't that lead to better insults? Or maybe you'll find out that hey you learned something and now the world is a richer place for it. Crazy, I know.

Update for all the website readers

This was sent as part of an email newsletter; you can subscribe here. Common topics are software history, formal methods, and theorycrafting. Updates are 1-2x a week with the first (like this) being public and the second being email-only.

If you're reading this on the web, you can subscribe here. Updates are once a week. My main website is here.

My new book, Logic for Programmers, is now in early access! Get it here.