March 6, 2024

How to argue for something without any scientific evidence

It's all about the show

Last week I got this interesting question:

I want to write a book about automated testing. Much of the book would be me explaining the best practices I’ve learned. I know these are good practices; I’ve seen them work over and over again. But have no [scientific] data at all to support my conclusions.

My question: lacking such data, how can I talk about something important like testing in an authoritative way? — Jake

This is near and dear to my heart. On one hand, I think empirical software engineering (ESE) is really important and have given several talks on it. On the other, I think ESE's often a trainwreck and find most of the research flawed. And my career is based on formal specification, which hasn't been studied at all.

Thing is, software still needs to get built regardless of how much we know about it. So how do you get people to believe you when there's no science backing you up?

I've written before that you need two things to be considered trustworthy:

1. Trustworthy expertise: your judgement is correct.
2. Trustworthy opinions: you aren't deceptive about your judgement.

Empirical evidence is a shortcut to both of these: we tend to think the scientific process is perfectly objective (it's not) and consider it unbiased (we shouldn't). At the very least, it's on average more objective and less biased than most other forms of gathering information. Without it, we need to work harder to earn people's trust.

Trustworthiness isn't the same as authority, but they're closely linked. You can't be an authority without being trustworthy.

ACOUP plug

One of my blogs is A Collection of Unmitigated Pedantry, about the intersection of history and pop culture. I've written more about it here. I want to call out one section of this essay:¹

Soldiers going into battle often look to their leaders to perform leadership. After all, the infantryman on the line has little sense if the battleplan is good or not ..., but he’s likely to be reassured if his leaders look like how he imagines good leadership to look. … seeing their king performing that role with excellence could be confidence building – “I may not understand the plan, but we have a good general and thus a good plan.”

Authority is performative. So if you want to talk about something like an authority, you need to perform being an authority. Do things that people expect an authority is able to do. I can't tell you the best way to perform authority, but I can tell you how I try perform it, and you can judge if that'd work for you too.

How I try to perform authority

1. Be up-front about my biases.
2. Demonstrate deep familiarity with the field.
3. Talk about the benefits and drawbacks.
4. Talk about alternatives in a deep, honest way. Say what they do worse, but also what they do better, and in what circumstances.
5. Discuss variants, that are still $THING but not how I do $THING.
6. Discuss common arguments against, rejecting some as invalid, acknowledging some as valid, why it's good despite those valid arguments.
7. Argue why, on the whole, I still take the position I do.

Let's try applying these principles to a testing topic like Test-Driven Development. I've written about my complicated feelings before but not written something trying to convince people to use it.² If I had to, this would be my first-draft outline:

1. Define test-driven development, both the general principle (write tests before code) and the specific form (strict red-green-refactor cycles).
2. Give a persuasive example of a case where TDD performs best: adding regression tests to fix a bug in code.
3. A more involved example of "greenfield development" with TDD. This would probably look like writing two modules, one with TDD and one where I write all of the tests at the end, and compare the two. Show where the two approaches lead to the same tests and where they lead to different tests.
4. Claimed merits of TDD: leads to better code, low coupling/high cohesion, "design pressure", etc. Mention which ones I believe, which ones I don't believe, which I think require more nuance, etc.
5. Common arguments against TDD: leads to "TDD code", slows you down, plenty of great software is written without it, etc. Respond to these in ways that show I respect the people making these claims even if I disagree with them. 1. Go deeper into the class of "types/integration tests/formal methods are better" arguments. Use this to talk about defense-in-depth.
6. Alternatives: writing tests at the end, having a separate QA team, cleanroom, not testing at all, etc. Show cases where TDD is better than these and cases where it's worth. Argue that the former are more common. 1. Talk about where I personally don't do TDD.
7. Common variants still in the TDD umbrella: behavior-driven development, acceptance-test driven development, test && commit || revert
8. Further reading. I would not recommend the original book because I haven't read it. That might undermine my expertise but it supports my honesty. If I really wanted to come across as an authority, I'd read that book first before writing my spiel.

(This looks like an awful lot of work, but that's mostly because TDD is such a huge and controversial topic, so I have to do a lot of work to perform authority. If I focused on a specific thing, like "mocks or not" or "should you test private methods", it'd all be a lot easier.)

The goal here is to "show my work" and demonstrate that 1) I'm well-versed in TDD, 2) I understand the flaws of TDD, 3) I genuinely value TDD anyway, 4) that my opinions are motivated by deeper engineering principles. Even if someone disagrees with my opinions, they still gain knowledge and don't leave thinking that I'm trying to hoodwink them.

There are other ways to perform authority

One thing I do a lot is showcase advanced knowledge on something that most practitioners will never need, but can appreciate the expertise behind it. Like it one's thing to show I know the basics of TLA+, another thing to show I know about hyperproperties and refinement. I don't know what the equivalent for TDD would be, maybe testing statistical properties?

In writing this I notice that this is specifically about performing trustworthy expertise. What would performing trustworthy opinions look like?

If someone comes in demanding empirical evidence

First ask yourself: are you making the kinds of radical claims that require lots of scientific backing? Not "TDD/Formal methods is good", but "TDD is equivalent to hand-washing" or "if you don't use formal methods you're unprofessional." If so, then you're not performing authority well and people are picking up on that. Dial it down.

Second: try to find out if they genuinely care about empirical software engineering. Are they interrogating their own practices under the same lens? If not, then they're just using ESE as a cheap gotcha and wouldn't change their mind even if you have studies.

Otherwise, talk about the paucity of ESE knowledge in general, raise whatever few papers you do know about, say they're suggestive and limited and more research is needed, etc etc. Show that you've at least thought about the scientific side of things.

---

Consulting stuff

My current gigs are winding down so I'm looking for new consulting work in formal methods (TLA+ and Alloy). Details here.

If you're reading this on the web, you can subscribe here. Updates are once a week. My main website is here.

My new book, Logic for Programmers, is now in early access! Get it here.