Software Engineering Research
It’s a Sunday afternoon, and I’m in the lab. I hear the low hum of the fridge, and the soft, almost inaudible “whoosh” of the HVAC system. Outside, the week-long embrace of fog has left Vancouver, the smell of mist has been displaced by petrichor, and the rain falls, droplets dancing in the wind. The deafening silence is punctuated by the taps I’m making on my keyboard as I write this newsletter, and, in a way, it’s almost meditative.
On Friday afternoon, one of my lab members asked “what’s research all about?”
I don’t think this is a question to which I have an answer that I’m fully happy about, myself. If you start counting from September of last year, it’s been a little over 4 months since I started doing “research,” but I don’t think I’ve gained enough experience to really capture all of what research is about in a short sentence. Additionally, I don’t presume to know what happens in research in other areas, so my experience is limited to empirical and experimental software engineering. I thought I’d use this newsletter to dive a little deeper into what research is all about (to me).
Initially, this is what I thought research looked like:
The Problem -> The Question -> Experiments -> The Solution -> New Knowledge
To the reader well-versed in how research really works, this is probably a laughable diagram. Research is rarely so clean-cut, well-defined, or linear in how it happens “in the field.” Over time, this is what my mental model began to evolve into:
loop
--------------
| |
The Problem -> The Question -> Experiments -> The Solution -> New Knowledge (maybe, rarely, probably not)
| | |
---------------- dead end
loop
I’ve omitted some loops because I’m tired of making ASCII art look nice, but you get the point. To draw a visual image, there should also be loops between the “Experiment” and “The Solution.” People may give the stages of this pipeline or diagram different names. However, I think I’d be hard-pressed to find someone who disagrees with my assessment that as researchers (if I’ve even earned the right to call myself that, yet), we keep finding ourselves revisiting and repeating some stages of research.
The Problem
Something that Gail asked me to do when I went on my internship last summer (and even before I started grad school in 2020), was to think about “the things that keep pissing you off.” This normally isn’t a problem for me, since I get pissed off at a number of things on a daily basis, but all jokes aside, I think this is incredibly important. I think it’s common for academics to tackle problems that aren’t necessarily important to practitioners but are very nicely formulated research problems. Of course, there are pros and cons to this, and I don’t assert that one is necessarily better than the other 1. In my personal experience as a researcher, especially in empirical and experimental SE, I find myself gravitating toward the school of thought that SE research needs to be informed by practitioners, and vice versa.
Anyway, once you identify a problem that keeps popping up, you can start to think about what comes next.
The Question
It’s not enough to have a problem. As humans, we deal an awful lot in terms of abstractions. They make it easier for us to reason about things that are otherwise difficult to completely “hold in one’s head.” It’s my view that abstracting a problem into a succinct question enables us to more easily communicate and express the problem to other researchers, which is eventually how knowledge is disseminated.
People employ abstraction in different ways. You know the phenomenon where you read paper X, and a few months later you read paper Y that seems to try to answer the same question in a slightly different way? I think that’s a consequence of how people formulate the same problem, but abstract it into different forms.
Experiments
This is probably the stage in the “pipeline” which I often find the most difficult but also most insightful. In empirical SE, we often find ourselves building prototypes and tools and asking users what they think. This “human-in-the-loop” form of scientific reasoning obviously opens things up to inherently human factors such as bias, and you aren’t going to get the same kind of purity you get get from building things up from first principles (like in Math). That said, I think the human aspect in SE research is incredibly important. For how much people talk about how human labour will be replaced (yes, even in software engineering), I think we’re still incredibly far off from that future.
We take what people say about a tool, and we use it to inform the continued design of the tool.
The Solution
Once we reach a certain point of saturation with our experiments, we can finally start thinking about the conclusions we can draw from our data. This is the point where we really have to look hard at what the data says, even if we don’t like what it’s saying. There’s been some work recently for conferences to publish what are described as “negative” results 2. I think it’s sometimes more important to know what not to do.
New Knowledge
This is pretty self-explanatory. Something I missed out in the diagrams above is that I think there’s also a stage that can be here instead of “New Knowledge,” which is “Reinforcement of Knowledge.” It’s often useful to have multiple pieces of evidence that serve to reinforce something that may have already been known before.
So that’s it. That’s what my mental model of research currently looks like. No doubt it will change over the course of the time I have left here, but I think it’s much better than the simplified model I had before.
In terms of non-research related stuff, I started playing Pokemon Legends: Arceus. It’s vastly different from any of the Pokemon games that I’ve previously played, and I have to say that I’m enjoying it a lot. There are some parts of the game that I don’t enjoy as much (Research Tasks for the Professor), because I already get too much of that in real life.
That’s it for now, this was a pretty different newsletter. Have a nice week!