Faltering from ethnography to design by Bentley, Hughes, et al
Why is it so hard to translate research into useful input for product, system or service creation?
This issue we’re looking at two related papers written by a team of researchers from Lancaster University in the UK. Richard Bentley, Tom Rodden, Pete Sawyer and Ian Sommerville were in computing. John Hughes, David Randall and Dan Shapiro were in sociology. One paper has Bentley as the first author and the other has Hughes. For my own convenience and to avoid subjecting you to academic referencing, I’m going to use “Bentley and Hughes” to refer toboth papers.
In 1992, Bentley and Hughes (and their collaborators) were researching how air traffic controllers worked. Controllers use three technologies. First, radar for tracking planes. Second, telephone and radio contact with pilots. And finally, “flight strips” which contain information from the flight database. Each strip represents a flight. Strips get printed out when a plane enters controlled airspace. Controllers work in a team and each one monitors a specific part of the airspace. Each controller has a tray of strips in front of them, representing the planes in their part of the controlled airspace. The work of air traffic control (ATC) involves monitoring the radar for each aeroplane’s dynamic position, using telephone and radio to tell pilots what to do, and using the strips to keep a record of instructions. Controllers talk to many flights so this external and persistent record is crucial. Controllers write on the strips as they issue instructions. As planes move around the airspace, controllers pass strips among the team.
When a controller gives an instruction to a pilot, for example to ascend to flight level 220, s/he marks this on the strip: in this case, with an upwards arrow and the number 220. When the pilot acknowledges the instruction, the controller crosses through the old flight level on the strip. When the new height is attained, the controller marks a check beside it.
It’s possible to do ATC without the radar. It’s not possible without the strips. The point of the research was to understand manual ATC in order to design more automated ATC systems. Automated ATC would produce dynamically updated flight databases which would lead to automated collision warnings. Automated ATC would also support an anticipated increase in air traffic.
The paper notes that that previous tries at creating more automated systems had been rejected by the controllers. Bentley and Hughes speculate that the failed systems didn’t fit with how controllers understood their work:
Previous experiments in automating the UK system have been undertaken primarily from a technical viewpoint. They have not been acceptable to the air traffic controllers and we believe this is because these systems have not effectively supported the ‘working division of labour’ as adopted by the controllers.
Although there’s a lot of teamwork in air traffic control, it’s not a strict part of doing the work.
The air traffic control system is deliberately organised to minimise explicit coordination and cooperation between controllers. […] A task-based systems analysis would therefore fail to reveal the subtle and complex cooperation which is actually going on.
Bentley and Hughes’ ethnography showed that the teamwork was fundamental to doing ATC. The teamwork “saturated” the work. They also showed that the flight strip is a key record and resource for this teamwork. The strip is how the team keeps track of the teamwork.
Essentially the strip is a shared notepad conveying to members of the team what actions have been taken with respect to particular aircraft, who authorised those actions and how these might affect other aircraft in the traffic configuration.
All this anthropological specificity isn’t obviously useful for creating new software systems. Anthropologists know that everything is (at least initially) relevant. Developers have to make decisions about what to build and what to ignore.
Generally, computers should do what computers are good at and people should do what people are good at. Computers are good at sorting lists. Flight strips in front of controllers are a list so it’s sensible to assume that a new system should sort the list itself. Controllers tend to sort their flight strips by time the plane arrived in their airspace. It would be rational to automatically sort incoming strips by time. But here’s the thing:
The ethnographic work suggested that the controller’s action of manually placing a strip in the appropriate position in a bay focussed attention on that strip and to related strips. This was an important safety device as it allowed the early identification of potential problems. Automated positioning is undesirable because these problems may not become apparent until there is an explicit need to use the new strip.
When a controller receives a new strip, they have to look at it and consider all the markings that are on it. Only then do they make a choice about how to put that strip into their tray. The design implication is that a new ATC system shouldn’t automatically sort the strips.
There’s a second layer to this insight. The strip is a record of how the work is done. It’s also why the system is so reliable. (At the time there had been no civil collision because of ATC.) Bentley and Hughes say the ATC system is reliable because the strips require so much work to make up for the deficiencies in the system. The controllers are always checking each other’s work. The collaboration between controllers as they pass strips around, which a purely task-based analysis would engineer out of the system, is the reason why the system works. Replacing that collaboration, in the interests of efficiency or consistency, would make the system worse.
These papers don’t really propose a final design for a new ATC system (which is why research papers are often frustrating — there’s often no answer or outcome in the way that practitioners understand). They do show that point of doing really deep ethnographic work in systems design is to find things that might be missed because of the reasonable assumptions that need to be made when creating software.
What it means in 2020
User Experience work often takes place in a limited number of simple well-understood situations. There are strong patterns for a reasonable shopping cart or a credit card payment flow. We know how to let people check their account balance or change their password.
But these are transactions. I don’t think UX really knows a lot about work.
Papers like these by Bentley and Hughes show us the complexity of work. Most of us won’t ever try to update ATC. Only some of us might work on safety critical systems. But we’ll all probably end up working on systems that support work. If we bring our assumptions about enabling transactions to supporting work, there’s a possibility that we’ll create systems that aren’t useable and useful. Or worse, we’ll create systems that are unsafe.
Transaction systems are easy to represent in the abstract — that’s why we get so much mileage out of journey maps. But as Bentley and Hughes say:
characteristics of systems do not exist in the abstract, as absolute features of their design, but are emergent properties of systems in use
We need to be sure that we can represent and account for these emergent properties of work. In situations of change, when we’re trying to find new ways to support situations that are potentially very complex, I’m not completely sure that our familiar tools are up to the task.
Serious things
Formal references
Bentley, R., Hughes, J. A., Randall, D., Rodden, T., Sawyer, P., Shapiro, D., & Sommerville, I. (1992, December). Ethnographically-informed systems design for air traffic control. In Proceedings of the 1992 ACM conference on Computer-supported cooperative work (pp. 123-129).
Hughes, J. A., Randall, D., & Shapiro, D. (1992, December). Faltering from ethnography to design. In Proceedings of the 1992 ACM conference on Computer-supported cooperative work (pp. 115-122).
Canonical URL
Bentley et al.: https://dl.acm.org/doi/abs/10.1145/143457.143470
Hughes et al.: https://dl.acm.org/doi/abs/10.1145/143457.143469
Publicly available source
Annoyingly, both of these seem hard to get a hold of. I suggest googling the titles and seeing what you get back.