Failed software projects seem to draw a certain amount of schadenfreude in our industry. It’s because we’ve all been there. Our reputation for projects that fail is so strong; it seems to elicit the question: “What’s wrong with those software developers?”
According to Tom DeMarco (co-author of the seminal book Peopleware), there is a reason for many software failures. In his experience as a technical expert during software litigation cases, failure happens for one reason:
”In poring over nearly a billion dollars worth of software litigation, I came across no failures due to poor quality, slow response time, or unworkable human interface; all the failures were about lateness.” (emphasis mine, source)
Failure happens not because of technical issues but because of lateness. Business leaders have three options when the time has come and gone, and the software is incomplete:
- They can roll out an unfinished (or vaporware) product.
- They can finish the project past the deadline.
- They can cancel the project.
In DeMarco’s experience, the projects were finished beyond the deadline or flat out canceled. He goes on to say:
”All projects that finish late have this one thing in common: they started late.”
This is fundamentally a business problem, not an engineering one.
There are two flavors of late project problems in play here. The first is the demand to meet an arbitrary date; the second is understanding the cost and value of software engineering work.
Delivery deadlines are not, in general, set by the engineering department. In some cases, competitors offer a feature that catches a business unaware, and the business is playing catch-up in the marketplace. Sometimes an arbitrary date is selected based on a trade show. One example I heard was an accountant canceling the license of the solution the software project was intending to replace.
“One major barrier for companies […] is the heavy bias toward applying MBA manufacturing business-based techniques to software development. In the widget replication business, delivering by date certain is highly valued.” [source](https://iism.org/article/how-many-of-you-know-deep-down-that-the-team-is-working-on-something-that-no-customer-wants-54)
Manufactured parts are produced reliably with little or no variation between them. The piece count of a machine is well known, and date-based estimates are easily calculated.
Software is not a widget, and programmers are not stamping machines. Each piece of software that is not unique must be automated, and every bit of software that was not automated is unique. Chris Moschini describes it like this:
”In programming, if a developer can break down everything in a project into a set of tasks they've done many times before, they're doing a bad job. Everything in programming can be automated, so if you've done it enough before that you have that strong an understanding of it, you should be automating it by now.”
When repetitive tasks are automated, only the complex, unclear, and unknown work remains. This work is where engineering provides its value and where the mismatch between a project goal and a project estimate occurs. According to Steve McConnell in Software Estimation, a mismatch between a project deadline and the estimate is an “incredibly useful, incredibly rare, early-in-the-project risk information.” Unfortunately, it isn’t used:
The most typical interaction is for the business sponsor and the project leadership to negotiate the estimate and for the project team eventually to be pressured into committing to try to achieve the [schedule]”.
While the engineering team’s estimate already shows the project has started too late, leadership is unwilling to look closely and see the project will cost more than they are willing to pay. DeMarco again:
“If a project offered a value of 10 times its estimated cost, no one would care if the actual cost to get it done were double the estimate. On the other hand, if expected value were only 10 percent greater than expected cost, lateness would be a disaster. Yes it would be a disaster, but instead of obsessing over "What's the matter with those software folks who didn't deliver on the schedule we gave them?" we need to ask instead, "Why did we ever kick off a project with such marginal expected value?"