Shipping Software on time and on budget
It’s almost axiomatic that estimating software projects is hard. Stories of I.T. project overruns are commonplace. There’s a whole software crisis devoted to it. The canonical book on the subject calls it the Black Art – one that we’re not good at:
The typical software organization is struggling to avoid estimates that are incorrect by 100% or more1
Nonetheless, it’s quite feasible to estimate even complex projects to within 5-10% accuracy. You can reliably ship software on time, and on budget. In my experience, the ability to do so is what separates senior developers, and mature teams, from their struggling counterparts. It’s a tough business. Ultimately, if you can’t ship on time, it’s difficult to survive. I want to look at that today.
§
The discourse around software often seems to be about swinging between extremes. In estimating, we might imagine two stereotypes, neither of which are very realistic.
On the one hand, we have projects that are fully specified in advance, where tasks have known durations, and will be completed in a determinate order. This is terrifyingly labelled a Waterfall methodology (not even doing justice to the strengths of actual such approaches) and is widely held as the source of all problems in software scheduling.
On the other hand, we have an enlightened Agile Hippy approach. Requirements can’t possibly be known in advance. Our software will be developed iteratively. Deadlines and schedules are artificially imposed by management, and will be rejected. The software will be ready when it’s ready. We will wear flowers in our hair. 🌻
Crass as they are, I’m sure you recognise these pictures.
In between are the rest of us.
A client wants an estimate. They need some idea of what they need to spend, and when their project might be delivered. I might have funding, but, ”Can THIS be built in the time that THAT buys us?” Simply, since ”How long is it going to take?” is directly related to ”How much is it going to cost?”, there’s no escaping the need to come up with an estimate.
But equally there is something in that bogeyman of the waterfall approach. Many times, as I stand here now, I can’t tell you at all how long the idea you’re pitching to me is going to take.
I can eyeball it. Here’s the rule:
Tweak: 6–8 days.
Feature: 6–8 weeks.
App: 6–8 months.
Why not? Well, that’s no better than not saying anything at all. But I think it’s what plenty of people do, and I think it’s largely why estimating software schedules has such a bad name.
Nevertheless, we can do better. Let’s see how.
The Basics
The process for making reliable estimates is well known, and it hasn’t changed essentially ever. Despite it being the Black Art, there is no mystery to it really.
Executive summary: you have break down your project into small tasks, which you can estimate, and you allow for a measure of uncertainty in that.
Allow me to point you to two series of posts that go into depth on that.
The first is by Joel Spolsky, from the year 2000:
Painless Functional Specifications – Part 1: Why Bother? – Joel on Software
Painless Functional Specifications – Part 2: What’s a Spec? – Joel on Software
Painless Functional Specifications – Part 3: But… How? – Joel on Software
Painless Functional Specifications – Part 4: Tips – Joel on Software
The second is from Jacob Kaplan-Moss, from 2021, with a last follow-up in 2024:
Go read those, take your time. There’s lots of wisdom in both of them.
Small details aside, they’re both telling the same story: you can’t estimate your project as a whole.
How long will it take to build an amorphous blob of undefined nothing? No idea!
Rather – not-so spaced repetition – you have to break down your project into small tasks, which you can estimate, and you allow for a measure of uncertainty in that.
For making the estimates themselves, I like to use the Program Evaluation and Review Technique: PERT. Here you calculate three estimates: a pessimistic one (P), an optimistic one (O), and a most likely one (M). You then do a bit of maths:
estimate = (O + 4M + P) / 6
In essence, the most likely outcome is weighted 4 times the other two.
What I like here isn’t the formula. Any time you do more than just summing, or maybe a little uncertainty multiplier, I have a worry that you’re adding fake precision. One that the estimates just don’t live up to.
Rather, what I like about PERT is that it forces you to actually think about whether your estimates are realistic. We said that we were going to break down the work and give good, per-task estimates, but even then, the reality is we don’t.
You ask a software developer (including me, including you) for an estimate, and we’re going to give you the most optimistic. Everything goes well, there are no hidden issues, and no distractions from getting it done.
Ideal time? Yeah, no problem! I have that done in a week.
That’s perfectly normal. We’re optimists. We have to be: would you even turn on a computer otherwise?
So when I say, give me your most likely estimate, you give me the optimistic one. You don’t mean to, but you do.
Asking for the worst-case scenario helps alleviate that. I ask what could go wrong? Well… OK… This could happen, that could happen; this part could take longer; actually, that bit there is under-specified. And so on. By thinking through the worst-case scenario, we suddenly have a lot more context to our supposedly most likely.
At that point, asking for the optimistic estimate helps us look again at our most likely from the other standpoint. Let’s assume everything goes to plan, how long would it take? Oh, that’s only 20 minutes less than our most likely prediction. Do we still think we’re sure about what’s most likely? Of course not, right.
The very process of coming up with a pessimistic and an optimistic estimate makes our most likely estimate a whole grade better. That pays for it. The little math-step after it? Well, OK, it gives us an extra number. The important work has already been done.
Discovery vs Delivery
So, job done? We know how to create estimates. We’re all doing it really well, right?
Well, no.
Smuggled into all of that above is the requirement to break down your project in to individual tasks so that we can estimate those.
Here it’s handy to introduce a concept known as the Cone of Uncertainty.

As your project approaches completion, the variability of your estimates from the actual delivery date decreases.
Just after you finish, you can say with certainty when it’s going to be done. As your boss stands there with a damp napkin, saying How quick can we build this, you’re in the land of McConnell’s ”incorrect by 100% or more”. As you go, you’ll be somewhere in between.
I like to split the cone of uncertainty into two halves. On the left we have the Discovery phase, where we’re not sure what we’re building or how long exactly it might take. On the right we’re into Delivery: we’ve pinned down the job in hand, we know what we’re about, and we’re head-down to ship some software.
Here’s my take. This is what to remember: The reason we’re bad at estimates as an industry isn’t because estimating itself is hard. It’s because we sit down to deliver software when we should be doing discovery.
§
Folks don’t like to hear this. Just as at the beginning there was something in that bogeyman of the Waterfall approach, there’s, equally, something in the caricature of the Agile Hippy.
A spec document handed down from on high, with no room for change, is guaranteed to fail. You can’t eliminate the risk entirely; you can’t narrow the cone of uncertainty to zero before actually finishing the project. When asked for an estimate, it feels like we’re being asked to do exactly that.
I think there are two things going on here.
The first is that creating the estimate is the very job of designing the software. By “breaking it down”, you’re forced into answering questions about what’s actually involved. It’s this that enables you to create the estimate, but it’s a lot of work.
Before serving as a Django Fellow, I spent over a decade freelancing, for clients large and small. Clients would always ask for estimates, and I’d require one to be able to come up with a realistic quote. But it can be a massive amount of work to do up front, without any guarantee of compensation.
No wonder there’s resistance to doing it.
As time went by, I learnt to have a different conversation with clients. I’d show them the cone of uncertainty. You know software projects are risky, right? You want yours to come-in on time and on budget, yes? Well, we’re here in the Discovery phase. If we do this small (fixed) amount of work, we can move towards the Delivery phase here, where your investment is de-risked.
Life was much better once I learnt to phrase it that way.
The second thing, though, is that writing software is a creative process. For any project, there are bits I’ve written before, that I know how they’ll go. But (at least I hope) there are also bits that are new — that I couldn’t possibly say what they were going to look like until I’ve written them. To be honest, that’s a large part of the fun. How on earth can I estimate what I haven’t even yet imagined? Being asked to estimate is — on this view — almost an insult. As if what we do could be reduced to a set of TODO
s.
I enjoy the creative side of programming as much as the next person. Nevertheless, I think there’s a mistake here. There’s a nice Einstein line that, “creativity is intelligence having fun” (my emphasis). It’s not mystical. It’s not genius. It’s not beyond the rational. The creativity we use is what humans in general excel at. It’s special, yes, but not such that normal rules can’t apply to it.
You do the work to break down your project, you start to fill in the details. Things become clearer. “Oh, yes. Over here we’ll need auth. We know that involves…”, and so on.
Of course, there’ll will be bits that can’t be fully specified. Again, that’s the work of making them. But you’ll have a feel for the size of that. Or if not, you’ll know you need to dig in deeper.
There’ll be a degree of variation in your best, worst and most likely estimates. That’s natural.
It’s work, sure. But it’s not mysterious. It’s not difficult to do. And it enables you to pretty accurately say what you can get done, and by when.
What about eyeballing?
Don’t eyeball. Do the work. Do it properly.
Yet sometimes you have to…
“Yeah, yeah, I know you can’t say, but how long, roughly? Come on…!”
When you’re eyeballing it, you’re basically saying, I’m at the wide end of the cone of uncertainty. I need time for the discovery phase, which will get me to the point of delivery, and all of that will fit in the time I’m about to give you.
Clearly, a lot can go wrong here. Discovery can eat your entire budget, and at the end you can find that what you’re trying to do isn’t even possible.
That’s just the nature of the beast. There’s no cure for it. All you can do is make sure to not make delivery commitments until that risk is eliminated, until the requisite discovery has been done.
I’ve long-used Django. Periodically, folks will ask me why I keep using it. Beyond it being mature and solid and all the rest of it, part of the answer is that, given I know it so well, I can eyeball a rough proposal and have a pretty decent idea about its scope.
Django has a certain grain. There’s a way it likes you to do things. You’re going to need a model. You’re going to need views for CRUD. You’re going to need forms, and templates. Is there custom UI? What’s the unique business logic here, if any? How risky does that feel?
Because all of that is well known, it’s feasible to say whether it’s likely that we can take something on in a given window. But it also lets us identify the unknowns: These are the bits that aren’t standard. That’s where the risk is. With that, you can add a caveat to the initial eyeball estimate, with a need to focus on those areas early on, and so de-risk them.
You’re not really eyeballing in this case. You still did the breakdown. You just did it mentally. It always worth writing that down and resending to stakeholders once you’ve got more time.
§
The long and short is that it’s a large part of the work of software design to get to the point where you can commit to a delivery estimate.
If you’re directly employed, get on and do it. See if those superpowers don’t get you a promotion.
If you’re selling that work, make sure you’re selling the discovery and design work, independently of whether you’re also doing the delivery. You might be interested in where the perceived value lies.
If you’re leading a team, or buying that work, make sure you know where you are in your project’s lifecycle before you start asking when it’s going to be done by.
We like to think of our profession as software engineering. If we can’t even do this, we don’t have much claim to that title.
Software Estimation: Demystifying the Black Art, by Steve McConnell ↩