Make peace with estimates and get good at system design along the way
Folks won't stop asking for them, so you might as well get good at estimates
Software developers, myself included, grouse and gripe about producing estimates. At their worst, estimates are framed as a carrot: “we just want a date to plan with!”. Later, and often unexpectedly, the estimate becomes a stick: “you gave this date, what’s the problem?”, despite the ground truth that every estimate is assembled from educated speculation and wild guesses.
Bad news folks (past versions of myself included): it will always be the case, even if language models live up to all the hype, that we can imagine software faster than we can realize it. There is always a trade-off to understand about how much time, effort, and money it takes to turn an idea into working code. Hence: our teammates aren’t going to stop asking for estimates.
The good news is:
Delivering software reduces the demand for estimates. The faster you can deliver software, the less you will need estimates.
Estimating software as a list of tasks scored by effort and variance conveys the most important information required for good-faith use of estimates.
Planning out a project by developing an estimate, rather than just hacking it out, is a way of developing software and enough of a creative effort that you might enjoy it more than sitting in meetings and updating budget spreadsheets.
1. The dynamics of estimates
Software estimates arise out of a supply and demand equilibrium. Your colleagues want to explore an idea: is it possible to build? How soon could we build and get it to customers? These questions come from everyone – product managers, designers, other developers, executives, support folks, and nearly everyone you work with. Responding to this demand, developers supply plans and estimates. Often these are informal (“it would take me a couple of weeks”). Occasionally, these are more formal documentation, possibly elaborate if that’s how your organization rolls.
Estimates (and planning) try to answer tough questions: "Is solving the problem more important than these other priorities?" or “Is the solution worth the time investment, given other possible work?” The back-and-forth often feels like a frustrating game.
On the bright side, delivering software reduces the demand for estimates. Estimates are demand for clarity and context when the ability to answer questions about real customer behavior is scarce. When a team can (quickly) build and deploy software to see how people use tools to solve their problems, the demand for estimates is replaced by demand for further iterations on software solving their problems.
This requires trading a less complete solution, delivered quickly, to acquire more information on how people use the solution or how the solution behaves at real user loads.
That said, you still need estimates. Silver lining: estimates can serve the same purpose as white-boarding, prototypes, and design documents. You’re going to have to think through an idea to turn it into software anyway.
2. Making and using estimates
It’s easy to hand-wave parts of a software plan. Maybe you don’t know how to implement something. Or, there is some kind of apparent symmetry that is too tempting to resist in writing up a plan. But it turns out that there’s more depth of detail or quantity of unknowns in some part of the plan. Estimating makes these weak-spots apparent.
Therefore, a possibly controversial statement: planning and estimates are table stakes at building software. You might as well get good at them!
I like Jacob Kaplan-Moss’ approach to estimating, planning, and iterating on said estimates. It focuses on answering the important questions: How do you assemble the code into shippable software? How much effort (time and people) is required to build the software? How are we likely to get surprised by the depth or complexity of the problem?
To do so, make a list of all the things to build, then score each task on two axes: effort and uncertainty. Effort is given on a relative scale (small, medium, large, jumbo) and an uncertainty multiplier (small, medium, large, jumbo). This means you can account for tasks that are probably small but could blow up (small effort, large variance), tasks that are large but well known (large effort, small variance), and large tasks that are highly variable (large effort, jumbo variance).
With that list in hand, you can share the estimate with your colleagues. Now the collaboration happens based on concrete (but not entirely specific) details. Discuss the options with teammates and stakeholders, identify the “lumps” of exceptional effort or variance in the estimate, and validate they pull their weight in the project. Which tasks are most essential to the problem we’re trying to solve? Can we split large and essential tasks into lower effort or variance tasks? Could we group the tasks into coherent scopes and sequence them to solve customer problems or answer high-risk feasibility questions sooner than later?
This approach addresses the essential risk that drives software developers to stress: accountability for an estimated date that was wildly speculative from the outset and is subject to near-constant scope expansion (and sometimes, contraction) throughout the project.
Remember: estimates and deadlines are different tools for different problems. Don’t copy-paste one to the other, even though they are both date-shaped.
3. The creative act of estimates
If you’re estimating a project, good news: you are now in the midst of designing the software! This may come as a surprise (it did for me!), since you’re likely entering your estimates in a spreadsheet.
For example, tackling the crucial “lumpy” elements of the project: high effort or variance tasks can be researched or prototyped to get a better idea of what the effort or solution domain looks like. That’s not dross spreadsheet action, that’s digging into the issue, cross-referencing with what’s feasible and fast in the software, and listing out options for doing more with less. That’s what we got into software for!
Decomposing the problem into tasks, before you’re calling it a project, is crucial. It feels spectacular to sit down and hack out a task in one glorious session of creative and engineering virtuosity. But more commonly, we get an hour or two into a project and our mental stack is overwhelmed by problems essential to the project, unessential and related to our technology choices, or possibilities to solve the problem differently. Whether it’s for the purpose of estimating, planning, or executing a project, you’re going to have to split up the work. If there’s a quick way to make that part of communicating your plan and a task list for people to review, even better.
Applying critical thinking to a sequence of project steps, whether it’s your plan or someone else’s, reveals problems and possibilities in the plan. The best feedback is early feedback, so why not review your plan and next steps before you get to hacking?
The trick is knowing when to use the creative tools at your disposal. Suppose there’s a difficulty your product or engineering manager has asked you to solve. They’ve given you the context and success conditions for a solution, but left most of the decision-making and problem-solving to you. Do you fire up a whiteboard, spreadsheet, document editor, or programming environment?
In my experience, reaching for a spreadsheet to start a new estimate is best in a few scenarios. If I only have thirty minutes to think holistically about the feasibility of a project, starting a list of tasks and estimating them is useful. If someone is asking me how much effort a specific solution to a customer problem will entail, and I can’t answer off the cuff, starting an estimate helps me think it through. When I know parts of a project are feasible, but other parts intimidate me, making that list of tasks and thinking about how to break up the intimidating parts is helpful. If my current understanding of a project is as ambiguous as ”draw two circles, then draw the rest of the owl”, an estimate helps me structure my thinking.
When is working on an estimate or plan the wrong move? Perhaps not-so-obviously, when you have high confidence, polishing a plan is not the best use of your time. If there are no remaining risks, start building. If the steps and sequence seem clear, start building. When you can see a path through the risks and pitfalls that ends in knowing more about the problem you’re trying to solve, it’s time to build and measure.
Your turn
Practice! You’ll get better, but probably not more accurate or precise. That’s fine; estimates are best used for discovery, not prediction.
Working on estimates builds system design skills. Which is, apparently, a skill that has become increasingly valuable in technical interviews. Use system design insights to inform estimates and estimates to spotlight areas needing deeper design exploration before the first code is written.
I haven’t seen evidence LLMs and copilots will help us develop estimates yet. That said, I think they’re going to make it easier to de-risk the “lumpy”, high variance/risk areas of estimates. Paste a task from your estimate, along with sufficient context on existing code or how you want to solve the problem, and see what the computer comes up with. Adjust your estimate, plan, or research to suit.
Remember, estimates are a discovery process. You’re going to find a lot of ways to come up short with estimates before you feel like you’re good (i.e. not bad) at them. Good luck!