The unique thing about estimates in software engineering is that if you do it right, projects should be impossible to estimate!
Tasks that are easiest to estimate are tasks that are predictable, and repetitive. If I ask you how long it'll take to add a new database field, and you've added a new database field 100s of times in the past and each time they take 1 day, your estimate for it is going to be very spot-on.
But in the software world, predictable and repetitive tasks are also the kinds of tasks that are most easily automated, which means the time it takes to perform those tasks should asymptotically approach 0.
But if the predictable tasks take 0 time, how long a project takes will be dominated by the novel, unpredictable parts.
That's why software estimates are very hard to do.
And I'd add that the need for them is a sign they aren't worth doing.
As you say, worthwhile software is usually novel. And to justify our expense, it needs to be valuable. So to decide whether a project is worth doing, we're looking at some sort of estimate of return on investment.
That estimate will also, at least implicitly, have a range. That range is determined by both the I and the R. If you don't have a precise estimate of return, making your estimate of investment more precise doesn't help anything. And I've never seen an estimate of return both precise and accurate; business is even less certain than software.
In my opinion, effort put into careful estimates is almost always better put into early, iterative delivery and product management that maximizes the information gained. Shipping early and often buys much clearer information on both I and R than you can ever get in a conference room.
Of course all of this only matters if running an effective business is more important than managerial soap opera and office politics. Those often require estimates in much the same way they're required from Star Trek's engineers: so the people with main character syndrome have something to dramatically ignore or override to prove their dominance over the NPCs and material reality.
Another type of worthwhile software is business CRUD. And unfortunately in my experience even for these mundane types of tasks time estimates are typically misjudged - for a multitude of reasons.
The typical reason is that developers in large teams often don't know all the places where a field may be used - and their IDE could very well miss some. So automated testing or QA finds an issue, and then the unaccounted-for, time-consuming bugtracking begins.
CRUD: Create, Read, Update, Delete. Basically just inputting and displaying data from a database.
> As you say, worthwhile software is usually novel.
This is an interesting assumption. I’d argue that the overwhelming majority of software is the most boring LoB CRUD apps you can imagine, and not novel at all. Yet, people need to estimate the tasks on these projects as well.
And starting in the late 1970s, there were tools available to simplify building LoB CRUD apps. [1] That has continued with things like Rails and Salesforce and no-code tooling.
If something is truly boring in software, it gets turned into a library or a tool for non-programmers to use. Our value is always driven by the novelty of the need.
And no, people don't need to estimate the tasks. My dad did LoB apps in the 1970s to the 1990s. E.g., order entry and shop floor management systems for office furniture factories. His approach was to get something basic working, see how it worked for the users, and then iteratively improve things until they'd created enough business advantage and/or cost savings to move on. Exploratory, iterative work like that can at best be done with broad ballpark estimates.
I grant that people want estimates. But that is usually about managerial fear of waste and/or need for control. But I think there are better ways to solve those problems.
All of that is besides the point. People need to estimate their tasks if their managers want them to, and no amount of philosophical navel-gazing will change that.
I want to be clear that I am being entirely practical here. This is not navel-gazing. I am describing something that works. That has worked for me and others for decades.
And yes, if you are in an environment where people with power want things, you have to take that into account.
But no, we don't have to just blindly do what people with power ask. The difference between being a professional an a minion is that professionals know much more about the work and how to do it than the people paying them. Personally, I think we are professionals, which gives us a variety of responsibilities not just to the person with the money, but to the profession and society at large.
Does that mean I never have given estimates? Not at all. But it does mean that if somebody asks me to do things in a way I think suboptimal, I'm at least going to say, "Hey, there's a better way to satisfy your goals here." And then I'm going to help them learn enough about the better way that they're at least willing to try it.
Because "the manager says so" or because "estimates actually add some value"?
I think it's important that our "work culture" allows us to critique why we do certain tasks. When push comes to shove, I guess a manager can say: "Because I say so!", but I also hope those kind of managers are few and far between.
Both, kind of. The demand to have at least a rough estimate when something will be available is justified IMHO—other departments obviously need to maintain their own timelines and schedule work that depends on output from engineering.
Also, I wholeheartedly agree that we do need to question the work culture we follow and the measures we make, and that managers with control issues shouldn't dictate them.
On the other hand, the point I was getting to is that a critique of estimation that amounts to "the work I do is so bespoke and unique and novel and important that I can't be bothered to estimate how long it'll take", is just… ignorant. Most software engineers are not lone wolf 10X wizards without any accountability, have managers and other departments to report to, and thus are not eligible to make that point.
> the work I do is so bespoke and unique and novel and important that I can't be bothered to estimate how long it'll take
This absolutely can be the case some of the time though. I've never pressed back on estimates of standard work but it can be a real bastard to have to work within the "process" when you are slaying a truly novel beast. Having some jackass pestering you for updates on how long it takes to climb the beanstalk and find the golden harp is just too much.
This is a gross and misleading caricature of what I'm saying. I prefer this approach precisely because it increases accountability. If you'd like to learn what I'm actually suggesting, I'm happy to answer questions. Or you can read many of the things that have been written by other people on the topic.
Sure you can move the goalposts here, but OP clearly meant to say software tasks cannot be estimated because people only work on novel problems, since everything else is "not worth doing" (what a massively privileged thing to say by the way).
Just because something hasn't been done the exact same way you're doing, that doesn't mean you can't apply a generic solution. I have never changed a tyre on an SUV before, yet I do know how to do so based on my previous experience with a sedan. The same applies to a car mechanic; even if I bring a brand new car to the workshop they have never seen before, I can and should expect them to be able to (at least roughly) estimate how long a tyre change is going to take.
< “Those often require estimates in much the same way they're required from Star Trek's engineers: so the people with main character syndrome have something to dramatically ignore or override to prove their dominance over the NPCs and material reality.”
Thanks. It was hard won. I spent maybe a decade naively thinking that if we just made software methods that worked in service of stated business goals and values, they'd get adopted and we'd all live happily ever after.
It took me a long time to come to grips with the POSIWID [1] version of the purpose of planning and estimates. One of the things that really blew my mind is Mary Poppendieck's story about how they built the Empire State Building on time and under budget even though they didn't have it fully designed when they started. [2] Different, more effective approaches are not only possible, they exist. But they can no longer win out, and I think it's because of the rise of managerialism, the current dominant ideology and culture of big business. [3]
Thanks for the links. To the limit of my influence I try to protect my team from distractions, be fluid about methodology (constant agile churn can be depressing), limit the toxicity of pull requests, and to spend as much time with them as I can. A happy team is a productive team. Oh and I try not to work with leaders who obsess over Gantt charts. To me estimates are more about trust and respect rather than metrics and velocity. It has to be the right kind of company though.
Further: In The Next Generation, when Scotty shows up, he mentions to Geordi he anyways padded his estimates because he knew Kirk would do things like that.
More specific methodologies that say they are agile may use concepts like estimates (story points or time or whatever), but even with Scrum I've never run into a Scrum-imposed "deadline". In Scrum the sprint ends, yes, but sprints often end without hitting all the sprint goals and that, in conjunction with whatever you were able to deliver, just informs your backlog for the next sprint.
Real "hard" deadlines are usually imposed by the business stakeholders. But with agile methods the thing they try to do most of all isn't manage deadlines, but maximize pace at which you can understand and solve a relevant business problem. That can often be just as well done by iteratively shipping and adjusting at high velocity, but without a lot of time spent on estimates or calendar management.
Yes, people keep linking to the agile manifesto as if it's some sort of amulet protecting software developers from any sort of accountability or responsibility for their work product in a professional setting.
It seems like you acknowledge some amount of estimating is needed and I agree that there is an overemphasis on estimation in many places, but I'll ask you the same thing I asked others, which is:
How do you do either of the following without spending any time at all on estimates?
"Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale."
"At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly."
I addressed the rest elsewhere, but done well a lack of estimates makes people more accountable. If I am shipping something every week (or as is common for my teams, more often), stakeholders can directly evaluate progress. There's no snowing them with paperwork and claims of progress against semi-fictional documents. They see what they see, they try it out, they watch people use it.
The reality of use is what we in software are ultimately accountable to, and that's what I am suggesting people optimize for. Closing that feedback loop early and often builds stakeholder trust such that they stop asking for elaborate fantasy plans and estimates.
You replied to me in like 10 different places, nearly all of which are in responses to posts that weren't directed at you, so I'm trying not to fragment this discussion too much.
I will ask this here: If you are shipping code to production on a weekly basis, is that not a schedule, also known as a deadline for delivery?
If you expect to ship code to production every week, how do you know whether there will be something to ship without doing any estimation of the effort and time required?
It is not a schedule, it's a standard. One I normally try to exceed. We ship when things are ready, which for my current team is ~2-3x/week, but in the past I've had teams that were faster.
We know that there will be things to ship because we try to break the work down into small units of deliverable value by focusing on seeking the highest-value things to do. Large requests are typically composed of a bunch of things of varying value, so we split them until we find things that advance the needs of the user, the customer, or the business. One that's often not intuitive to people is the business need for getting information about what's truly valuable to some part of the audience. So we'll often ship a small thing for a particular audience and see how they react and what actually gets used. (You can save an amazing amount of time by not building the things nobody would end up using.)
Sometimes we can't figure out how to break down something smaller enough that we have something to release right away. Or sometimes a chunk of work surprises us and it drags out. We avoid that, because compared to working on smaller things, it's much less comfortable for both developers and business stakeholders. But if it happens, it happens. We try to learn from it for the next time.
Regarding deadlines, we sometimes have them. Broadly, efforts break down into two categories: driven by date or driven by need. For the former, releasing early and often means we adjust scope to hit the date. For the latter, scope is primary and they get stuff when they get it. Either way because the business side sees steady improvement and has fine-grained control over what gets shipped, they feel in control.
This can be a learning experience for business stakeholders used to waterfall-ish, plan-driven approaches. But I have never once had somebody successfully work this way and want to go back. I have, however, had some product managers get thrown back into document-driven development and tell me how much they missed working like we did.
No, shipping code to production on a weekly basis is not a deadline. A deadline is a time by which a task must be completed. A task is something like "fix bug 3831" or "allow users to log in with OAuth2". "Ship code" is not, in any useful sense, a task.
Such "timeboxed iterations" can indeed result in "shipping" a null update. Unless you have a time-consuming QA gate to pass, that's not very likely, especially on a team containing several people, but it can happen. You don't know that you will have "something" to ship.
Typically we try to break changes down into shippable tasks that can be done in under a day, so the expected number of tasks completed by a four-programmer team in a week is on the order of 30, or 15 if you're pairing. For this to fall all the way to 0, everybody has to be spending their time on things that could not be thus broken down. It's pretty unlikely to happen by chance. But sometimes a difficult bug or two really is the thing to focus on.
In XP, estimates are used for prioritizing which tasks to work on and which tasks to break down into smaller tasks. The "product owner" is supposed to choose the tasks that have the most user value for the estimated cost. But those estimates aren't commitments in any sense; they're guesses. Sometimes tasks take more time than estimated; other times, they take less. This is the reason for the shift to estimating in "story points": to prevent the estimates from being interpreted as referring to a period of time.
If someone in your organization is interpreting estimates as commitments, this can have a corrosive effect on the accuracy of those estimates, because estimators respond by padding their estimates, in inconsistent ways. Often this destroys the usefulness of the whole iteration planning process, because the project owner no longer knows which tasks are the easiest ones and thus worth doing even if the benefit is marginal. Organizations can recover from this pathology in a variety of ways, but often they don't. Eliminating estimation is one response that sometimes works.
Yes, this sounds very familiar to me. I started with dates and went to story points used to estimate dates. Then as we turned up the release cadence, we eventually dropped estimating altogether, even in points, because it wasn't really helping anything.
That doesn't mean we refuse to solve the problems that estimates are used to solve. E.g., "Can we have something by the big conference," or "Can we keep our customers happy." We just solve them other ways.
And totally agreed about the corrosive effect of treating estimates as commitments. It's such a negative-sum game, but people play it all the time.
I already said this to your fellow interlocutor who is also responding to nearly every comment of mine with the same thought process, but I'll say it here as well in different terms:
The product owners, customers, salespeople, supervisors, peers, etc. you interact with as part of the software development process on any project outside of a personal hobby don't care about your semantic games.
If functionality is needed in an application, and they ask you to implement it, and you agree, there is no real-world scenario where they just say "Cool, I'll sit idly by while you work at this until you declare it ready, and then and only then will I let anyone else know about it or take any action on its supposed existence," and repeat that for every piece of functionality you implement in perpetuity.
And if you keep failing to deliver required functionality over time, no one is going to accept your arguments that: "Oh sorry, our weekly deliveries to production aren't a deadline, it's a timeboxed iteration",
"Oh that estimate wasn't a commitment to do anything, we work on our own schedule", and so on.
Yes, the relationship between developers and "other stakeholders" can turn toxic, but in most organizations the developers don't have much power, probably due to repeated attempts to play the games you've laid out above. The way to combat that is to be reliable and professional so your team has the authority to stand their ground on the difficulty of a given task, not effectively refuse to participate in what is a completely reasonable conversation about the relationship between your work and the objectives of the organization.
I have to keep responding to your posts because you keep grossly mischaracterizing what I'm saying. I have great relationships with my stakeholders. Just this we we delivered something greatly desired by the organization. We are being reliable and professional, just in a way that apparently confuses and frightens you and others. Indeed, I'd claim I'm being more professional that people who give nonsense estimates just because bosses ask for them.
> How do you do either of the following without spending any time at all on estimates?
> "Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale."
This is 'just' bum-standard continuous delivery (which is where most organizations should be heading). You pull the next todo from the backlog, start working on it. If it takes more than a day to commit something, you split the task into something smaller.
You don't need to estimate ahead of time at all as long as the task is small enough, all you need is to be able to put the near-term backlog of work into a good priority order of business value.
If the high-value task was small it doesn't prevent you from doing more work, because the next unit of work to do is the same either way (the next item on the backlog).
If the high-value task was too big, it can cause you to take a pause to reflect on whether you scoped the task properly and if it is still high-value, but an estimate wouldn't have saved you from it because if you'd truly understood the work ahead of time you wouldn't be pausing to reflect. An estimate, had you performed it, would not have changed the priority.
But this Kanban-style process can be performed without estimates at all, and organizations that work to setup an appropriate context for this will find that they get faster delivery of value than trying to shoehorn delivery into prior estimates instead. But there are people who work faster with the fire of a deadline under their tail so I can't say it's unilaterally better.
> "At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly."
If it's hard to do the work as a team, you should be able to tell it was hard retrospectively, with or without having done an estimate ahead of time.
You might say that failing to hit your prior schedule estimates would be a good topic to discuss at a retrospective session, but I would tell you that this is a self-licking ice cream cone. If your customers are happy despite missing internal schedule estimates you're in a good spot, and if your customers are unhappy even though you're "hitting schedule projections" you're in a bad spot.
There's a lot more productive discussions to be done when the team reflects on how things are going, and they typically relate to identifying and addressing obstacles to the continuous flow of value from the product team to the end users.
But that is rarely how it works. In the dozens of different projects across ten or twelve companies I’ve had insight into, “doing Agile” is analogous with “we have a scrum master, hold stand ups, and schedule iterations” while the simple reality is “Agilefall.”
I have had many successful projects where we spent approximately zero time on estimates. The fact that a successful approach is culturally seen as illegitimate to even talk about is a great example of why I wrote that last paragraph.
On the contrary, constraints often mean you don't need formal estimates. (I'll come back to prioritization in a sec.)
Startups are a great example. When you raise your first chunk of money, the size of that isn't really driven by a carefully considered, detailed plan with engineering hours estimated per task. What you get is basically determined what's currently fashionable among angels and small-end VCs, plus who's doing your fundraising. (If you're Jeffery Katzenberg and Meg Whitman, you can raise $1bn. [1] https://en.wikipedia.org/wiki/Quibi But the rest of us have to make do with what we can get.)
So at that point you have a strong constraint (whatever you raised) and some relatively clear goal. As I said, cost isn't nearly as relevant as ROI, and nobody can put real numbers on the R in a startup. At that point you have two choices.
One is just to build to whatever the CEO (or some set of HiPPOs wants). Then you launch and find out whether or not you're fucked. The other is to take something akin to the Lean Startup approach, where you iteratively chase your goal, testing product and marketing hypotheses by shipping early and often.
In that later context, are people making intuitive ROI judgments? Absolutely. Every thing you try has people doing what you could casually call estimating. But does that require an estimation practice, where engineers carefully examine the work and produce numbers? Not at all. Again, I've done it many times. Especially in a startup context, the effort required for estimation is much better put into maximizing learning per unit of spending.
And how do you do that? Relentless prioritization. I was once part of a team that was so good at it that they launched with no login system. Initial users just typed their names in text fields. They wanted proper auth, and eventually they built it, but for demonstrating traction and raising money there were higher priorities. It worked out for them; they built up to have millions of users and were eventually acquired for tens of millions. On very little investor money.
Being great at prioritization makes estimation way less necessary. The units of work get small enough that the law of large numbers is on your side. And the amount of learning gained from the things released change both the R and I numbers frequently enough that formal estimates don't have a long shelf life.
So I get what you're saying in theory, but I'm telling you in practice it's different.
If you're saying there's some sort of wisps-and-moonbeams notion of estimation in everything that we do, sure. I'm not going to argue with that. One world, brah.
What I am talking about here, though, is a practice of software estimation where programmers produce detailed numbers on the amount of time requested work will take. Which is certainly the common meaning of estimating around here, and also what the original article is about.
I'm not upset. But as far as I can tell you've wandered off into some sort of philosophical space when I'm speaking of practicalities, without apparent recognition of the transition. I thought that was a bit ridiculous, something from the "wow dude have you ever really looked at your hand" school of conversation, so I made a joke. Apparently the joke didn't land for you. Alas.
Research is subject to constraints of money, time, and resources, but is not normally estimated in the sense that software industry people would use the term.
Yes, yet estimates are still made. The author of the article didn't use some highly formal definition of estimation, didn't imply one, and seems to be focused on devops (not software development) as a practitioner.
Estimates are difficult, and in unhealthy environments are weaponized against developers. That doesn't mean they're unnecessary or impossible.
I think that the replies I am getting are demonstrating why developers have estimates used against them - people forget that they are estimates, and they also forget that when new information comes to hand that invalidates that estimate a completely new one may need to be created to take into account the new data.
If developers (or anyone giving estimates) discovers that the initial estimate was based on faulty information then they need to push that information back to whomever they are reporting it to (Team Lead, Product Owner, Manager, customer, angel investor...). The receiver of that information then needs to decide on how to react according to the changes.
Yes, agile is a reaction to spreadsheet driven development and some very dumb ways of tracking progress towards completion and managing work in general.
In my experience, people don't forget they're estimates, they just want to force developers to meet whatever they agreed to that's most convenient for management.
If you want to fight back against that, my experience has been that giving terrible estimates or refusing to give them at all will not result in more autonomy or authority.
> If you want to fight back against that, my experience has been that giving terrible estimates or refusing to give them at all will not result in more autonomy or authority.
In my experience giving terrible estimates or refusing to give them at all is the least bad course of action. It wastes less of your time than any realistic alternative, it does no noticeable damage to the business or your own position, and the people who want to paint you as just trying to avoid accountability were going to find a way to do that anyway.
Research is estimated, sometimes those estimates are hilariously bad (Computer vision is easy, a summer research for a student should be enough), but more often than not it's "We expect that this research will take someone doing a Ph.D approximately 3 - 5 years to do"
The entire premise of a project is "Look at this, with the intent to find X, and, if it's not possible, break it down so that we can create more projects to work toward that goal" which is an estimate, or a breakdown into sub projects that also come with estimates.
I, and others, don't agree with the blanket statement that "no estimates" is not a legitimate argument in any scenario. Can you expand on why you think there isn't a single case where estimates don't add value? Similarly, is there anything specifically in that post's claims that you think was incorrect, leading to their false conclusion?
Okay, a scenario where you're building a hobby project alone and you don't care if or when it gets finished would be one where estimates aren't needed.
There is no scenario where it's appropriate or necessary when developing software professionally or even as a side project where others are expecting you to complete work at some point.
One of the many misconceptions in the original comment in this thread is that "worthwhile software is usually novel", which is not the case without a very specific and novel definition of worthwhile that I don't believe was intended.
If software isn't novel, that means some other, existing software does the same thing just as well in the same way on the same platform. So, unless it's a hobby project you're building alone, why don't you just use the existing software?
I think that writing software that isn't novel fails to be worthwhile by a perfectly ordinary, mainstream definition of "worthwhile".
So you would consider a CRUD app with some basic business rules to be novel? Basically meaning that any software that requires any development effort is novel?
That's a completely valid definition of worthwhile software, but to claim it's impossible to create an estimate to complete said development is absurd.
You just keep saying things are absurd or obvious but not putting anything behind it.
I hope this isn't a semantics game where things like "1 - 6 months" counts as an estimate in this context.
The point way back up this thread was accurate timelines for complicated, novel work have large error bars but those error bars aren't as bad as the equivalent error bars on estimating whatever "return" it is being pitted against.
I wouldn't consider something like "1-6 months" as a valid estimate, as that would indicate there is too much uncertainty and it needs to be broken down into subtasks that can be estimated with much less variance.
I've written what is probably several pages now in response to two individuals who are redefining terms in order to play the exact semantic games you mentioned, but in order to claim no estimation of any sort needs to be done. We seem to be done talking past each other now that I explicitly pointed out their usage of non-standard terms and my suspicions of why (having also unfortunately lived through software development managed by Gantt chart and other unpleasant experiences where someone who had no idea what they were managing was in control of a project), which is fine with me.
Feel free to describe your experience in practice when working in an organization where software developers answer to no one but themselves and are never asked for any justification for their progress or any projections of when they will be finished (both of which would require estimation to provide).
If you are able to tell stakeholders something like you'll be done in 1-6 months or provide no insight at all into when your tasking will be done, do no tracking of progress internally, and perform no collaboration around the completion of synchronous tasks within your team, I'll acknowledge no estimation is taking place during that process.
But to my dismay, the Agile world quickly got colonized by certification schemes and consultants targeting large companies, so it rapidly turned into something that a lot of the early people were very disappointed with. I wrote about the dynamic some years back: https://williampietri.com/writing/2011/agiles-second-chasm-a...
How do you do either of the following without spending any time at all on estimates?
"Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale."
"At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly."
If I was your colleague, had invested in your company, or was considering giving you money to deliver some product and you refused to tell me when you think it'll be done, it would be the last time any of the above ever happened.
You wouldn't accept that from any service you were attempting to hire, so I don't know why it would be acceptable for software development.
I feel like I've already answered the how question elsewhere. But the short version is a product-driven kanban-ish approach with frequent releases, small units of work, and engaged product management focused on actual real-world goals. This approach originated 25 years ago in Kent Beck's "Extreme Programming". Which did start with estimates, but teams doing well at it quickly realized (via said regular reflection) that they could do without it and still deliver well.
That some random person on the internet who wouldn't have hired me anyhow will now not hire me is a blow I'll have to learn to live with. But I have literal decades of stakeholders happy working this way. One way that happens is that they pick a date and then we built what fits between now and then. So they get the date, but no promises about what will get delivered.
In practice people like this better because beyond a certain very coarse level, estimates are about feelings of engagement and control. What this approach gives them instead is not just feelings, but actual engagement and control. What they get from me is not some unreliable bullshit date, but a commitment to deliver useful things early and often, so that we can together discover something better than their initial visions.
And yes, of course I accept this from people who do work for me. My side hustle is starting a pinball museum. When volunteers take on repairing a just-arrived machine, I never ask them for estimates. I ask them to focus on doing it right. They discover problems, figure out how to fix them, and try it out to see if there are more problems. The work takes as long as it takes. Or I hired a friend to design the logo. I never asked for a date. We iterated on it, discovering an number of things along the way, including through user tests. It took a lot longer than I would have guessed, and that's great, because it turned out better than I could have hoped.
I understand that this is unfamiliar to people, and apparently it's quite upsetting. But I swear this works. There are more things, Horatio.
It's not possible to do what you're describing (kanban/XP) for any commercial application or open source project that is intended to be taken remotely seriously without some amount of estimating involved.
I saw elsewhere that you're defining estimating as a very detailed and involved process, which the author of the article did not, the person I originally claimed estimates are impossible did not, and I did not.
I agree that's not necessary in most cases, if not all, to do the level of estimating you described.
And you'll note that I didn't include "people who are doing me a favor" in the list of individuals I'd insist on an estimate from.
You don't sound like you're one of these people, but I personally feel that software developers who act like they're performing special incantations over their keyboard and can't be expected to answer to anyone about their deliverables do us all a disservice, though maybe I should just be happy that they allow me to provide an alternative that is much easier to work with and results in additional business for me.
I am always fascinated when people tell me it is impossible to do something I have done. That plenty of people have done. For decades. But I shouldn't be surprised. Kuhn was not kidding around.
And yes, I strongly believe in accountability to business stakeholders, customers, and colleagues. I just think the better way is not paper fantasies, but demonstrated realities. Focusing on working software, customer collaboration, and responding to change, one might say. But people have said it before and it didn't make much difference.
You aren't doing what you're doing without performing what is commonly known as estimating, you just avoid using what seem like triggering words to you and the one other person who is almost spamming responses to my comments with the same information.
See "standard" vs. "schedule", and your very specific and formal definition of an estimate that is almost certainly not what anyone else has been using during this discussion and is not used in practice in most software development shops.
Kudos to you for delivering working features on a consistent enough basis that you've earned enough goodwill that people basically leave you to your internal processes and trust that you'll come through for them. I believe that's necessary to have a healthy working environment where you don't end up with what you, I, and every other software developer on earth is trying to avoid, which is an acrimonious relationship with the people with the money where they dictate what, where, when, and how we do our job.
But to claim that there is literally no estimating or scheduling taking place as you perform software development is just not true. You can post your disagreement on every single comment I've made on this topic if you want, your existing comments already speak for themselves on the matter.
As an outsider, I understand what wpietri argues, but I fail to understand yours --- even though I've read other branches of the discussion. Do you argue that wpietri's method still estimates tasks but the estimation is always around two weeks?
Look, software estimates are a whole studied field and have been for decades. Barry Boehm comes to mind here, as well as McConnell. When I talk about estimation, I'm talking about that. And I'm also talking about the bullshit-production process which yields similarly-shape artifacts that also gets called estimation. Something also discussed extensively in the literature.
If your point is that making things has a temporal reality to it, sure. But when people ask me for estimates, I give them two options: we start a formal estimation practice, where we do activities that produce what actual engineers would recognize as reliable numbers, or we do it kanban-style, which does not involve an estimation practice, but gives product stakeholders direct control over time/cost tradeoffs.
Some people, here and elsewhere, argue for a third option, which is going through various levels of rigamarole that gives horseshit numbers so that the more mendacious managers and executives can put on a performance of rigor while telling people with power what they want to hear. I refuse.
And I think it's that bucking the primate dominance hierarchy that makes people absolutely freak out when people talk about no-estimates approaches. Per POSIWID, the purpose of standard corporate "estimation" is not to produce reliable numbers. It's to avoid conflict. I get why people would rather freak out at me than make their bosses unhappy. But I would suggest that's a short-term gain for long-term pain. For organizations that want to actually get things done, hallucinatory project "plans" are bad for everybody. I think ultimately software engineering should be an actual profession. It's not looking great, but I live in hope.
None of which requires estimates. And the bit about working software as the primary measure of progress is specifically targeted against the estimate-driven culture of the time, where people would treat GANTT charts, "percentage complete" numbers, etc, as meaningful measures of progress.
Say you have problem A to solve. Then either one of those is true:
1) it has been solved before, ergo by virtue of software having a zero cost of copying (contrary to, say, a nail, a car, or a bridge), so there is no actual problem to be solved.
2) it hasn't been solved before, ergo it is a new problem, and thus at any moment you may turn a stone and discover something that was not foreseen (whether they are rabbits, yaks, bikesheds, dragons, or what have you eldritch horrors) and thus of unknown cost.
Any task that cannot be obviously fit into one or the other can nonetheless be split into an assembly of both.
Thus any attempt at estimates is as futile as gambling to win, tasks are only ever done when they're done, and "successful estimators" are kings of retconning.
> Thus any attempt at estimates is as futile as gambling to win, tasks are only ever done when they're done, and "successful estimators" are kings of retconning.
> It's all make-believe.
Software estimates are not futile or make believe. They are useful even if they are not always precise. That’s why the industry continues to use them.
"Bloodletting is not futile or make believe. It is useful even if the patient does not always survive. That's why physicians continue to use it."
"Trial by ordeal is not futile or make believe. It is useful even if sometimes Inquisitors reach mistaken conclusions. That's why Inquisitors continue to use it."
"Lottery-number-picking systems are not futile or make believe. They are useful even if some players never win. That's why players continue to use them."
It is a fully general argument which, if correct, would demonstrate that no practice that had continued for a period of time could ever be ineffective or counterproductive.
The amount of tap dancing and philosophizing some developers are willing to do to dodge estimates is hilarious.
It’s a skill… a basic part and critical part of engineering. IME the common thread between objectors is that they haven’t made a consistent effort to improve — developing, iterating, and refining their estimation process over time.
Yeah, every line of code is a unique snowflake piece of undefinable research the universe has never seen, equally unknowable and inscrutably enigmatic. But the workers at EngiCorp building EngiCorp products using EngiCorp project routines and resources first, second, and third quarter of 2025 are literal world experts at EngiCorp outcomes. They very reasonably should be able to estimate EngiCorp work in Q4, and account for EngiCorp realities, providing maps of future costs that can drive EngiCorp process improvement and investment.
If I ask for a decking estimate and get back sophistry and smug incompetence, I’m not talking with a super skilled professional deck builder. Doesn’t matter how they hammer, saw, or draw.
In my experience, they have negative usefulness in general.
I did work at one company that used an Agile/Scrum process, and though we didn't estimate actual time, we did estimate effort to complete tasks. That worked pretty well: we had an idea which tasks would be easy and which would be hard, and used that in sprint planning to more effectively use developer time.
But the key is no one was held to an estimate, since it was only that: an estimate. If something took 2x as long as predicted, oh well: we just adjusted our planning and worked around it, delaying other tasks for instance.
The problem I see is when managers demand an estimate, and then turn the estimate into a deadline and try to hold developers to that estimate no matter what.
> Software estimates are not futile or make believe. They are useful even if they are not always precise. That’s why the industry continues to use them.
The industry continues to fail when trying to use them. They have negative usefulness.
Knowing nothing else about him, I like him based on this alone.
I've been in planning sessions where someone would confidently declare something would take half a day, was surprised when I suggested that it would take longer then that since they were basically saying "this'll be finished mid-afternoon today"...and was still working on it like 3 weeks later.
Besides the usual unknown unknowns, I've also seen this happen with tasks that involve a lot of coordination in the SDLC. Oh the PR went up at at 2pm PST? Coworkers in EST won't review it until tomorrow. Maybe some back and forth = another day until it clears review. Then QA happens. QA is heavily manual and takes a few hours, maybe with some false starts and handholding from engineering. Another day passes. Before you know it the ticket that took an hour of programming has taken a week to reach production.
Time estimations, or conversations to days or other units, typically fail because if a developer says 1 day they might mean 8 focused uninterrupted development hours while someone else hears 1 calendar day so it can be done by tomorrow, regardless of if a developer spends 8 or 16 hours on it.
Are we estimating developer cost (investment cost, writing code only tome), development costs (investment costs including QA time), or time to delivery and first use? People want and use estimates for different purposes. You point out great reasons why knowing what the estimates are for is important.
This is essentially t-shirt sizing without all the baggage that comes from time. Your boss is trying to use the relative magnitude but it's inevitable that people will (at least internally) do math like "7 day tasks is the same as one week task", or worse over-rotate on the precision you get from day/week/month, or even worse immediately map to the calendar. Suggestion: don't use time.
But still we are much better at estimating complexity
Time estimations usually tends to be overly optimistic. I don’t know why. Maybe the desire to please the PO. Or the fact that we never seem to take into account factors such as having a bad day, interruptions, context switch.
T-shirt sizes or even story points are way more effective.
The PO can later translate it to time after the team reaches certain velocity.
I have been developing software for over twenty years, I still suck at giving time estimates.
Time estimations, or conversations to days or other units, typically fail because if a developer says 1 day they might mean 8 focused uninterrupted development hours while someone else hears 1 calendar day so it can be done by tomorrow, regardless of if a developer spends 8 or 16 hours on it.
Yes, I've seen this too in sprint planning. However, the layer of indirection I think is helpful. If you use actual time, then when something isn't done after 1 week when that was the estimate, then bosses are asking why. If your estimate was simply "8 story points", then the bosses can't point to a calendar and complain. They can try to argue that an 8-point task should be done in a week, but then you and the scrummaster remind him that points don't map directly to time, just effort.
Here's my observation: ballparking an estimate for a whole project, in my experience, tends to be more accurate than estimating each task and adding them together.
I like to think of this as 'pragmatic agile': for sure break it down into tasks in a backlog, but don't get hung up on planning it out to the Nth degree because then that becomes more waterfall and you start to lose agility.
Hours is insane. But ultimately time is money and opportunity cost. Software engineering can’t be the only engineering where you ask the engineers how much something will cost or how much time it will take and the answer is “it’s impossible to know”. Even very inaccurate estimates can be helpful for decision making if they are on the right order of magnitude
First, people asking for estimates know they aren't going to get everything they want, and they are trying to prioritize which features to put on a roadmap based on the effort-to-business-value ratio. High impact with low effort wins over high impact high effort almost every time.
Second, there's a long tail of things that have to be coordinated in meat space as soon as possible after the software launches, but can take weeks or months to coordinate. Therefore, they need a reasonable date to pick- think ad spend, customer training, internal training, compliance paperwork etc.
"It is impossible to know" is only ever acceptable in pure science, and that is only for the outcome of the hypothesis, not the procedure of conducting the experiment.
> "as soon as possible after the software launches"
This isn't true, just desired, and is one of the main roots of the conflict here. OF COURSE you would like to start selling in advance and then have billing start with customers the instant the "last" pr is merged. That isn't a realistic view of the software world though and pretending it is while everyone knows otherwise starts to feel like bad faith. Making software that works, then having time to deploy it, make changes from early feedback, and fix bugs is important. THEN all the other business functions should start the cant-take-back parts of their work that need to coordinate with the rest of the world. Trying to squeeze some extra days from the schedule is a business bet you can make but it would be nice if the people taking this risk were the ones who had to crunch or stay up all night or answer the page.
Trying to force complicated and creative work into a fake box just so you can make a gantt chart slightly narrower only works on people a couple times before they start to resent it. 10x that if management punishes someone when that fantasy gantt chart isn't accurate and 100x that if the one punished is the person who said "it's impossible to know" and then was forced into pretending to know instead of the person doing the forcing.
My take: if they have not done the work to get to at least some degree of a spec, and they won't give you time to review and investigate, they get nothing more than a vague, relative t-shirt size.
I think software is one of those VERY rare things, where inaccurate estimates can actually be inaccurate by "orders of magnitude". After 20 years in the field, I still managed to use 2 months of time on a task that I estimated as 10 days.
A rule that has suited me well is to take an estimate, double it, and increase by an order of magnitude for inexperienced developers.
So a task the say would take two weeks ends up being 4 months.
For experienced developers, halve the estimate and increase by an order of magnitude. So your 10 days estimate would be 5 weeks.
The biggest estimation effort I ever took part in was a whole-system rewrite[1] where we had a very detailed functional test plan describing everything the system should do. The other lead and I went down the list, estimated how long everything would take to re-implement, and came up with an estimate of 2 people, 9 months.
We knew that couldn't possibly be right, so we doubled the estimate and tripled the team, and ended up with 6 people for 18 months - which ended up being almost exactly right.
[1]: We were moving from a dead/dying framework and language onto a modern language and well-supported platform; I think we started out with about 1MLOC on the old system and ended up with about 700K on the rewrite.
10 days was already after I used this algorithm. Previous several tasks on that codebase were estimated pretty good. Problem with this is that some tasks can indeed take SEVERAL orders of magnitude more time that you thought.
One of the hardest problems with estimating for me is that I mostly do really new tasks that either no one wants to do because they are arduous, or no one knows how to do yet. Then I go and do them anyway. Sometimes on time, mostly not. But everybody working with me already knows, that it may be long, but I will achieve the result. And in rare instances other developers ask me how did I managed to find the bug so fast. This time I was doing something I have never before done in my life and I missed some code dependencies that needed changing when I was revieving how to do that task.
Something often overlooked in cost/schedule estimates is the nature of joint probability of actions slipping. Action A slips and causes action B to slip. I think software is tougher to estimate because the number of interfaces is often much higher, and sometimes more hidden, than in hardware.
My home construction slipped 6 months on 2 year build time. It happens in construction very often.
> software is only tougher to estimate if incompetent people (vast majority of the industry, like 4+ million) is doing the estimating :)
No, it is tough to estimate, but not only for incompetent people. And "incompetent" can be stretched to "don't know what he's doing", which is how I operate most of the time. I don't know what really needs to be done until it's done. Main part of my work is research on what actually needs to be done, then "just" implementing it. If I waited with estimating until I know what needs to be done, I would spend 3/4 time estimating and then 1/4 with clear understanding and good schedules (example description: I will be clicking keys for 5 hours).
> My home construction slipped 6 months on 2 year build time. It happens in construction very often.
Tangent, but I have at least 3 friends that would've (in retrospect) been nothing short of ecstatic if their home construction had "only" slipped 6 months on a 2 year timeline.
That’s a bit of a strawman considering I was deliberate in saying hardware interfaces are limited and not saying they are zero. The number of interfaces in software is often going to be orders of magnitude greater. The network effects and failure modes will often increase geometrically with the number of interfaces. In fact, big construction design firms have tools to easily identify and mitigate the “clashes” you bring up and those tools tend to work well because there is a finite number and the designs are well-documented (as opposed to software where changes are relatively cheap and easy so they often occur without documentation)
Saying incompetence is the reason is a trivial rebuttal that ignores the central claim about complexity. It’s like saying “the reason why we don’t have a theory of everything is because we don’t have competent physicists”
The next natural progression of this line of discussion between "the business" and engineering is for them to agree on a time range as an estimate. Engineering doesn't want to say it'll be done in 6 weeks, but they feel okay saying it will take between 4 and 20 weeks so this estimate is accepted.
You can guess what happens next, which is that around week 8 the business is getting pretty angry that their 4-week project is taking twice as much time as they thought, while the engineering team has encountered some really nasty surprises and is worried they'll have to push to 24 weeks.
it is still better to give a range though because 1. it explicitly states the degree of unknown and 2. no boss is going to accept 4-20 weeks, which means you start talking about how you can estimate with better accuracy and the work required to do so, which is a major goal of planning & estimation.
> Software engineering can’t be the only engineering where you ask the engineers how much something will cost or how much time it will take and the answer is “it’s impossible to know”.
Because it's not engineering at all. But even if it was, plenty of engineering projects are impossible to estimate - the ones that are doing something novel - and disliking that fact doesn't make it go away.
> Even very inaccurate estimates can be helpful for decision making if they are on the right order of magnitude
If what the business wants is an order-of-magnitude, they should ask for that; often (not always!) that's a lot easier.
> I honestly don’t know what the PO and TL gains with this absurd obscenity
There are marketing campaigns that need to be set up, users informed, manuals written. Sales people want to sell the new feature. People thinking about road maps need to know how many new features to can fit in a quarter.
Another reason is that figuring out what the software to be written should actually do, and how it should work, is work that is part of the project and the time it will take needs to be estimated.
As well as the actual development work that will result, which isn't known yet at the time of estimation.
I call it the "Persistent Incompetence of Software Development", which is another perspective on estimation, focused more on expertise. A chef that cooks pizzas, cooks the same pizza over and over again and becomes amazing at it. If you are a developer that writes the same code over and over, you are terrible at software development. A good software developer should always be solving new problems, as by the nature of software, once they solve a problem, they never solve that (exact) problem again. So we are persistently incompenent.
Which is why software development can't be estimated, as well. Because it is all, as you say, novel. With infinite error bars.
At this point, I can't take anyone seriously that believes software dev can be estimated.
There is very little novel about most B2B CRUD and internal bespoke apps that most developers are doing. The novel part if any is implementing the business vertical logic
"... anybody with any brains has already left town..." -- Bob Dylan
Anybody with any business experience has already isolated themselves from the certainty of software project failure, where "failure" is a euphemism for "late." So it doesn't matter if software can't be estimated.
This can be nerve-wracking to a beginner, but one gets used to it over time.
It’s like choosing to crash random parties everyday — at first, exceptionally novel but facing the unknown becomes itself mundane. Humans adapt. They build mental modals to approach the unknown and these mental modals are predictable.
> If I ask you how long it'll take to add a new database field, and you've added a new database field 100s of times in the past and each time they take 1 day, your estimate for it is going to be very spot-on.
But in the software world, predictable and repetitive tasks are also the kinds of tasks that are most easily automated
The slow part isn't following a written "log in and run ALTER TABLE" runbook, it's the review of "does this solution make sense given what we're trying to do".
Spot on! Now add one more dimension (from machine learning projects and pretty much other research fields) -- you also never know if the result are going to even work.
Maybe the data just doesn't have the correlation you want. Maybe you just didn't try harder, never know for sure. But owners reasonably demand some estimations.
Repetitive and easy or worth automating are not the same. The fundamental problem is that a setup capable of solving any of the requests that come to it is as complex as just a whole programming language and then you’re back to square one.
As a project manager, it sounds like you're making excuses. Just give me a number, trust your gut!
We have a fundamental failure to communicate, what we're doing. The game project managers and finance believe we're all playing is a regression towards the mean, where everything is additive and rounds up to nice consistent formulaic sums. Whereas software development follows power law distributions. A good piece of software can deliver 10x, 100x, or 1000x the cost to produce it (ex: distribution, cost of delivering another copy of software is near 0 once written). You don't get that sort of upside with many other investments. Finance is happy with an NPV 8% above what they invest in. This means that when software people talk, everything they say sounds foreign, and everyone assumes it's because of jargon. It's not. The fish don't know they're swimming in water. When the fisherman comes, everyone is caught off guard.
So we get what the author talks about
> The estimates stopped being estimates. They became safety railings against being held accountable for unreasonable expectations.
We. Pad. Like. Crazy. Yes this is inefficient. Some project managers recognize this. We get theory of constraints. But rather than cull the layers of hierarchy that lead to the padding in the first place, all the blame for failure goes back to developers. Get hit on the head enough and you will stop acting in good faith and pad to save your ability to feed and cloth yourself.
Padding ties up capital, it reduces credibility, it delays deployment, it adds costs through delay. It is bad for organizations. However, it is a great solution if you're a worker in a bureaucratic environment that can tolerate large costs, but is intolerant of 1-day of schedule slips. It's a great solution for complacent management, who are confused about the game they're playing and wants to report that they're "on track", which means "not late".
The agile solution of incremental value delivery is a compromise, and can produce good outcomes for functional changes. But agile has unacceptable failure modes when working on infrastructure and satisfying system constraints. Agile can work okay for programmers, but it's not a solution for engineers. Acknowledging, owning, and managing risk is more scalable, but you have to have leaders who acknowledge that they exist and have the maturity to take on that responsibility.
> Padding ties up capital, it reduces credibility, it delays deployment, it adds costs through delay.
Well done timelines are a negotiation between the stakeholders and engineers. The stakeholders need something done for the business, the engineers give a timeline. If that timeline works for everyone, great. If it doesn't, then the stakeholders will ask if it can be done in a faster time.
A timeline that lands on time, or early, is good. The point of timelines is that teams outside of engineering are resourcing their projects based on your timelines. They may have made outside commitments to customers, they may be lining up marketing, they may have embargoed PR, it may be delivered by someone at a conference, etc.
A project running late can be catastrophic. Bad customer relations, wasted marketing spend, pulling back stories from PR, delays for dependent teams, etc.
You pad to make sure your timelines aren't overly optimistic, because we're all bad at estimating, and it's possible our dependencies are too. By padding, when it comes time to negotiate for shorter timelines, you also have some wiggle room.
Bureaucratic environments tend to be larger companies and they care about schedule slips, because they have more teams being impacted, and those teams are handling larger numbers of overall projects. Schedule slips can lead to cascading failures.
This is going to be immediately downvoted, but whenever I see software developers mention that estimating is uniquely challenging for software it immediately calls out a bunch of red flags. Estimates in software are challenging but no more or less challenging than other industries.
Typically the people that single out software as a unique snowflake in the world of delivery and estimation really just say out loud they have no experience in project management and no experience working outside of authoring code. The things that make estimations most challenging in software are the same factors that make estimations challenging everywhere else.
I don't think it's unique, but many software projects are inherently prone to big surprises you can't catch until you start the work. I would love to keep estimating and estimating and get an accurate number, but it costs money and, eventually, you're just working on the project.
I don't know how typical teams work, but in my case, new projects always come on top of other obligations. It may take 1 day to add the field, but how many meetings, fires, or other disturbances will happen during that day?
If software developers want to be then seriously as a profession, they need to be able to provide and justify estimates for their work.
Everything you said could apply to a new bridge, building, pharmaceutical compound, or anything else that is the result of a process with some known and some unknown steps.
Pharmaceutical compounds frequently don't make it to market after significant investment.
No one in that industry is giving estimates based on developing brand new drugs - they're giving estimates related to manufacturing lead times, unalterable physics time lines, and typical time to navigate administrative tasks which are well known and generally predictable (but also negotiable: regulations have a human on the other end). All of this after they have a candidate drug in hand.
Same story with bridge building basically: no one puts an estimate on coming up with a brand new bridge design: they're a well understood, scalable engineering constructions which are the mostly gated by your ability to collect the data needed to use them - i.e. a field survey team etc. - and also once again, regulatory processes and accountability.
Yes, that's my point. There's way more uncertainty in trying to bring a new drug to market or build a new bridge than creating yet another CRUD app, yet somehow they are any able to break these efforts into tasks that can be estimated and tracked and many software engineers think they should be exempt from any accountability to schedule or budget.
Take a look at the top of this thread and see what we're talking about.
The fact that people in many industries are not good at estimating doesn't mean that it's impossible in software development specifically and uniquely, as was originally claimed.
> Everything you said could apply to a new bridge, building, pharmaceutical compound
"Everything"? So
> predictable and repetitive tasks are also the kinds of tasks that are most easily automated, which means the time it takes to perform those tasks should asymptotically approach 0.
Also applies to bridges? Bridges require a ton of manual human input at every stage of construction, regardless of how predictable and repetitive the work is. With software, we can write software to make those tasks disappear. I've yet to see the bridge that can build itself.
You should be talking about the compilation process, because that's the thing that puts the cost of non-novel software at near zero, and you can't recompile a bridge for free.
Estimating the duration of each step of a process only works when you know what the steps are.
Effectively free distribution is what makes the cost of non-novel software so low. Compilation isn't even needed for executables that can be used as is.
That said, implementation is one part of developing software. Design and test are also necessary and can take a non trivial amount of time.
And yes, you need to know what the steps are to build something. If you don't, you don't know what you're doing, which is a bad thing.
You think the pharmaceutical industry just gives a bunch of resources to chemists, says "godspeed", and waits around for the chemists to report back at their convenience?
In my limited exposure to the industry, that's not how it worked. They have budgets, timelines, and progress is tracked as it is determined whether there is a viable path to a marketable drug.
From both the developer and manager side of things, I've found that the most important attribute of estimates is frequently the least paid attention to: that they be kept up to date.
When you discover more work hidden under that "simple" pile of code, you absolutely HAVE to update your estimate. Add more points, add more tickets, whatever. But then your various managers have the ammunition to decide what to do next - allocate more resources to the project, descope the project, push back the release date, etc.
Far too frequently, the estimate is set in stone at the start of the project and used as a deadline that is blown past, with everyone going into crisis mode at that point. The earlier the estimate is updated, the calmer and more comprehensive action everyone responsible can take.
Which if you try to do - those agile people will kill you for it.
They wrangle a number out of you which goes into an user story estimate, which feeds into a Gantt chart they use to make their pretty powerpoints they present to upper management to say that the feature will make it into the Q4 release.
If you move this number around the whole estimation will crumble, not that it wont in real life but you deprave them of two things - an illusion of control and somebody to blame when things go south.
> Which if you try to do - those agile people will kill you for it.
Does this actually happen to you? This is literally the whole point of agile, is to change the plan as you learn more about your work. If you didn't want to change the plan you'd spend a lot of time on up-front planning and do waterfall.
Like, a Gantt chart is more or less explicitly anti-agile. I'm aware of the 'no true Scotsman' thing but we shouldn't buy into people using agile terms for what is really a BDUF-based plan.
Yes and millions of other devs who work in an enterprise 'agile' environment - where a single huge project is/was developed by armies of developers work on a single product with a strict-ish release cadence?
Have you heard about the horror that is SaFe?
I'm not convinced that true agile works or has ever worked on a project that was bigger than a dozen devs.
In practice, it's just another dishonest way of selling consulting hours, infantilizing and disempowering devs, and putting folks who have zero subject matter knowledge in charge by doing these feelgood rituals.
Agile (scrum) in practice at enterprise-scale projects tends to be a combination of feelgood BS +top-down micromanagement (product owners dicking around with task priorities) +traditional project management.
One of the key ways these agile people are incredibly dishonest, is that Agile at the top level is sold to enterprises as a way of keeping the old high-level project management style, with push-only command-structures, and agile people subsequently try to sugarcoat it as it somehow 'empowering' the devs and giving them autonomy, when the truth couldn't be farther from it.
Yes, and I successfully argued against it by pointing it was was a wolf in sheep's clothing. It has as much to do with agile as waterfall does.
> I'm not convinced that true agile works or has ever worked on a project that was bigger than a dozen devs.
It works fine, large projects are inherently trouble which is why organizations should spend some energy into reducing the scale of an individual team's work rather than piling dozens or hundreds of people into yet another Too Big to Fail megaproject. If Google can build and maintain Bigtable with like a dozen devs then maybe you don't need 200 people and a PMO for your enterprise data warehouse consolidation project.
In fact the biggest issue with SAFe is the size of the project you'd try to use it on, not that it references agile style methods. Waterfall methods were even worse, which is the only reason charlatan consultation manage to keep selling organizations on things like SAFe.
> One of the key ways these agile people are incredibly dishonest, is that Agile at the top level is sold to enterprises as a way of keeping the old high-level project management style, with push-only command-structures, and agile people subsequently try to sugarcoat it as it somehow 'empowering' the devs and giving them autonomy, when the truth couldn't be farther from it.
You're right that this is dishonest and that people try and fail to cargo cult successful efforts they watch from afar. But that doesn't mean these successful teams weren't successful, or that there aren't common attributes of those successes.
That's always been the problem with methodological fixes to the software delivery process for organizations, you usually can't impose the structure from the outside anymore than you can meld your bones with adamantium without having a crazy mutant healing factor...
Yes, but the comment is targeting those people that usually would say about themselves that they embrace agile, while actually fighting everything that changes their little schemes...
So should we be talking about how "those democratic people" put everyone who disagrees with them in concentration camps because that's what the Democratic People's Republic of Korea does?
There are, of course, extreme-right-wing reactionaries who make exactly that argument, but I don't think their example is a good one to follow.
I have noticed the same for a lot of long-running software projects. The estimate is created at the start an never revised.
Projects can and will fail or run late; but heck; a 6-months projects cannot found late after 5 months and 29 days; things must be discovered early, so that the most important issues can be addressed.
> But then your various managers have the ammunition to decide what to do next - allocate more resources to the project, descope the project, push back the release date, etc.
In your view, is this a good thing or a bad thing for developers? In other words, are developers incentivized to update their tickets because it gives their managers "ammunition" to make project changes?
I find that this is usually more evidence of organizational dysfunction.
Sometimes, its developers who don't have enough ownership - someone higher-ranking filled out all the tickets for them and left no room for interpretation, so changes require trashing a bunch of irrelevant tickets and writing a bunch of new ones, making for more communication overhead and less time spent in development. Developers are disincentivized to update tickets since doing so just becomes write-amplification for more bureaucratic overhead.
Sometimes, it's a communication breakdown. Tools like Jira are necessarily asynchronous, and decisions about what to build are necessarily political. Politics and asynchronicity don't mix, such discussions need to be held face-to-face. If they're held face-to-face, then having developers update Jira is pure overhead. Developers are disincentivized from updating Jira because either it's a one-sided conversation (lag in getting notified that your ticket was closed and the work on it should be thrown out, PMs don't always follow through with issue links from the closed issue to the newly opened issues, so developers lack context on what decisions were made on replacements), which is emotionally draining, or the conversation happens outside of Jira, in which case updating the ticket is a formality/not actual communication, in which case it feels like a waste of time.
For most development work, I'd argue that developers shouldn't be making changes in Jira, their managers should be, on the basis of frequent, synchronous communication with the developers they manage.
Exactly. And I’ve found this is incredibly hard to get people to do, developers especially. No one wants to say a task is going over / taking longer than expected but will gladly run over. It’s hard for pms to resource if estimates are not updated as scope and complexity changes.
Also doesn’t help when estimates become due dates.
This is why stuff like Jira is so polarizing for many developers. It is an additional translation tax on me, the developer, to have to go constantly keep this tracking tool thing up to date so that the product owners/managers stay off my nuts. The burden of effort is placed on me to make their tool functional that they can have what they need to provide roll up status. This tool virtually never benefits me, it's always a time consuming sidequest that breaks focus and scatters important information all over a kanban board where tickets disappear and makes it really hard to cohesively work as a unit to get anything done. Like many orgs we've had PM's and stuff try to mandate expectations on when those updates should be provided, somehow completely oblivious to the human nature of development: you should never expect that a developer can simply hop in and out of moments of deep focus and suddenly provide a bunch of updates regularly when it's convenient for you, as if that has zero impact on their work. It takes a toll, full stop. I won't say no to it, but I do expect PM's to know that style of PM'ing is effectively asking me to make your job easier by populating your tool with updates that are useful to you and not me because I'm well aware you aren't involved enough to assess where things are yourself.
We've gone through so many cycles of how to use Jira well at my org where these frustrations are shared and we try a different approach, and we're finally starting to converge on the idea that this has historically been a little too lopsided requiring too much tax on the developer doing the actual work. We agreed on a new approach that has actually been pretty awesome: the product owners or managers that are trying to direct a body of work must be a little more in the trenches with us to have an overall understanding of where the different pieces are in a moving body of work. We don't expect them to understand the nitty gritty work in the trenches, but at the same time no more 30,000 foot view product managers who just ask for status updates at EOD. _Everyone_, developers included, is responsible for keeping documentation up to date as we go. So we have central working-bodies of information to find details without having to cruise thru 100+ jira tickets to find details we're looking for. The expectation is that they're engaged enough with development whether in chat or on meetings that if they were blindsided by an executive asking for an update, they could speak to it with some authority over at the water cooler without having to go check Jira. This has really helped weed out the lazy product owners/managers, has forced them to thoughtfully consider their meeting schedules, and has placed the exceptional ones in the pod of work being done and really added a lot of velocity and together-ness about the things we're pushing along.
This approach we're using now was born out of some hurt feelings from projects that didn't go so well & we had to have some real restrospective convos where everyone aired out their beef. Those are good convos to have, I think a lot of teams would find that people aren't deceptively trying to screw you over. Being encouraged to level set human-to-human genuinely is one of the greatest parts of working where I work. Always walk away from those types of chats learning valuable things: for the most part our product owners really do care. Not just about their career aspirations but also about _us_ nerdy and sometimes socially maladjusted developers. They look forward to working with us, and they want to make this as easy as possible for themselves but also for the developers. In the past they spent a lot of time in planning phases trying to scaffold out a project in Jira and attaching timelines to it so that their needs are met to provide predictable timelines to their bosses... but also with the hope that by plainly outlining work it sort of 2 in 1 satisfies our needs and will make development timelines a breeze. We've had to ask them to cede rigidity on that latter part because even the software architects admit the work being done is often a moving target. And when that target moves, maybe you realized you need to add a pivotal software solution to the stack, you can sometimes throw like 45 planned tickets into the dumpster. New ship dates need to be assessed. This was our reality check that we were all collectively shit at adapting to the dynamic nature of any given project. Now, our product owners decided that the expectation they have of their own role is that they understand this dynamic and are prepared and willing to make the case about why the shipping timeline must change. So there's actually a pain point solved here: don't break your back doing so much up front work to try and guess/capture what the body of work might look like only for it all to possibly get thrown away, involve architecture a bit more in the planning phases, but most importantly let's engage throughout the project and we'll try our best to have shared ownership/interest in making sure where we are in the project is broadly understood by everyone involved.
We're currently in phases of implementing a major platform right now and it's just night and day better and dare I say fun. We're still keeping Jira up to date, but the product owners and PMs are more or less managing this as it is a tool they find useful. Removing the "can you update this ticket please" 24/7 has forced them to be a little more involved and have the right chats, but also makes us developers happier to jump in and get it updated on our own volition because we also want to help them have an easier time. If my PM pings me and says "hey I'm looking at this ticket that's stuck in blocked, I just wanted to make sure we got an update from so-and-so about provisioning this credential so I can follow up if needed" I will likely automagically jump in and be like "still stuck, but let me update that ticket for you there's a couple key details I want to make sure our there for you before you reach out". There's an inherent "we're both interested in seeing this ticket through" here that doesn't strike a nerve with either party. Pretty much everyone involved both developers & non developers has a really solid read on where anything's at and we're all just talking a lot more. And for developers I find that it's really good, even if you've got them committed in one narrow body of work, to understand the larger pieces in motion. When they're in tune with the broader orchestration of a projects timeline, they tend to weigh in during unsuspecting moments that might tie seemingly unrelated pieces together. They might be assigned to work on x, but in a group chat about y they notice y has a dependency on x and they'll speak up and call out the need to test that they both work together. We've had a lot of great callouts materialize like this, and on a human-psyche level I think it snowballs & avalanches encouraging developer participation in a way that is really meaningful for PMs. It's interesting that Jira & the expectation that predicting development time in an arena of uncertainty was previously in the way of forming the group dynamics we have now. Jira, despite just being a tool, can really amplify a lot of bad outcomes when it's used by those who aren't fit to be near development, it sort of devolves into a two dimensional behind-schedule tracker that detrimentally impacts how team members on the ground communicate with each other.
And since we're talking a lot more there's just like... way more memes. And memes are important in any development timeline. We prioritize laughing through the pain together.
This is a fantastic description of why Technical Program Managers (TPMs) can be force multipliers. Imagine involving someone who understands both the needs of Product and Engineering and whose job it is to allow both of them to focus on where they add the most value. We do exist!
It is very much this, your types are insanely valuable to how quickly and effectively we can blast through massive bodies of work that are well beyond the scale of personal-hobby-project. We're in the annoying HR-involved process of revising the role internally to actually include the word 'Technical _' & the bullet points of expectations will demand such capacities. It's been an exciting shift for us and we've reappropriated non-technical product owners/managers to roles they are happier in (and believe it or not we are happy for them).
We didn't get it right the first, second, third, fourth or fifth time. I'd say as an org we are learning lessons that other orgs may have learned a decade ago, but it's just nice to come to these conclusions on our own & really understand how our setup came to be. We hope we remember that things can evolve again in the future, but are grateful that we journeyed together and didn't just throw in the towel and fire people when things didn't work out. We believe sending people through some fires of hell, reassessing, then reattempting is sort of a cycle that levels people up.
We hope to have more technical product guys on board up ahead cause it's a dream setup that really organizes and harnesses velocity in all the right places, but also effectively stops unnecessary side-quests. It's so nice to have a technical product guy step in and say "no" to some absurd executive request because he/she is well aware of what such an implementation would look like. They can actually be vanguards and stewards over development personnel in their own way and it seems to go hand in hand with a lot of mutual respect for each other. Always get a kick out of nerding out over possibilities with our technical product dudes.
The biggest problem I've seen isn't the estimate itself but the telephone game that happens after. You say "probably 2-3 weeks" to your manager, who tells the PM "about 2 weeks", who tells sales "mid-month", who tells the customer "the 15th".
By the time it reaches the customer, your rough guess with explicit uncertainty has become a hard commitment with legal implications. And when you miss it, the blame flows backward.
What's worked for me: always giving estimates in writing with explicit confidence levels, and insisting that any external date includes at least a week of buffer that I don't know about. That way when the inevitable scope creep or surprise dependency shows up, there's room to absorb it without the fire drill.
This is why I push for Kanban whenever I am a PO. If we can ballpark an estimate, I can prioritize it. If we cannot ballpark an estimate, I can prioritize the research to clear out some of the unknowns. But most importantly, we set an expectation of rolling feature rollouts, not inflexible release dates. We communicate both internally and externally the next few things we are working on, but no hard dates. The article correctly identifies that hard release dates communicated to customers are the root cause of problems, so I simply don't give such things out.
From your description, SCRUM, could work just as equally.
Don't get me wrong, I'm a fan of Kanban, it's awesome for visually presenting where bottlenecks are for tasks, but estimations aren't a feature of that.
But SCRUM maybe, where people are having a sprint planning meeting maybe more what you're thinking?
Kanban in IT world in my experience implies approach where you focus on the work and tasks as they come based on priority. It doesn't imply what is on the board is finished strictly by some date, as the whole premise is that you can't really know.
SCRUM implies sprints where you agree in advance what will be actually pulled into sprints and delivered by the team so spillovers are not really expected / wanted.
When working with kanban I maintained a average number of card done per days. if someone asked when some card woud be done, I just multiplied the number of card ahead of that one by average and get an estimate. You can estimate the cards but usually it doesnt really improve accuracy as tasks are on average the same size.
What the article suggests is basically Kanban. It's the most effective SW development method, and similar scheduling system (dispatch queue) is used by operating systems in computers. However, management doesn't want Kanban, because they want to promise things to customers.
You can make good estimates, but it takes extra time researching and planning. So you will spend cycles estimating instead of maximizing throughput, and to reduce risk, plan is usually padded up so you lose extra time there according to the Parkinson's law. IME a (big) SW company prefers to spend all these cycles, even though technically it is irrational (that's why we don't do it in the operating systems).
Another reason kanban doesn't work for large projects is because you have to coordinate your cycles with multiple dependencies teams roadmaps and releases.
I don't think so, only if they need to have a schedule as well. Most OSS projects operate as Kanban and it's just fine.
Waiting on a dependency is kinda like waiting on a lock held by another process in the operating system. It has little bearing on whether dispatch queue is effective or not; in fact, it shows the solution: Do something else instead of waiting. (This is why the OS analogy is so useful for project management, if only PM's would listen!)
It's again, if you need to plan things ahead (for some reason) when the dependencies become a problem.
But maybe I misunderstand what you mean, if you still disagree provide a more specific example.
As an investor, I don't like a investment that throws away 10-30% of its resources, perpetually lowers morale except among the least creative and misses opportunities because their competition is faster.
When teams don't need strong estimates, then Kanban works well.
When teams do need strong estimates, then the best way I know is doing a project management ROPE estimate, which uses multiple perspectives to improve the planning.
> E = Equilibristic estimate. This is based on success as 50% likely such as for critical chains and simulations.
I've found giving probabalistic estimates to be hopeless to effectively communicate, even if you assume the possible outcomes are normally distributed, which they aren't.
Critical chain planning has a purpose for the 50%. It's not perfect-- but it's better than other ways, in my experience.
Wikipedia: "The justification for using the 50% estimates is that half of the tasks will finish early and half will finish late, so that the variance over the course of the project should be zero... Because task duration has been planned at the 50% probability duration, there is pressure on resources to complete critical chain tasks as quickly as possible, overcoming student's syndrome and Parkinson's Law."
Three-point or PERT estimates are in the same vein, but are just an old and established business process concept and not a trademarked "project" from a (defunct?) consultancy.
And yeah, they can be useful when you can must put numbers to things and can decompose the work into familiar enough tasks that your estimation points are informed and plausible. Unfortunately, in a field often chasing originality/innovation, with tools that turn over far too often, that can be a near-impossible criteria to meet.
This sounds like that if you don't trust a fortune teller, you can mitigate it by going to 4 different fortune tellers, and then somehow combining their predictions into your much more certain future.
Yes except in practice it's worked out more like this... Realistic is favored by tech teams. Optimistic is favored by salespeople and marketers. Pessimistic is favored by lawyers and compliance people. Equilibristic is favored by project managers and military.
Each stakeholder gets the kind of fortune they want. The important aspect (to me) is that the four viewpoints show all the stakeholders that their viewpoints are quite different.
So now you have 4 wrong estimates to work with :)
To have some predictability you should have small features. That's the only thing that can give you strong estimates. No industry has solved giving strong and correct estimates - it's in the name, it's estimation!
Yes small features help enormously. I've used ROPE and work breakdown structures (WBS) to estimate multiple projects at thousands of staff hours to under 4% of actual. The ROPE advantage is that it uses multiple perspectives, which gets stakeholders understanding that the estimates are imperfect and depend on many factors.
The trouble with estimation is that few places record the estimates and the actuals for future reference.
As I've pointed out before, the business of film completion bonds has this worked out. For about 3% to 5% of the cost of making a movie, you can buy an insurance policy that guarantees to the investors that they get a movie out or their money back.
What makes this work is that completion bond companies have the data to do good estimations.
They have detailed spending data from previous movie productions. So they look at a script, see "car chase in city, 2 minutes screen time", and go to their database for the last thousand car chase scenes and the bell curve of how much they cost. Their estimates are imperfect, but their error is centered around zero. So completion bond companies make money on average.
The software industry buries their actual costs. That's why estimation doesn't work.
Movie people are doing essentially the same thing over and over for 100 years. Everyone in the industry does things the same way. Their projects last only 1-2 years and are totally done once "in the can". In my experience this is almost never the case with technology based projects.
If your database of software projects has a thousand web browsers in it, you have to ask why all those programmers couldn't work together and write just ten or fifteen web browsers; maybe they're hobbyists who are reinventing the wheel for fun? If so, is that a good basis for estimating a new web browser?
The first time I worked on a database-backed web site, my team had written their own ORM, and while I was there we put together a thing so we could use the same view on more than one web page. The second time, we used web.py and no ORM. The third time, we used Ruby on Rails, which solved most of the problems we had on the first project I mentioned out of the box. Most recently, I used Django, which is roughly as convenient as Rails but easier to debug. I haven't tried it, but I'm guessing Claude Code could probably put together a usable Django site in a few minutes.
So achieving equivalent results requires very different activities over time, because each time someone does it, we accumulate not just knowledge but code that makes it easier the next time—thousands of times easier, if the novelty is small enough. This impedes estimation.
Often, too, the things we're trying to estimate are things like "add automatic Dart dependency support to Blaze". The time-consuming part of this is not breaking the rest of Blaze in the process. What reference class do we use for estimating adding new languages to Blaze without breaking it? Is Dart more like C++ or more like Go? These are questions for lawyers, not actuaries.
Maybe the videogames industry is different because they don't share code and don't maintain it? But it seems like an awful lot of games these days are getting written in Godot and Unity where 15 years ago people would have written their own engines... and problems that were major technical challenges for Quake are trivial now.
If someone wants ragdolls in their game, they can just license your code, right? They don't have to write it themselves. A dataset of how long it took people to implement ragdolls from scratch wouldn't provide any insight into how long it should take to integrate your ragdoll library.
My employer does, but only analyses the overage, so when a project comes in under estimate they think it's because it went smoothly, the reality is actually that estimates only ever tell you how good you are at estimating.
The best hack for improving estimation is first never giving a single number.
Anyone asking for a single number, without context, doesn't know what they are doing; it's unlikely that their planning process is going to add any value.
I think they call this being "not even wrong".
Instead you should be thinking in probability distributions.
When someone asks for your P90 or P50 of project completion, you know they are a serious estimator, worth your time to give a good thoughtful answer.
What is the date at which you would bet 90:10 that the project is finished?
What about 99:1? And 1:99?
Just that frameshift alone solves a lot of problems.
The numbers actually have agreed-upon meaning, there is a straightforward way to see how bad an estimate really was, etc.
At the start of a project have people give estimates for a few different percentiles, and record them.
I usually do it in bits, since there is some research that humans can't handle more than about 3 bits +/- for probabilistic reasoning.
That would be 1:1, 2:1, 4:1, 8:1, and their reciprocals.
Revisit the recorded estimates during the project retrospective.
You can make this as much of a game as you want.
If you have play-money at your company or discretionary bonuses, it can turn into a market.
But most of the benefit comes from playing against yourself, and getting out of the cognitive trap imposed by single number/date estimates.
One interesting angle for me is that I am seldom given complete specs or requirements when asked for an estimate. Of course you ask questions to try to determine key information that has not been specified but often the answers are not available or fully reliable.
So any estimate has to include uncertainty about _the scope of the work itself_ as well as the uncertainties involved in delivering the work.
The natural follow on question when you present a range as the answer to an estimate is: what would help you narrow this range? Sometimes it is "find out this thing about the state of the world" (how long will external team take to do their bit) but sometimes it is "provide better specs".
This is what I do but I don't try to make it complicated with too many numbers. "2 weeks but there's a 10% chance something bad happens and it takes longer".
I have no problem if they just hear the "2 weeks" part. If they come complaining in 3 weeks I just say "we hit that 10%".
The other important thing is to update estimates. Update people as soon as you realise you hit the 10%. Or in a better case, in a week I might be able to say it's now "1% chance of taking more than a week".
I've found that these incompetent folks (mostly in management as you say) really don't want to reveal their intellectual inferiority; they are desperate to hide it.
If you started using this with your team, everyone adding value would benefit from the clarity, and the incompetent folks will just go along with it so as not to reveal their incompetence.
They aren't adding value anyways, so it's fine if they fall behind.
I think that executives requiring estimates of time from product owners (PMs, Engineering Managers) is an instrument for putting them into de-facto 'debt' servitude, and provides a constant stream of justification for dismissal with cause. As others have commented, if the ability to time perfectly was there, it would no longer have been an innovative product. Same with requiring sales forecasts from salespeople. There's no way for the salesperson to know, so they are constantly on the chopping block for falling short of forecasts they are forced to generate.
I imagine above is more or less tacitly acknowledged in tip-sharing conversations between & among execs & their investors.
A friend told me my post was gaining some momentum on HN and I've read through the comments and found a bunch of good insights.
I especially liked the one from @arach: "this feels like language from another era", which I hope means it's evident my post was written by someone who loves the craft and refrain from prompting an LLM for "a blog post on topic X".
I also try to decide whether I side with the numerous people who want to sort development task into either trivial or novel. Someone wrote that if we break each issue down to its components, the above holds, but idk whether the developers who does not want to estimate are keen on doing such a rigorous breakdown of each feature.
Regardless, super fun to read all the comments. You have already found my blog and feel free to connect in any way as you see fit (or not at all).
One thing frustrating for me is when folk leave $BigCo, with it's methods (ie: estimate time to complete, sprint planning) and try to apply those same methods at a very early company.
Estimates don't work there at all - everything is new.
So, flip it. Use known values to prioritize work. That is: client demand and (potential) revenue. Then allocate known time/budget to the necessary project, see how far you get, iterate. Team can move faster. Looks chaotic.
At some (uncomfortable) point however, need to rotate into the "standard" process.
There’s a well-established Agile technique that in my experience actually succeeds at producing usable estimates.
The PM and team lead write a description of a task, and the whole team reads it together, thinks about it privately, and then votes on its complexity simultaneously using a unitless Fibonacci scale: 1,2,3,5,8,13,21... There's also a 0.5 used for the complexity of literally just fixing a typo.
Because nobody reveals their number until everyone is ready, there's little anchoring, adjustment or conformity bias which are terribly detrimental to estimations.
If the votes cluster tightly, the team settles on the convergent value. If there’s a large spread, the people at the extremes explain their thinking. That’s the real value of the exercise: the outliers surface hidden assumptions, unknowns, and risks. The junior dev might be seeing something the rest of the team missed. That's's great. The team revisits the task with that new information and votes again. The cycle repeats until there’s genuine agreement.
This process works because it forces independent judgment, exposes the model-gap between team members, and prevents anchoring. It’s the only estimation approach I’ve seen that reliably produces numbers the team can stand behind.
It's important that the scores be unitless estimates of complexity, not time. How complex is this task? not How long will this task take?
One team had a rule that if a task had complexity 21, it should be broken down into smaller tasks. And that 8 meant roughly implementing a REST API endpoint of complexity.
A PM can use these complexity estimations + historical team performance to estimate time. The team is happy because they are not responsible for the PM's bad time estimation, and the PM is happy because the numbers are more accurate.
I do like the blind estimation aspect, but I don’t like:
* the arbitrary usage of the Fibonnaci sequence
* a non-clear conversion from complexity to time. Complexity and time aren’t always correlated. Some things can be easy but take a long time. Should that be a 1 or a 5?
Let’s just cut the layer of terminology, in this difficult task, and estimate in units of time directly.
The Fibonacci scale isn’t sacred; it’s just a coarse, non-linear scale that keeps people from pretending to have more precision than they actually have. Any non-linear sequence would work as long as it forces you to acknowledge uncertainty as the numbers grow.
As for “just estimate in time,” the problem is that teams are consistently bad at doing that directly. Mixing “how hard is this” with “how long will it take” collapses two separate variables: intrinsic complexity and local throughput. Story points deliberately avoid that conflation. The team’s velocity is what translates points into time at the sprint level, and that translation only stabilizes if the underlying unit is complexity rather than hours.
The whole point of the method is to strip away the illusion of precision. Time estimates look concrete, but they degrade immediately under uncertainty and project pressure. Relative complexity estimates survive discussion, converge reliably, and don’t invite the fallacy that a complex task with a high risk of surprises somehow has an exact hour count.
That’s why the technique exists. Estimating time directly sounds simpler, but in practice it produces worse forecasts because it hides uncertainty instead of exposing it.
One thing that changed my way of thinking about estimates is reading that 86% of engineering projects, regardless of what kind of engineering (chemical, infrastructure, industrial, etc) go over budget (in time or money).
Missing estimates isn't unique to software, but it's common across all engineering fields.
What's wild about software is we don't do 1/10th as much planning as "real" engineering projects do, and also find ways to create problems akin to the ones they run into from interfacing with an unpredictable real-world environment, by inventing ways for all our platforms and libraries and such to be insanely unstable.
If you're not subject to the batshit insanity of the broader software market, and you crank the planning up a little closer to what "real" engineering does, software delivery gets extremely predictable. See: NASA's Space Shuttle software development processes.
(There are actual, not self-inflicted, problems with software development that "real" engineering doesn't see, though—for one thing, you can't on a whim completely restructure a building that's already been built, and for another, you generally don't have to design real-world engineering projects to defend against the case of intentional sabotage by someone who's just e.g. using the bathroom—it may happen, but it's rare and short of wacky Mission Impossible type plans, busting a water pipe in the bathroom isn't going to get you access to super-secret documents or anything like that)
Physical real-world projects include a buffer for this, called "contingencies" or "change orders" so that if a requirement changes or they discover something like previously unknown site geology that will require changes to the foundation they can absorb it. Based on a large history of similar projects their estimates are usually pretty good but occasionally they will run over.
Even accounting for contingencies most of civil engineering projects go way over budget.
Two elements (the first quite obvious, the second not really) seem to be particularly common in overruns:
- the bigger the project the likelier the overrun. Small road projects tend to be over estimated, complex rail projects are virtually always way underestimated, mega projects are never close to the budget.
- the lengthier the planning and pre-construction phase the likelier the overrun. This is particularly interesting because it's counter intuitive: you would expect that the more analysis is done, the more accurate the estimates, but experience tells us the truth is the very opposite.
I know it's easy internet points to say we should have accurate estimates, but that's not an actionable comment. That's like saying "cars should be safer" and expecting applause.
Most clients would rather you just give an estimate with 15 minutes of eyeballing it and hold you to that. That's not realistic, and it's not on developers.
As a largely solo dev I found I can't estimate well unless it's a common task, and it's easy to find tasks grow exponentially if it touches too many layers.
Asking "how long do you want me to spend on this?" got better results, because I got more idea how important tasks were to the business and can usually tell if something is going to take longer than they want. (Or know when we need to discuss scoping it back, or just abandoning the feature)
It also helps because software is one of the rare engineering fields where code is non-destrictive and can be changed after the fact.
The carpenters trick of measure many times, cut once, can instead just be cut and re-size if wrong size, which can often be quicker.
Asking how long they want me to spend on it also let's me know how solid it needs to be engineered. Is it something that needs doing right then first time, or do we just want something rough that we can refine later with feedback.
I think that that's how you solve the estimate problem - by having an understanding of how many layers it's going to touch, and how big of a change it's going to be for each layer.
Re-planning in isolation is not a failure. I think re-planning is a failure when nothing is learned from previous assumptions not being correct, which usually leads to serial re-planning. I've seen this pattern form at practically every company I've worked for, in particular whenever there's a merger or a private equity buy-out. More often than not, this is ironically caused by leadership failing to learn from previous assumptions because they're too busy chasing the shiny new bobble of the week, but the blame still gets diffused across the organization.
> I think re-planning is a failure when nothing is learned from previous assumptions not being correct
This is the exact problem, and why it is so extremely difficult to communicate about the subject.
Another way to say "learn from previous [failures]" is "bureaucracy". We did something wrong last time, so we need a rule about that failure.
It SHOULD NOT be seen as failure, we SHOULD NOT add bureaucracy, simple rules CAN NOT fix the issue.
What should happen: Line managers and developers should be able to communicate about what they are learning as they do the work of software engineering. Developers should be able to re-plan their approach to problem solutions.
This simply will not work if there is not strong trust between upper management/[the money] and the engineering/execution side of the organization.
----
The meta-problem is that you have to see things go wrong many times in many ways before you really understand what is going on, and by that point ... well everything gets very tiresome.
Re-planning when you gain enough information to know you need to isn’t a failure — not re-planning then would be — but repeatedly finding yourself re-planning when you had or should have had that information before the initial plan is.
I've always seen estimates as trying to guess the highest number the PO will accept, the time or effort involved in actually completing the task is irrelevant. I have never had a PO or anyone else complain that a task was completed more quickly than expected. However I do have to be careful to not tell them it is complete too early, lest they start expecting shorter cycles.
At least in my company we've stopped calling them "estimates". They are deadlines, which everyone has always treated "estimates" as anyway.
Unfortunately in the real world deadlines are necessary. The customer is not just mad that they didn't get the shiny new thing, especially in the case of B2B stuff, the customer is implementing plans and projects based on the availability of X feature on Y date. Back to the initial point, these deadlines often come down to how quickly the customer is going to be able to implement their end of the solution, if they aren't going to be ready to use the feature for six months there's no reason for us to bust our asses trying to get it out in a week.
I can't say I know many engineers who object outright to deadlines. They just get frustrated when an estimate turns into a deadline as those are different questions.
The far more common pattern is being asked to provide such an estimate off hand and those are all about what you mentioned, giving the PM whatever number you think they will accept.
In the 70's a Brazilian programmer told me his method was to make his best guess how long something would take, double it, then promise that amount plus-or-minus 50%.
The thing that sucks is that when I avoid giving estimates, I'm not trying to be difficult, I'm being honest about the unknowns of the project and the inherent uncertainties and messiness of software development. I'm helping protect myself and the rest of the team from making plans based off of bad estimates.
But I get all this pushback when I do that, such that the path of least resistance is to give some bullshit estimate anyway. Or I get asked to make a "rough guesstimate", which inevitably turns itself into some sort of deadline anyway.
Garbage in, garbage out. Inaccurate estimates, unreasonable timelines, stressed devs and upset PMs.
A couple of decades back PMs used to look at historical data to guide the estimates for a new project. If a similar coding work took 2 weeks on average in the past, that gives some basis.
So, I think the issue is about whether it is a routine workflow work which has well-tested historical timelines or not.
Nevertheless, estimates are needed at some granularity level. When you order something on Amazon, would like an estimate on when the item would be delivered to you?
Even if coding work can't be estimated, the overall project requires estimation. Someone need to commit to timelines and come under pressure. Distribution of that pressure is only fair.
> A couple of decades back PMs used to look at historical data to guide the estimates for a new project. If a similar coding work took 2 weeks on average in the past, that gives some basis.
Even that doesn’t work because the time taken isn’t just about similarity to other work, it’s about how your new feature interacts with the current state of the codebase which is not the same as when the similar feature was implemented before.
Ultimately, it’s a complexity problem that’s borderline impossible for our feeble human brains to properly understand. And we consistently misunderstand the nature of that complexity
An aircraft engine could be highly complex, but the flight times can be estimated. Complexity is not issue. Reproducibility, and similarity is missing. Every coding work became an adventure into unknown landscape, with no trekking map.
And that cavalier attitude to complexity is literally the cause of so many issues with software engineering. An aircraft engine is almost simple compared to a multimillion line codebase.
The attitude with belief that software work cannot be compared with any other work in other domains, is not going to help. There are far more complex projects out there. Complexity is not feature limited to the work artifacts. It could arise from overall context - funding, worker issues, climate, raw material, political factors, technical issues. But still, estimates can't be avoided.
> The attitude with belief that software work cannot be compared with any other work in other domains, is not going to help.
I didn't say that, at all. But it is an extreme form of abstract complexity which is different to the more tangible and physical complexity that you might have in say a jet engine. That is not to say a jet engine isn't complex, it is and obviously so; but in many ways I think most people don't fully understand the extreme complexity of a medium to large software project because, on the surface, we've apparently captured a lot of it behind reusable libraries and the like. The reality is much more nuanced due to the fact that most software engineering is not following a technique of provably sound composition of components and therefore complexity emerges and is a sum of its parts. That summing of complexity creeps up on a development team and emerges as bugs, security issues, difficulty in planning, and difficulty in maintenance, etc.
My singular argument is: reduce complexity, improve predictability.
> But still, estimates can't be avoided.
I also didn't state that estimates can be avoided. I ran a tech company for 20 years and I needed to be able to tell everyone else in the company when things are likely to be done. That's just a fact of life.
Businesses have many aspects of their operation that are unpredictable: ask the legal team exactly what the result of a legal action will be, and when it will be completed by, and you'll get a fuzzy answer. Ask the marketing team exactly how many new signups will result from this new ad campaign and you'll get a fuzzy answer. Businesses can easily cope with unpredictability, this is not a problem.
The problem is that software development has not been treated as one of these unpredictable operations. It absolutely is unpredictable, we know this from decades of experience and all the academic research on the subject. But, probably because of history, politics and power imbalances, software dev teams have very rarely managed to convince executive teams that software development must be treated as unpredictable. As a result, software dev as a profession has a bad reputation for not delivering on their promises.
When we do manage to convince executive teams that software development is unpredictable, good things happen. The author's point that "tax software must be released at tax time" still happens, but the planning around the specific feature set takes into account that if the release date is fixed then the feature set in the release will not be. Again, it's not a problem to deal with the unpredictability, as long as everyone accepts that it is unpredictable.
I've found the best estimates are not estimates but non negotiable deadlines with fixed outcome.
"You have six weeks to do X for $$$" or "I'll get it done in six weeks or you don't pay"
Where i work there is no penalty for being late or not hitting a deadline. Life goes on and work continues. I have seen when there are specific dates and metrics and suddenly people work in focused effort and sometimes work weekends or celebrate with finishing early.
I have never been in a project, where estimates were spot-on, and I do this for 15 years now. By now hundreds of features have floated by the river and hundreds of meetings have been held.
Estimations are often complicated, because there are way too many variables involved to give accurate estimates. Politics within companies, restructuring of teams. The customer changes their mind, the reality you've expected is slightly different, your architecture has shortcomings you find out late in a project, the teams your work depends on disband, … and a million other things.
Theoretically you could update your estimates in a SCRUM meeting, sure, but to be honest, this has always been nothing but a fantasy. We rarely do work in a void. Our features have been communicated higher up and have already been pitched to customers. In a fully transparent and open organization you might update your estimates, and try to explain this to your customers. In reality though? I have never seen this.
While this sounds very negative, my take on it is not to waste too much time on estimates. Give a range of time you expect your features to fall into, and go on with getting your work done.
I wonder how different the perception of these projects being late or (massively) over budget would be if we used different words. Bear with me here...
Words mean things. Estimate carries a certain weight. It's almost scientific sounding. Instead, we should use the word "guess".
It's exactly equivalent, but imagine the outcome if everyone in the chain, from the very serious people involved in thinking up the project, to funding the project, to prioritising and then delivering the project, all used the word "guess"
Now, when the project is millions of dollars over budget and many months/years late, no one is under any pretence that it was going to be anything else.
I tried this once. It turns out serious people don't like the idea of spending millions of dollars based on "guessing", or even letting developers "play" in order to better understand the guesses they are forced to make, even when it turns un-educated guesses into educated guesses.
Of course, none of this would improve the outcome, but at least it sets expectations appropriately.
Estimates are difficult, but if you monitor your estimate vs actual, and then adjust your next estimate accordingly, then it becomes easier to have decent estimates.
There is also the adage that if you are late, then 100% of the time the customer will be unhappy, but if you are early, then 100% of the time the customer will be pleased.
I think the main issue with time estimates is that they don't follow a normal distribution (like many things we are taught in school) but log-normal distribution, which is very skewed and has a big right tail. This misconception is the reason people fail to understand why time estimates are inherently hard.
All estimates should be accompanied by a "cone of uncertainty". Teams should evolve their ability to describe a cone of uncertainty as well as their ability to estimate. Estimating "accurately" is only possible when understanding the accuracy is elevated to the same importance as the estimate itself.
I don’t agree with the given answer to “Why the PO wants you to estimate”. The author cites external reasons (made commitments etc) that shouldn’t have been there in the first place.
Sure, in some lines of software development you really have a “deadline”, if you are working towards some event or launch date. But most of the time estimates are needed to determine *if it is worth it*. Can it be built in 2 weeks, let’s go. Is it gonna cost 2 months, then let’s prioritise something else. That isn’t to say you always have to go with the shortest te build features (you can apply "CD3" – Cost of Delay Divided by Duration), but you need it to compare alternatives.
I've always found this sound, rational ROI driven approach to product management a little off the mark. Software isn't like real estate or investing in TBills - you don't invest $X in development and get a nice 10% annualized return on investment over the next 10 years or something like that despite how seductive such thinking can be.
It is largely a "hits" business where 1% of the activities that you do result in 99% of the revenues. The returns are non-linear so there should be almost no focus on the input estimation. If your feature only makes sense if it can be done in 3 months but doesn't make economic sense if it takes > 6 months - delete feature.
30 years ago my boss at a large defense/aviation contractor told me estimating software projects was a very valuable skill, but all estimates were always wrong because they are simplifications and to keep that in mind -- his words.
Mainly they are useful to build belief and keep a direction towards the goal.
Models of any kind in whatever domain are necessarily always something less than reality. That is both their value and weakness.
So estimates are models. Less than reality. Therefore we should not expect them to be useful beyond 'plans are useless, but planning is indispensable' -- I think thats' Eisenhower.
I literally filled out TPS reports every week at this place down to the hour, breaking down internal project billing and customer billing. In order get the charge codes from the PM, you need to give them estimates. That was the main driver for estimates - convince the non-coders.
It's also difficult for LLMs it seems. If I forget to add instructions to skip resource estimates, Claude will estimate a week or two, then bang it out in under an hour.
I tried to search for the CMMI keyword in the whole discussion to find something relatable. Guess like everything else people happen to loose memory over a cycle and discover it again as if thats something new. Another thing I realize from this is that I am probably getting older.
We know _why_ you want to know how long everything will take. It seems to escape these guy's attention that if software development was as predictable as they seem to insist it must be, they could be replaced with a spreadsheet.
I've tried running teams with and without estimates and I've noticed that when work isn't estimated it tends to drag on forever. But when you put a number on it, even if it's a rough or inaccurate guess, it gets done much faster.
Giving an estimate is not hard and shouldn't be a pain if if was truly an "estimate" PMs were asking for. But after a while you learn that when they say "What's the estimate for feature F?", they're in fact saying "What deadline you wish to impose on yourself for feature F?". Then the next interaction is most likely to be "Are you done yet?" and not "Is the estimate holding up?".
I don’t think I dislike being forced to estimate. I dislike being asked to estimate by the same people that consistently make those estimates worthless by introducing changes at literally every point in the process.
My experience is that I have to basically always overestimate if I can get away with it because otherwise if something goes wrong, I will pushed to do free overtime to complete all the work assigned in a given sprint.
It is often not just about the difficulty of estimation the time for specific tasks but also what the assumptions put in were. Sometimes these assumptions werent written out explicitly or other times they were (deliberately?) removed. Just one example of a broken example: a project can be run in many ways. You can have an estimate done based on A-team resources and high priority but the moment the contract or whatever is done, it is decided to outside the whole work to a new team who never worked on the code before and sit thousands of miles away. To compensate 2-3x as many people are assigned. Add in a non technical project manager and scrum master and all kinds of resources that were never envisaged but who will report time in on the project etc. You get the idea. And this was just done type of assumption that could broken!
I have a solution but I don't think companies care about this level of meta-analysis on how people work together. They think they do but in reality they just care about optics, and the status quo culture has a huge weight on continuing in the same direction, largely dictated by industry "standards".
In essence, estimates are useless. There should only be deadlines and the confidence of engineers of achieving the deadline. To the extent there are estimates, it should be an external observation on the part of PMs and POs based not only on the past but also on knowledge of how each team member performs. This of course only works if engineers are ONLY focusing on technical tasks, not creating tickets or doing planning. The main point of failure in an abstract sense is making people estimate or analyze their own work, which comes with a bias. This bias needs to be eliminated and at the same time you give engineers the opportunity to optimize their workflows and maximize their output.
TLDR, engineers should only focus on strictly technical because it allows to optimize within the domain, meanwhile other roles (whoever PM, PO or other) should be creating tasks, and estimating. Of course this doesn't work because there are hard biases in the industry they are hard to break.
"Evidence-based" stats tracking is the only somewhat scientific method i know of that can give you a good range of estimates. But it takes effort. Thus why many don't do it.
But accurate estimate ranges are a super-power for businesses if they can trust them. Never understood why that's not demanded more... oh yeah, it's work :p
estimates are only difficult cause entire industry has convinced everyone that we do something special which we have no idea how long it will take to do :)
Estimates are like traveling from point A to point B in completely unpredictable terrain. Sometimes you can just go straight and beat the estimate easy. Sometimes you will find massive valley and you need to build a bridge to continue while your PM is furiously yelling in the radio that estimated deadline was a week ago.
I will often estimate 3x as a rule of thumb. Estimating is indeed difficult especially when dealing with unforeseen task switching.
I was surprised to not see “story points” mentioned in the context of scrum. This is an estimation concept I have found baffling because it is a measure of “complexity”, not time, yet is used directly to estimate how much can be done. At least this is how it is done at my work.
The unique thing about estimates in software engineering is that if you do it right, projects should be impossible to estimate!
Tasks that are easiest to estimate are tasks that are predictable, and repetitive. If I ask you how long it'll take to add a new database field, and you've added a new database field 100s of times in the past and each time they take 1 day, your estimate for it is going to be very spot-on.
But in the software world, predictable and repetitive tasks are also the kinds of tasks that are most easily automated, which means the time it takes to perform those tasks should asymptotically approach 0.
But if the predictable tasks take 0 time, how long a project takes will be dominated by the novel, unpredictable parts.
That's why software estimates are very hard to do.
And I'd add that the need for them is a sign they aren't worth doing.
As you say, worthwhile software is usually novel. And to justify our expense, it needs to be valuable. So to decide whether a project is worth doing, we're looking at some sort of estimate of return on investment.
That estimate will also, at least implicitly, have a range. That range is determined by both the I and the R. If you don't have a precise estimate of return, making your estimate of investment more precise doesn't help anything. And I've never seen an estimate of return both precise and accurate; business is even less certain than software.
In my opinion, effort put into careful estimates is almost always better put into early, iterative delivery and product management that maximizes the information gained. Shipping early and often buys much clearer information on both I and R than you can ever get in a conference room.
Of course all of this only matters if running an effective business is more important than managerial soap opera and office politics. Those often require estimates in much the same way they're required from Star Trek's engineers: so the people with main character syndrome have something to dramatically ignore or override to prove their dominance over the NPCs and material reality.
The typical reason is that developers in large teams often don't know all the places where a field may be used - and their IDE could very well miss some. So automated testing or QA finds an issue, and then the unaccounted-for, time-consuming bugtracking begins.
CRUD: Create, Read, Update, Delete. Basically just inputting and displaying data from a database.
> As you say, worthwhile software is usually novel.
This is an interesting assumption. I’d argue that the overwhelming majority of software is the most boring LoB CRUD apps you can imagine, and not novel at all. Yet, people need to estimate the tasks on these projects as well.
And starting in the late 1970s, there were tools available to simplify building LoB CRUD apps. [1] That has continued with things like Rails and Salesforce and no-code tooling.
If something is truly boring in software, it gets turned into a library or a tool for non-programmers to use. Our value is always driven by the novelty of the need.
And no, people don't need to estimate the tasks. My dad did LoB apps in the 1970s to the 1990s. E.g., order entry and shop floor management systems for office furniture factories. His approach was to get something basic working, see how it worked for the users, and then iteratively improve things until they'd created enough business advantage and/or cost savings to move on. Exploratory, iterative work like that can at best be done with broad ballpark estimates.
I grant that people want estimates. But that is usually about managerial fear of waste and/or need for control. But I think there are better ways to solve those problems.
[1] e.g., https://en.wikipedia.org/wiki/DBase
All of that is besides the point. People need to estimate their tasks if their managers want them to, and no amount of philosophical navel-gazing will change that.
I want to be clear that I am being entirely practical here. This is not navel-gazing. I am describing something that works. That has worked for me and others for decades.
And yes, if you are in an environment where people with power want things, you have to take that into account.
But no, we don't have to just blindly do what people with power ask. The difference between being a professional an a minion is that professionals know much more about the work and how to do it than the people paying them. Personally, I think we are professionals, which gives us a variety of responsibilities not just to the person with the money, but to the profession and society at large.
Does that mean I never have given estimates? Not at all. But it does mean that if somebody asks me to do things in a way I think suboptimal, I'm at least going to say, "Hey, there's a better way to satisfy your goals here." And then I'm going to help them learn enough about the better way that they're at least willing to try it.
Because "the manager says so" or because "estimates actually add some value"?
I think it's important that our "work culture" allows us to critique why we do certain tasks. When push comes to shove, I guess a manager can say: "Because I say so!", but I also hope those kind of managers are few and far between.
Both, kind of. The demand to have at least a rough estimate when something will be available is justified IMHO—other departments obviously need to maintain their own timelines and schedule work that depends on output from engineering.
Also, I wholeheartedly agree that we do need to question the work culture we follow and the measures we make, and that managers with control issues shouldn't dictate them.
On the other hand, the point I was getting to is that a critique of estimation that amounts to "the work I do is so bespoke and unique and novel and important that I can't be bothered to estimate how long it'll take", is just… ignorant. Most software engineers are not lone wolf 10X wizards without any accountability, have managers and other departments to report to, and thus are not eligible to make that point.
> the work I do is so bespoke and unique and novel and important that I can't be bothered to estimate how long it'll take
This absolutely can be the case some of the time though. I've never pressed back on estimates of standard work but it can be a real bastard to have to work within the "process" when you are slaying a truly novel beast. Having some jackass pestering you for updates on how long it takes to climb the beanstalk and find the golden harp is just too much.
This is a gross and misleading caricature of what I'm saying. I prefer this approach precisely because it increases accountability. If you'd like to learn what I'm actually suggesting, I'm happy to answer questions. Or you can read many of the things that have been written by other people on the topic.
But it's doing something novel, something the same people haven't done before, otherwise there would be no point in writing it.
Sure you can move the goalposts here, but OP clearly meant to say software tasks cannot be estimated because people only work on novel problems, since everything else is "not worth doing" (what a massively privileged thing to say by the way).
Just because something hasn't been done the exact same way you're doing, that doesn't mean you can't apply a generic solution. I have never changed a tyre on an SUV before, yet I do know how to do so based on my previous experience with a sedan. The same applies to a car mechanic; even if I bring a brand new car to the workshop they have never seen before, I can and should expect them to be able to (at least roughly) estimate how long a tyre change is going to take.
< “Those often require estimates in much the same way they're required from Star Trek's engineers: so the people with main character syndrome have something to dramatically ignore or override to prove their dominance over the NPCs and material reality.”
This is so good.
Thanks. It was hard won. I spent maybe a decade naively thinking that if we just made software methods that worked in service of stated business goals and values, they'd get adopted and we'd all live happily ever after.
It took me a long time to come to grips with the POSIWID [1] version of the purpose of planning and estimates. One of the things that really blew my mind is Mary Poppendieck's story about how they built the Empire State Building on time and under budget even though they didn't have it fully designed when they started. [2] Different, more effective approaches are not only possible, they exist. But they can no longer win out, and I think it's because of the rise of managerialism, the current dominant ideology and culture of big business. [3]
[1] https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_wha...
[2] Talk: https://www.infoq.com/presentations/tyranny-of-plan/ And transcript: https://web.archive.org/web/20140311004931/https://chrisgagn...
[3] See, e.g., https://www.amazon.com/Confronting-Managerialism-Business-Ec...
Thanks for the links. To the limit of my influence I try to protect my team from distractions, be fluid about methodology (constant agile churn can be depressing), limit the toxicity of pull requests, and to spend as much time with them as I can. A happy team is a productive team. Oh and I try not to work with leaders who obsess over Gantt charts. To me estimates are more about trust and respect rather than metrics and velocity. It has to be the right kind of company though.
Further: In The Next Generation, when Scotty shows up, he mentions to Geordi he anyways padded his estimates because he knew Kirk would do things like that.
And that's such a good summary of the toxic dynamic around estimates!
The solution you described is basically agile, and that definitely includes estimates and deadlines.
There are agile methods that forgo estimates and deadlines though
This is what "agile" is: https://agilemanifesto.org/
More specific methodologies that say they are agile may use concepts like estimates (story points or time or whatever), but even with Scrum I've never run into a Scrum-imposed "deadline". In Scrum the sprint ends, yes, but sprints often end without hitting all the sprint goals and that, in conjunction with whatever you were able to deliver, just informs your backlog for the next sprint.
Real "hard" deadlines are usually imposed by the business stakeholders. But with agile methods the thing they try to do most of all isn't manage deadlines, but maximize pace at which you can understand and solve a relevant business problem. That can often be just as well done by iteratively shipping and adjusting at high velocity, but without a lot of time spent on estimates or calendar management.
Yes, people keep linking to the agile manifesto as if it's some sort of amulet protecting software developers from any sort of accountability or responsibility for their work product in a professional setting.
It seems like you acknowledge some amount of estimating is needed and I agree that there is an overemphasis on estimation in many places, but I'll ask you the same thing I asked others, which is:
How do you do either of the following without spending any time at all on estimates?
"Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale."
"At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly."
I addressed the rest elsewhere, but done well a lack of estimates makes people more accountable. If I am shipping something every week (or as is common for my teams, more often), stakeholders can directly evaluate progress. There's no snowing them with paperwork and claims of progress against semi-fictional documents. They see what they see, they try it out, they watch people use it.
The reality of use is what we in software are ultimately accountable to, and that's what I am suggesting people optimize for. Closing that feedback loop early and often builds stakeholder trust such that they stop asking for elaborate fantasy plans and estimates.
You replied to me in like 10 different places, nearly all of which are in responses to posts that weren't directed at you, so I'm trying not to fragment this discussion too much.
I will ask this here: If you are shipping code to production on a weekly basis, is that not a schedule, also known as a deadline for delivery?
If you expect to ship code to production every week, how do you know whether there will be something to ship without doing any estimation of the effort and time required?
It is not a schedule, it's a standard. One I normally try to exceed. We ship when things are ready, which for my current team is ~2-3x/week, but in the past I've had teams that were faster.
We know that there will be things to ship because we try to break the work down into small units of deliverable value by focusing on seeking the highest-value things to do. Large requests are typically composed of a bunch of things of varying value, so we split them until we find things that advance the needs of the user, the customer, or the business. One that's often not intuitive to people is the business need for getting information about what's truly valuable to some part of the audience. So we'll often ship a small thing for a particular audience and see how they react and what actually gets used. (You can save an amazing amount of time by not building the things nobody would end up using.)
Sometimes we can't figure out how to break down something smaller enough that we have something to release right away. Or sometimes a chunk of work surprises us and it drags out. We avoid that, because compared to working on smaller things, it's much less comfortable for both developers and business stakeholders. But if it happens, it happens. We try to learn from it for the next time.
Regarding deadlines, we sometimes have them. Broadly, efforts break down into two categories: driven by date or driven by need. For the former, releasing early and often means we adjust scope to hit the date. For the latter, scope is primary and they get stuff when they get it. Either way because the business side sees steady improvement and has fine-grained control over what gets shipped, they feel in control.
This can be a learning experience for business stakeholders used to waterfall-ish, plan-driven approaches. But I have never once had somebody successfully work this way and want to go back. I have, however, had some product managers get thrown back into document-driven development and tell me how much they missed working like we did.
No, shipping code to production on a weekly basis is not a deadline. A deadline is a time by which a task must be completed. A task is something like "fix bug 3831" or "allow users to log in with OAuth2". "Ship code" is not, in any useful sense, a task.
Such "timeboxed iterations" can indeed result in "shipping" a null update. Unless you have a time-consuming QA gate to pass, that's not very likely, especially on a team containing several people, but it can happen. You don't know that you will have "something" to ship.
Typically we try to break changes down into shippable tasks that can be done in under a day, so the expected number of tasks completed by a four-programmer team in a week is on the order of 30, or 15 if you're pairing. For this to fall all the way to 0, everybody has to be spending their time on things that could not be thus broken down. It's pretty unlikely to happen by chance. But sometimes a difficult bug or two really is the thing to focus on.
In XP, estimates are used for prioritizing which tasks to work on and which tasks to break down into smaller tasks. The "product owner" is supposed to choose the tasks that have the most user value for the estimated cost. But those estimates aren't commitments in any sense; they're guesses. Sometimes tasks take more time than estimated; other times, they take less. This is the reason for the shift to estimating in "story points": to prevent the estimates from being interpreted as referring to a period of time.
If someone in your organization is interpreting estimates as commitments, this can have a corrosive effect on the accuracy of those estimates, because estimators respond by padding their estimates, in inconsistent ways. Often this destroys the usefulness of the whole iteration planning process, because the project owner no longer knows which tasks are the easiest ones and thus worth doing even if the benefit is marginal. Organizations can recover from this pathology in a variety of ways, but often they don't. Eliminating estimation is one response that sometimes works.
Yes, this sounds very familiar to me. I started with dates and went to story points used to estimate dates. Then as we turned up the release cadence, we eventually dropped estimating altogether, even in points, because it wasn't really helping anything.
That doesn't mean we refuse to solve the problems that estimates are used to solve. E.g., "Can we have something by the big conference," or "Can we keep our customers happy." We just solve them other ways.
And totally agreed about the corrosive effect of treating estimates as commitments. It's such a negative-sum game, but people play it all the time.
I already said this to your fellow interlocutor who is also responding to nearly every comment of mine with the same thought process, but I'll say it here as well in different terms:
The product owners, customers, salespeople, supervisors, peers, etc. you interact with as part of the software development process on any project outside of a personal hobby don't care about your semantic games.
If functionality is needed in an application, and they ask you to implement it, and you agree, there is no real-world scenario where they just say "Cool, I'll sit idly by while you work at this until you declare it ready, and then and only then will I let anyone else know about it or take any action on its supposed existence," and repeat that for every piece of functionality you implement in perpetuity.
And if you keep failing to deliver required functionality over time, no one is going to accept your arguments that: "Oh sorry, our weekly deliveries to production aren't a deadline, it's a timeboxed iteration", "Oh that estimate wasn't a commitment to do anything, we work on our own schedule", and so on.
Yes, the relationship between developers and "other stakeholders" can turn toxic, but in most organizations the developers don't have much power, probably due to repeated attempts to play the games you've laid out above. The way to combat that is to be reliable and professional so your team has the authority to stand their ground on the difficulty of a given task, not effectively refuse to participate in what is a completely reasonable conversation about the relationship between your work and the objectives of the organization.
I have to keep responding to your posts because you keep grossly mischaracterizing what I'm saying. I have great relationships with my stakeholders. Just this we we delivered something greatly desired by the organization. We are being reliable and professional, just in a way that apparently confuses and frightens you and others. Indeed, I'd claim I'm being more professional that people who give nonsense estimates just because bosses ask for them.
Do you mean "just this week"?
> How do you do either of the following without spending any time at all on estimates?
> "Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale."
This is 'just' bum-standard continuous delivery (which is where most organizations should be heading). You pull the next todo from the backlog, start working on it. If it takes more than a day to commit something, you split the task into something smaller.
You don't need to estimate ahead of time at all as long as the task is small enough, all you need is to be able to put the near-term backlog of work into a good priority order of business value.
If the high-value task was small it doesn't prevent you from doing more work, because the next unit of work to do is the same either way (the next item on the backlog).
If the high-value task was too big, it can cause you to take a pause to reflect on whether you scoped the task properly and if it is still high-value, but an estimate wouldn't have saved you from it because if you'd truly understood the work ahead of time you wouldn't be pausing to reflect. An estimate, had you performed it, would not have changed the priority.
But this Kanban-style process can be performed without estimates at all, and organizations that work to setup an appropriate context for this will find that they get faster delivery of value than trying to shoehorn delivery into prior estimates instead. But there are people who work faster with the fire of a deadline under their tail so I can't say it's unilaterally better.
> "At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly."
If it's hard to do the work as a team, you should be able to tell it was hard retrospectively, with or without having done an estimate ahead of time.
You might say that failing to hit your prior schedule estimates would be a good topic to discuss at a retrospective session, but I would tell you that this is a self-licking ice cream cone. If your customers are happy despite missing internal schedule estimates you're in a good spot, and if your customers are unhappy even though you're "hitting schedule projections" you're in a bad spot.
There's a lot more productive discussions to be done when the team reflects on how things are going, and they typically relate to identifying and addressing obstacles to the continuous flow of value from the product team to the end users.
It’s Agile philosophically, and how it should be.
But that is rarely how it works. In the dozens of different projects across ten or twelve companies I’ve had insight into, “doing Agile” is analogous with “we have a scrum master, hold stand ups, and schedule iterations” while the simple reality is “Agilefall.”
Agreed, but the parent poster said that estimates shouldn't be done at all, which is not a legitimate argument to make in any scenario.
I have had many successful projects where we spent approximately zero time on estimates. The fact that a successful approach is culturally seen as illegitimate to even talk about is a great example of why I wrote that last paragraph.
Whenever there are constraints (money, time, resources) there are going to be estimates and prioritisation.
You might be speaking a little more broadly than I am interpreting.
On the contrary, constraints often mean you don't need formal estimates. (I'll come back to prioritization in a sec.)
Startups are a great example. When you raise your first chunk of money, the size of that isn't really driven by a carefully considered, detailed plan with engineering hours estimated per task. What you get is basically determined what's currently fashionable among angels and small-end VCs, plus who's doing your fundraising. (If you're Jeffery Katzenberg and Meg Whitman, you can raise $1bn. [1] https://en.wikipedia.org/wiki/Quibi But the rest of us have to make do with what we can get.)
So at that point you have a strong constraint (whatever you raised) and some relatively clear goal. As I said, cost isn't nearly as relevant as ROI, and nobody can put real numbers on the R in a startup. At that point you have two choices.
One is just to build to whatever the CEO (or some set of HiPPOs wants). Then you launch and find out whether or not you're fucked. The other is to take something akin to the Lean Startup approach, where you iteratively chase your goal, testing product and marketing hypotheses by shipping early and often.
In that later context, are people making intuitive ROI judgments? Absolutely. Every thing you try has people doing what you could casually call estimating. But does that require an estimation practice, where engineers carefully examine the work and produce numbers? Not at all. Again, I've done it many times. Especially in a startup context, the effort required for estimation is much better put into maximizing learning per unit of spending.
And how do you do that? Relentless prioritization. I was once part of a team that was so good at it that they launched with no login system. Initial users just typed their names in text fields. They wanted proper auth, and eventually they built it, but for demonstrating traction and raising money there were higher priorities. It worked out for them; they built up to have millions of users and were eventually acquired for tens of millions. On very little investor money.
Being great at prioritization makes estimation way less necessary. The units of work get small enough that the law of large numbers is on your side. And the amount of learning gained from the things released change both the R and I numbers frequently enough that formal estimates don't have a long shelf life.
So I get what you're saying in theory, but I'm telling you in practice it's different.
[1] https://en.wikipedia.org/wiki/Quibi
Wait, your first example is "We raised X dollars" which is a literal estimate of the worth of the company?
I think you are well missing the point - everything you put into your rebuttal is about estimates - in time, money, or resources
If you're saying there's some sort of wisps-and-moonbeams notion of estimation in everything that we do, sure. I'm not going to argue with that. One world, brah.
What I am talking about here, though, is a practice of software estimation where programmers produce detailed numbers on the amount of time requested work will take. Which is certainly the common meaning of estimating around here, and also what the original article is about.
"Brah"
I mean, if this is upsetting you, take a minute and go for a walk.
> You might be speaking a little more broadly than I am interpreting.
(Or I could be interpreting it more broadly)
In either case I don't think your response is apt
I'm not upset. But as far as I can tell you've wandered off into some sort of philosophical space when I'm speaking of practicalities, without apparent recognition of the transition. I thought that was a bit ridiculous, something from the "wow dude have you ever really looked at your hand" school of conversation, so I made a joke. Apparently the joke didn't land for you. Alas.
Research is subject to constraints of money, time, and resources, but is not normally estimated in the sense that software industry people would use the term.
Yes, yet estimates are still made. The author of the article didn't use some highly formal definition of estimation, didn't imply one, and seems to be focused on devops (not software development) as a practitioner.
Estimates are difficult, and in unhealthy environments are weaponized against developers. That doesn't mean they're unnecessary or impossible.
I think that the replies I am getting are demonstrating why developers have estimates used against them - people forget that they are estimates, and they also forget that when new information comes to hand that invalidates that estimate a completely new one may need to be created to take into account the new data.
If developers (or anyone giving estimates) discovers that the initial estimate was based on faulty information then they need to push that information back to whomever they are reporting it to (Team Lead, Product Owner, Manager, customer, angel investor...). The receiver of that information then needs to decide on how to react according to the changes.
Yes, agile is a reaction to spreadsheet driven development and some very dumb ways of tracking progress towards completion and managing work in general.
In my experience, people don't forget they're estimates, they just want to force developers to meet whatever they agreed to that's most convenient for management.
If you want to fight back against that, my experience has been that giving terrible estimates or refusing to give them at all will not result in more autonomy or authority.
> If you want to fight back against that, my experience has been that giving terrible estimates or refusing to give them at all will not result in more autonomy or authority.
In my experience giving terrible estimates or refusing to give them at all is the least bad course of action. It wastes less of your time than any realistic alternative, it does no noticeable damage to the business or your own position, and the people who want to paint you as just trying to avoid accountability were going to find a way to do that anyway.
Research is estimated, sometimes those estimates are hilariously bad (Computer vision is easy, a summer research for a student should be enough), but more often than not it's "We expect that this research will take someone doing a Ph.D approximately 3 - 5 years to do"
The entire premise of a project is "Look at this, with the intent to find X, and, if it's not possible, break it down so that we can create more projects to work toward that goal" which is an estimate, or a breakdown into sub projects that also come with estimates.
I, and others, don't agree with the blanket statement that "no estimates" is not a legitimate argument in any scenario. Can you expand on why you think there isn't a single case where estimates don't add value? Similarly, is there anything specifically in that post's claims that you think was incorrect, leading to their false conclusion?
Okay, a scenario where you're building a hobby project alone and you don't care if or when it gets finished would be one where estimates aren't needed.
There is no scenario where it's appropriate or necessary when developing software professionally or even as a side project where others are expecting you to complete work at some point.
One of the many misconceptions in the original comment in this thread is that "worthwhile software is usually novel", which is not the case without a very specific and novel definition of worthwhile that I don't believe was intended.
If software isn't novel, that means some other, existing software does the same thing just as well in the same way on the same platform. So, unless it's a hobby project you're building alone, why don't you just use the existing software?
I think that writing software that isn't novel fails to be worthwhile by a perfectly ordinary, mainstream definition of "worthwhile".
So you would consider a CRUD app with some basic business rules to be novel? Basically meaning that any software that requires any development effort is novel?
That's a completely valid definition of worthwhile software, but to claim it's impossible to create an estimate to complete said development is absurd.
You just keep saying things are absurd or obvious but not putting anything behind it.
I hope this isn't a semantics game where things like "1 - 6 months" counts as an estimate in this context.
The point way back up this thread was accurate timelines for complicated, novel work have large error bars but those error bars aren't as bad as the equivalent error bars on estimating whatever "return" it is being pitted against.
I wouldn't consider something like "1-6 months" as a valid estimate, as that would indicate there is too much uncertainty and it needs to be broken down into subtasks that can be estimated with much less variance.
I've written what is probably several pages now in response to two individuals who are redefining terms in order to play the exact semantic games you mentioned, but in order to claim no estimation of any sort needs to be done. We seem to be done talking past each other now that I explicitly pointed out their usage of non-standard terms and my suspicions of why (having also unfortunately lived through software development managed by Gantt chart and other unpleasant experiences where someone who had no idea what they were managing was in control of a project), which is fine with me.
Feel free to describe your experience in practice when working in an organization where software developers answer to no one but themselves and are never asked for any justification for their progress or any projections of when they will be finished (both of which would require estimation to provide).
If you are able to tell stakeholders something like you'll be done in 1-6 months or provide no insight at all into when your tasking will be done, do no tracking of progress internally, and perform no collaboration around the completion of synchronous tasks within your team, I'll acknowledge no estimation is taking place during that process.
I can say with some confidence, having been involved in the movement since before the term "Agile" was coined, that it requires neither.
I grant that both of those are common, but that's because the median "Agile" implementation quickly devolved into mini-Waterfall with more hip names.
I missed that part of the manifesto
Right? For those who want to check, the four values: https://agilemanifesto.org/
And the 12 principles: https://agilemanifesto.org/principles.html
But to my dismay, the Agile world quickly got colonized by certification schemes and consultants targeting large companies, so it rapidly turned into something that a lot of the early people were very disappointed with. I wrote about the dynamic some years back: https://williampietri.com/writing/2011/agiles-second-chasm-a...
How do you do either of the following without spending any time at all on estimates?
"Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale."
"At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly."
If I was your colleague, had invested in your company, or was considering giving you money to deliver some product and you refused to tell me when you think it'll be done, it would be the last time any of the above ever happened.
You wouldn't accept that from any service you were attempting to hire, so I don't know why it would be acceptable for software development.
I feel like I've already answered the how question elsewhere. But the short version is a product-driven kanban-ish approach with frequent releases, small units of work, and engaged product management focused on actual real-world goals. This approach originated 25 years ago in Kent Beck's "Extreme Programming". Which did start with estimates, but teams doing well at it quickly realized (via said regular reflection) that they could do without it and still deliver well.
That some random person on the internet who wouldn't have hired me anyhow will now not hire me is a blow I'll have to learn to live with. But I have literal decades of stakeholders happy working this way. One way that happens is that they pick a date and then we built what fits between now and then. So they get the date, but no promises about what will get delivered.
In practice people like this better because beyond a certain very coarse level, estimates are about feelings of engagement and control. What this approach gives them instead is not just feelings, but actual engagement and control. What they get from me is not some unreliable bullshit date, but a commitment to deliver useful things early and often, so that we can together discover something better than their initial visions.
And yes, of course I accept this from people who do work for me. My side hustle is starting a pinball museum. When volunteers take on repairing a just-arrived machine, I never ask them for estimates. I ask them to focus on doing it right. They discover problems, figure out how to fix them, and try it out to see if there are more problems. The work takes as long as it takes. Or I hired a friend to design the logo. I never asked for a date. We iterated on it, discovering an number of things along the way, including through user tests. It took a lot longer than I would have guessed, and that's great, because it turned out better than I could have hoped.
I understand that this is unfamiliar to people, and apparently it's quite upsetting. But I swear this works. There are more things, Horatio.
It's not possible to do what you're describing (kanban/XP) for any commercial application or open source project that is intended to be taken remotely seriously without some amount of estimating involved.
I saw elsewhere that you're defining estimating as a very detailed and involved process, which the author of the article did not, the person I originally claimed estimates are impossible did not, and I did not.
I agree that's not necessary in most cases, if not all, to do the level of estimating you described.
And you'll note that I didn't include "people who are doing me a favor" in the list of individuals I'd insist on an estimate from.
You don't sound like you're one of these people, but I personally feel that software developers who act like they're performing special incantations over their keyboard and can't be expected to answer to anyone about their deliverables do us all a disservice, though maybe I should just be happy that they allow me to provide an alternative that is much easier to work with and results in additional business for me.
I am always fascinated when people tell me it is impossible to do something I have done. That plenty of people have done. For decades. But I shouldn't be surprised. Kuhn was not kidding around.
And yes, I strongly believe in accountability to business stakeholders, customers, and colleagues. I just think the better way is not paper fantasies, but demonstrated realities. Focusing on working software, customer collaboration, and responding to change, one might say. But people have said it before and it didn't make much difference.
You aren't doing what you're doing without performing what is commonly known as estimating, you just avoid using what seem like triggering words to you and the one other person who is almost spamming responses to my comments with the same information.
See "standard" vs. "schedule", and your very specific and formal definition of an estimate that is almost certainly not what anyone else has been using during this discussion and is not used in practice in most software development shops.
Kudos to you for delivering working features on a consistent enough basis that you've earned enough goodwill that people basically leave you to your internal processes and trust that you'll come through for them. I believe that's necessary to have a healthy working environment where you don't end up with what you, I, and every other software developer on earth is trying to avoid, which is an acrimonious relationship with the people with the money where they dictate what, where, when, and how we do our job.
But to claim that there is literally no estimating or scheduling taking place as you perform software development is just not true. You can post your disagreement on every single comment I've made on this topic if you want, your existing comments already speak for themselves on the matter.
As an outsider, I understand what wpietri argues, but I fail to understand yours --- even though I've read other branches of the discussion. Do you argue that wpietri's method still estimates tasks but the estimation is always around two weeks?
Look, software estimates are a whole studied field and have been for decades. Barry Boehm comes to mind here, as well as McConnell. When I talk about estimation, I'm talking about that. And I'm also talking about the bullshit-production process which yields similarly-shape artifacts that also gets called estimation. Something also discussed extensively in the literature.
If your point is that making things has a temporal reality to it, sure. But when people ask me for estimates, I give them two options: we start a formal estimation practice, where we do activities that produce what actual engineers would recognize as reliable numbers, or we do it kanban-style, which does not involve an estimation practice, but gives product stakeholders direct control over time/cost tradeoffs.
Some people, here and elsewhere, argue for a third option, which is going through various levels of rigamarole that gives horseshit numbers so that the more mendacious managers and executives can put on a performance of rigor while telling people with power what they want to hear. I refuse.
And I think it's that bucking the primate dominance hierarchy that makes people absolutely freak out when people talk about no-estimates approaches. Per POSIWID, the purpose of standard corporate "estimation" is not to produce reliable numbers. It's to avoid conflict. I get why people would rather freak out at me than make their bosses unhappy. But I would suggest that's a short-term gain for long-term pain. For organizations that want to actually get things done, hallucinatory project "plans" are bad for everybody. I think ultimately software engineering should be an actual profession. It's not looking great, but I live in hope.
"Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale."
"Working software is the primary measure of progress."
"At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly."
None of which requires estimates. And the bit about working software as the primary measure of progress is specifically targeted against the estimate-driven culture of the time, where people would treat GANTT charts, "percentage complete" numbers, etc, as meaningful measures of progress.
the fact that there was a “manifesto” is always been the funniest shit to me (been hacking since the ‘90’s…)
From another thought-experiment-y perspective:
Say you have problem A to solve. Then either one of those is true:
1) it has been solved before, ergo by virtue of software having a zero cost of copying (contrary to, say, a nail, a car, or a bridge), so there is no actual problem to be solved.
2) it hasn't been solved before, ergo it is a new problem, and thus at any moment you may turn a stone and discover something that was not foreseen (whether they are rabbits, yaks, bikesheds, dragons, or what have you eldritch horrors) and thus of unknown cost.
Any task that cannot be obviously fit into one or the other can nonetheless be split into an assembly of both.
Thus any attempt at estimates is as futile as gambling to win, tasks are only ever done when they're done, and "successful estimators" are kings of retconning.
It's all make-believe.
I was with you until this part:
> Thus any attempt at estimates is as futile as gambling to win, tasks are only ever done when they're done, and "successful estimators" are kings of retconning.
> It's all make-believe.
Software estimates are not futile or make believe. They are useful even if they are not always precise. That’s why the industry continues to use them.
This argument proves too much:
"Bloodletting is not futile or make believe. It is useful even if the patient does not always survive. That's why physicians continue to use it."
"Trial by ordeal is not futile or make believe. It is useful even if sometimes Inquisitors reach mistaken conclusions. That's why Inquisitors continue to use it."
"Lottery-number-picking systems are not futile or make believe. They are useful even if some players never win. That's why players continue to use them."
It is a fully general argument which, if correct, would demonstrate that no practice that had continued for a period of time could ever be ineffective or counterproductive.
The amount of tap dancing and philosophizing some developers are willing to do to dodge estimates is hilarious.
It’s a skill… a basic part and critical part of engineering. IME the common thread between objectors is that they haven’t made a consistent effort to improve — developing, iterating, and refining their estimation process over time.
Yeah, every line of code is a unique snowflake piece of undefinable research the universe has never seen, equally unknowable and inscrutably enigmatic. But the workers at EngiCorp building EngiCorp products using EngiCorp project routines and resources first, second, and third quarter of 2025 are literal world experts at EngiCorp outcomes. They very reasonably should be able to estimate EngiCorp work in Q4, and account for EngiCorp realities, providing maps of future costs that can drive EngiCorp process improvement and investment.
If I ask for a decking estimate and get back sophistry and smug incompetence, I’m not talking with a super skilled professional deck builder. Doesn’t matter how they hammer, saw, or draw.
In my experience, they have negative usefulness in general.
I did work at one company that used an Agile/Scrum process, and though we didn't estimate actual time, we did estimate effort to complete tasks. That worked pretty well: we had an idea which tasks would be easy and which would be hard, and used that in sprint planning to more effectively use developer time.
But the key is no one was held to an estimate, since it was only that: an estimate. If something took 2x as long as predicted, oh well: we just adjusted our planning and worked around it, delaying other tasks for instance.
The problem I see is when managers demand an estimate, and then turn the estimate into a deadline and try to hold developers to that estimate no matter what.
> Software estimates are not futile or make believe. They are useful even if they are not always precise. That’s why the industry continues to use them.
The industry continues to fail when trying to use them. They have negative usefulness.
I am in a project where we have to give estimates in hours and days.
Needless to say we always underestimate. Or overestimate. Best case we use the underestimated task as buffer for the more complex ones.
And it has been years.
Giving estimations based on complexity would at least give a clear picture.
I honestly don’t know what the PO and TL gains with this absurd obscenity.
The last director I had would ask "is it a day, a week, a month, or a year" he understood that's about as granular as it's possible to be.
And he really only used them in comparison to estimates for other tasks, not to set hard deadlines for anything.
Knowing nothing else about him, I like him based on this alone.
I've been in planning sessions where someone would confidently declare something would take half a day, was surprised when I suggested that it would take longer then that since they were basically saying "this'll be finished mid-afternoon today"...and was still working on it like 3 weeks later.
Besides the usual unknown unknowns, I've also seen this happen with tasks that involve a lot of coordination in the SDLC. Oh the PR went up at at 2pm PST? Coworkers in EST won't review it until tomorrow. Maybe some back and forth = another day until it clears review. Then QA happens. QA is heavily manual and takes a few hours, maybe with some false starts and handholding from engineering. Another day passes. Before you know it the ticket that took an hour of programming has taken a week to reach production.
As mentioned in a sibling comment reply:
Time estimations, or conversations to days or other units, typically fail because if a developer says 1 day they might mean 8 focused uninterrupted development hours while someone else hears 1 calendar day so it can be done by tomorrow, regardless of if a developer spends 8 or 16 hours on it.
Are we estimating developer cost (investment cost, writing code only tome), development costs (investment costs including QA time), or time to delivery and first use? People want and use estimates for different purposes. You point out great reasons why knowing what the estimates are for is important.
This is essentially t-shirt sizing without all the baggage that comes from time. Your boss is trying to use the relative magnitude but it's inevitable that people will (at least internally) do math like "7 day tasks is the same as one week task", or worse over-rotate on the precision you get from day/week/month, or even worse immediately map to the calendar. Suggestion: don't use time.
If you pretend not to use time, everyone will do an implicit time mapping in their head anyway. I've never seen it go any other way.
Surprisingly prob yes
But still we are much better at estimating complexity
Time estimations usually tends to be overly optimistic. I don’t know why. Maybe the desire to please the PO. Or the fact that we never seem to take into account factors such as having a bad day, interruptions, context switch.
T-shirt sizes or even story points are way more effective.
The PO can later translate it to time after the team reaches certain velocity.
I have been developing software for over twenty years, I still suck at giving time estimates.
Time estimations, or conversations to days or other units, typically fail because if a developer says 1 day they might mean 8 focused uninterrupted development hours while someone else hears 1 calendar day so it can be done by tomorrow, regardless of if a developer spends 8 or 16 hours on it.
Yes, I've seen this too in sprint planning. However, the layer of indirection I think is helpful. If you use actual time, then when something isn't done after 1 week when that was the estimate, then bosses are asking why. If your estimate was simply "8 story points", then the bosses can't point to a calendar and complain. They can try to argue that an 8-point task should be done in a week, but then you and the scrummaster remind him that points don't map directly to time, just effort.
It's probably not possible to fully prevent people from thinking about time at all, but the more friction you can add, the better.
That's true. Anyplace I've worked where we did planning poker, "points" were always just a proxy for time.
Here's my observation: ballparking an estimate for a whole project, in my experience, tends to be more accurate than estimating each task and adding them together.
I like to think of this as 'pragmatic agile': for sure break it down into tasks in a backlog, but don't get hung up on planning it out to the Nth degree because then that becomes more waterfall and you start to lose agility.
Hours is insane. But ultimately time is money and opportunity cost. Software engineering can’t be the only engineering where you ask the engineers how much something will cost or how much time it will take and the answer is “it’s impossible to know”. Even very inaccurate estimates can be helpful for decision making if they are on the right order of magnitude
There's two things here that get overlooked.
First, people asking for estimates know they aren't going to get everything they want, and they are trying to prioritize which features to put on a roadmap based on the effort-to-business-value ratio. High impact with low effort wins over high impact high effort almost every time.
Second, there's a long tail of things that have to be coordinated in meat space as soon as possible after the software launches, but can take weeks or months to coordinate. Therefore, they need a reasonable date to pick- think ad spend, customer training, internal training, compliance paperwork etc.
"It is impossible to know" is only ever acceptable in pure science, and that is only for the outcome of the hypothesis, not the procedure of conducting the experiment.
> "as soon as possible after the software launches"
This isn't true, just desired, and is one of the main roots of the conflict here. OF COURSE you would like to start selling in advance and then have billing start with customers the instant the "last" pr is merged. That isn't a realistic view of the software world though and pretending it is while everyone knows otherwise starts to feel like bad faith. Making software that works, then having time to deploy it, make changes from early feedback, and fix bugs is important. THEN all the other business functions should start the cant-take-back parts of their work that need to coordinate with the rest of the world. Trying to squeeze some extra days from the schedule is a business bet you can make but it would be nice if the people taking this risk were the ones who had to crunch or stay up all night or answer the page.
Trying to force complicated and creative work into a fake box just so you can make a gantt chart slightly narrower only works on people a couple times before they start to resent it. 10x that if management punishes someone when that fantasy gantt chart isn't accurate and 100x that if the one punished is the person who said "it's impossible to know" and then was forced into pretending to know instead of the person doing the forcing.
My take: if they have not done the work to get to at least some degree of a spec, and they won't give you time to review and investigate, they get nothing more than a vague, relative t-shirt size.
I think software is one of those VERY rare things, where inaccurate estimates can actually be inaccurate by "orders of magnitude". After 20 years in the field, I still managed to use 2 months of time on a task that I estimated as 10 days.
A rule that has suited me well is to take an estimate, double it, and increase by an order of magnitude for inexperienced developers. So a task the say would take two weeks ends up being 4 months. For experienced developers, halve the estimate and increase by an order of magnitude. So your 10 days estimate would be 5 weeks.
The biggest estimation effort I ever took part in was a whole-system rewrite[1] where we had a very detailed functional test plan describing everything the system should do. The other lead and I went down the list, estimated how long everything would take to re-implement, and came up with an estimate of 2 people, 9 months.
We knew that couldn't possibly be right, so we doubled the estimate and tripled the team, and ended up with 6 people for 18 months - which ended up being almost exactly right.
[1]: We were moving from a dead/dying framework and language onto a modern language and well-supported platform; I think we started out with about 1MLOC on the old system and ended up with about 700K on the rewrite.
10 days was already after I used this algorithm. Previous several tasks on that codebase were estimated pretty good. Problem with this is that some tasks can indeed take SEVERAL orders of magnitude more time that you thought.
One of the hardest problems with estimating for me is that I mostly do really new tasks that either no one wants to do because they are arduous, or no one knows how to do yet. Then I go and do them anyway. Sometimes on time, mostly not. But everybody working with me already knows, that it may be long, but I will achieve the result. And in rare instances other developers ask me how did I managed to find the bug so fast. This time I was doing something I have never before done in my life and I missed some code dependencies that needed changing when I was revieving how to do that task.
I’ll send my friend that has a construction company to build your next 3500 sq ft house for $13.6 million dollars :)
Something often overlooked in cost/schedule estimates is the nature of joint probability of actions slipping. Action A slips and causes action B to slip. I think software is tougher to estimate because the number of interfaces is often much higher, and sometimes more hidden, than in hardware.
as opposed to say building a house where framing can totally slip while we run electricity and build a roof floating in mid-air
software is only tougher to estimate if incompetent people (vast majority of the industry, like 4+ million) is doing the estimating :)
My home construction slipped 6 months on 2 year build time. It happens in construction very often.
> software is only tougher to estimate if incompetent people (vast majority of the industry, like 4+ million) is doing the estimating :)
No, it is tough to estimate, but not only for incompetent people. And "incompetent" can be stretched to "don't know what he's doing", which is how I operate most of the time. I don't know what really needs to be done until it's done. Main part of my work is research on what actually needs to be done, then "just" implementing it. If I waited with estimating until I know what needs to be done, I would spend 3/4 time estimating and then 1/4 with clear understanding and good schedules (example description: I will be clicking keys for 5 hours).
> My home construction slipped 6 months on 2 year build time. It happens in construction very often.
Tangent, but I have at least 3 friends that would've (in retrospect) been nothing short of ecstatic if their home construction had "only" slipped 6 months on a 2 year timeline.
That’s a bit of a strawman considering I was deliberate in saying hardware interfaces are limited and not saying they are zero. The number of interfaces in software is often going to be orders of magnitude greater. The network effects and failure modes will often increase geometrically with the number of interfaces. In fact, big construction design firms have tools to easily identify and mitigate the “clashes” you bring up and those tools tend to work well because there is a finite number and the designs are well-documented (as opposed to software where changes are relatively cheap and easy so they often occur without documentation)
Saying incompetence is the reason is a trivial rebuttal that ignores the central claim about complexity. It’s like saying “the reason why we don’t have a theory of everything is because we don’t have competent physicists”
That's a factor four or five ir so, so still less than an order of magnitude.
The next natural progression of this line of discussion between "the business" and engineering is for them to agree on a time range as an estimate. Engineering doesn't want to say it'll be done in 6 weeks, but they feel okay saying it will take between 4 and 20 weeks so this estimate is accepted.
You can guess what happens next, which is that around week 8 the business is getting pretty angry that their 4-week project is taking twice as much time as they thought, while the engineering team has encountered some really nasty surprises and is worried they'll have to push to 24 weeks.
You're forgetting the part where the business expects the engineers to pull all-nighters so they can meet the deadline.
it is still better to give a range though because 1. it explicitly states the degree of unknown and 2. no boss is going to accept 4-20 weeks, which means you start talking about how you can estimate with better accuracy and the work required to do so, which is a major goal of planning & estimation.
> Software engineering can’t be the only engineering where you ask the engineers how much something will cost or how much time it will take and the answer is “it’s impossible to know”.
Because it's not engineering at all. But even if it was, plenty of engineering projects are impossible to estimate - the ones that are doing something novel - and disliking that fact doesn't make it go away.
> Even very inaccurate estimates can be helpful for decision making if they are on the right order of magnitude
If what the business wants is an order-of-magnitude, they should ask for that; often (not always!) that's a lot easier.
> I honestly don’t know what the PO and TL gains with this absurd obscenity
There are marketing campaigns that need to be set up, users informed, manuals written. Sales people want to sell the new feature. People thinking about road maps need to know how many new features to can fit in a quarter.
Development isn't the only thing that exists.
Another reason is that figuring out what the software to be written should actually do, and how it should work, is work that is part of the project and the time it will take needs to be estimated.
As well as the actual development work that will result, which isn't known yet at the time of estimation.
I call it the "Persistent Incompetence of Software Development", which is another perspective on estimation, focused more on expertise. A chef that cooks pizzas, cooks the same pizza over and over again and becomes amazing at it. If you are a developer that writes the same code over and over, you are terrible at software development. A good software developer should always be solving new problems, as by the nature of software, once they solve a problem, they never solve that (exact) problem again. So we are persistently incompenent.
Which is why software development can't be estimated, as well. Because it is all, as you say, novel. With infinite error bars.
At this point, I can't take anyone seriously that believes software dev can be estimated.
There is very little novel about most B2B CRUD and internal bespoke apps that most developers are doing. The novel part if any is implementing the business vertical logic
"... anybody with any brains has already left town..." -- Bob Dylan
Anybody with any business experience has already isolated themselves from the certainty of software project failure, where "failure" is a euphemism for "late." So it doesn't matter if software can't be estimated.
This can be nerve-wracking to a beginner, but one gets used to it over time.
As I’ve gotten older, I find this to be untrue.
Estimating the unknown is itself a skill.
It’s like choosing to crash random parties everyday — at first, exceptionally novel but facing the unknown becomes itself mundane. Humans adapt. They build mental modals to approach the unknown and these mental modals are predictable.
I just don’t think most people get much practice.
That's the point: how can you tell WHEN you are going to reach a place you've never been before traveling an uncertain path?
Making mistakes over and over again. And adding a lot of time buffers just to be safe
> If I ask you how long it'll take to add a new database field, and you've added a new database field 100s of times in the past and each time they take 1 day, your estimate for it is going to be very spot-on.
But in the software world, predictable and repetitive tasks are also the kinds of tasks that are most easily automated
The slow part isn't following a written "log in and run ALTER TABLE" runbook, it's the review of "does this solution make sense given what we're trying to do".
Spot on! Now add one more dimension (from machine learning projects and pretty much other research fields) -- you also never know if the result are going to even work.
Maybe the data just doesn't have the correlation you want. Maybe you just didn't try harder, never know for sure. But owners reasonably demand some estimations.
Repetitive and easy or worth automating are not the same. The fundamental problem is that a setup capable of solving any of the requests that come to it is as complex as just a whole programming language and then you’re back to square one.
As a project manager, it sounds like you're making excuses. Just give me a number, trust your gut!
We have a fundamental failure to communicate, what we're doing. The game project managers and finance believe we're all playing is a regression towards the mean, where everything is additive and rounds up to nice consistent formulaic sums. Whereas software development follows power law distributions. A good piece of software can deliver 10x, 100x, or 1000x the cost to produce it (ex: distribution, cost of delivering another copy of software is near 0 once written). You don't get that sort of upside with many other investments. Finance is happy with an NPV 8% above what they invest in. This means that when software people talk, everything they say sounds foreign, and everyone assumes it's because of jargon. It's not. The fish don't know they're swimming in water. When the fisherman comes, everyone is caught off guard.
So we get what the author talks about
> The estimates stopped being estimates. They became safety railings against being held accountable for unreasonable expectations.
We. Pad. Like. Crazy. Yes this is inefficient. Some project managers recognize this. We get theory of constraints. But rather than cull the layers of hierarchy that lead to the padding in the first place, all the blame for failure goes back to developers. Get hit on the head enough and you will stop acting in good faith and pad to save your ability to feed and cloth yourself.
> As a project manager, it sounds like you're making excuses. Just give me a number, trust your gut!
If we're just making up numbers, why don't you just make it up yourself and save the developers the trouble?
Ah! But I want you to own it. If you say it first... you own it. And I do not have to get you to agree to it.
This is usually how I get tricked into setting deadlines. I get asked for a "rough estimate", then it magically becomes a deadline.
It's not obvious to me that we should avoid padding, or why it's seen as undesirable.
Padding ties up capital, it reduces credibility, it delays deployment, it adds costs through delay. It is bad for organizations. However, it is a great solution if you're a worker in a bureaucratic environment that can tolerate large costs, but is intolerant of 1-day of schedule slips. It's a great solution for complacent management, who are confused about the game they're playing and wants to report that they're "on track", which means "not late".
The agile solution of incremental value delivery is a compromise, and can produce good outcomes for functional changes. But agile has unacceptable failure modes when working on infrastructure and satisfying system constraints. Agile can work okay for programmers, but it's not a solution for engineers. Acknowledging, owning, and managing risk is more scalable, but you have to have leaders who acknowledge that they exist and have the maturity to take on that responsibility.
> Padding ties up capital, it reduces credibility, it delays deployment, it adds costs through delay.
Well done timelines are a negotiation between the stakeholders and engineers. The stakeholders need something done for the business, the engineers give a timeline. If that timeline works for everyone, great. If it doesn't, then the stakeholders will ask if it can be done in a faster time.
A timeline that lands on time, or early, is good. The point of timelines is that teams outside of engineering are resourcing their projects based on your timelines. They may have made outside commitments to customers, they may be lining up marketing, they may have embargoed PR, it may be delivered by someone at a conference, etc.
A project running late can be catastrophic. Bad customer relations, wasted marketing spend, pulling back stories from PR, delays for dependent teams, etc.
You pad to make sure your timelines aren't overly optimistic, because we're all bad at estimating, and it's possible our dependencies are too. By padding, when it comes time to negotiate for shorter timelines, you also have some wiggle room.
Bureaucratic environments tend to be larger companies and they care about schedule slips, because they have more teams being impacted, and those teams are handling larger numbers of overall projects. Schedule slips can lead to cascading failures.
Very insightful. This is also my go-to argument for why software engineering is real engineering.
This is such an interesting observation, makes a lot of sense.
This is going to be immediately downvoted, but whenever I see software developers mention that estimating is uniquely challenging for software it immediately calls out a bunch of red flags. Estimates in software are challenging but no more or less challenging than other industries.
Typically the people that single out software as a unique snowflake in the world of delivery and estimation really just say out loud they have no experience in project management and no experience working outside of authoring code. The things that make estimations most challenging in software are the same factors that make estimations challenging everywhere else.
I don't think it's unique, but many software projects are inherently prone to big surprises you can't catch until you start the work. I would love to keep estimating and estimating and get an accurate number, but it costs money and, eventually, you're just working on the project.
Can you give an actual counterargument?
I don't know how typical teams work, but in my case, new projects always come on top of other obligations. It may take 1 day to add the field, but how many meetings, fires, or other disturbances will happen during that day?
If software developers want to be then seriously as a profession, they need to be able to provide and justify estimates for their work.
Everything you said could apply to a new bridge, building, pharmaceutical compound, or anything else that is the result of a process with some known and some unknown steps.
Pharmaceutical compounds frequently don't make it to market after significant investment.
No one in that industry is giving estimates based on developing brand new drugs - they're giving estimates related to manufacturing lead times, unalterable physics time lines, and typical time to navigate administrative tasks which are well known and generally predictable (but also negotiable: regulations have a human on the other end). All of this after they have a candidate drug in hand.
Same story with bridge building basically: no one puts an estimate on coming up with a brand new bridge design: they're a well understood, scalable engineering constructions which are the mostly gated by your ability to collect the data needed to use them - i.e. a field survey team etc. - and also once again, regulatory processes and accountability.
Yes, that's my point. There's way more uncertainty in trying to bring a new drug to market or build a new bridge than creating yet another CRUD app, yet somehow they are any able to break these efforts into tasks that can be estimated and tracked and many software engineers think they should be exempt from any accountability to schedule or budget.
And do you think those things are delivered on schedule any more often than software projects?
Take a look at the top of this thread and see what we're talking about.
The fact that people in many industries are not good at estimating doesn't mean that it's impossible in software development specifically and uniquely, as was originally claimed.
> Everything you said could apply to a new bridge, building, pharmaceutical compound
"Everything"? So
> predictable and repetitive tasks are also the kinds of tasks that are most easily automated, which means the time it takes to perform those tasks should asymptotically approach 0.
Also applies to bridges? Bridges require a ton of manual human input at every stage of construction, regardless of how predictable and repetitive the work is. With software, we can write software to make those tasks disappear. I've yet to see the bridge that can build itself.
1. When you have meaningful software that can build itself (and I'm not talking about the compilation process), let me know.
2. You can estimate the duration of each step of a process, regardless of how much human involvement is required.
You should be talking about the compilation process, because that's the thing that puts the cost of non-novel software at near zero, and you can't recompile a bridge for free.
Estimating the duration of each step of a process only works when you know what the steps are.
Effectively free distribution is what makes the cost of non-novel software so low. Compilation isn't even needed for executables that can be used as is.
That said, implementation is one part of developing software. Design and test are also necessary and can take a non trivial amount of time.
And yes, you need to know what the steps are to build something. If you don't, you don't know what you're doing, which is a bad thing.
Drug discovery chemists do not, to my knowledge, provide estimates on how long it will take them to discover a marketable drug.
You think the pharmaceutical industry just gives a bunch of resources to chemists, says "godspeed", and waits around for the chemists to report back at their convenience?
In my limited exposure to the industry, that's not how it worked. They have budgets, timelines, and progress is tracked as it is determined whether there is a viable path to a marketable drug.
From both the developer and manager side of things, I've found that the most important attribute of estimates is frequently the least paid attention to: that they be kept up to date.
When you discover more work hidden under that "simple" pile of code, you absolutely HAVE to update your estimate. Add more points, add more tickets, whatever. But then your various managers have the ammunition to decide what to do next - allocate more resources to the project, descope the project, push back the release date, etc.
Far too frequently, the estimate is set in stone at the start of the project and used as a deadline that is blown past, with everyone going into crisis mode at that point. The earlier the estimate is updated, the calmer and more comprehensive action everyone responsible can take.
Which if you try to do - those agile people will kill you for it.
They wrangle a number out of you which goes into an user story estimate, which feeds into a Gantt chart they use to make their pretty powerpoints they present to upper management to say that the feature will make it into the Q4 release.
If you move this number around the whole estimation will crumble, not that it wont in real life but you deprave them of two things - an illusion of control and somebody to blame when things go south.
> Which if you try to do - those agile people will kill you for it.
Does this actually happen to you? This is literally the whole point of agile, is to change the plan as you learn more about your work. If you didn't want to change the plan you'd spend a lot of time on up-front planning and do waterfall.
Like, a Gantt chart is more or less explicitly anti-agile. I'm aware of the 'no true Scotsman' thing but we shouldn't buy into people using agile terms for what is really a BDUF-based plan.
> Does this actually happen to you?
Yes and millions of other devs who work in an enterprise 'agile' environment - where a single huge project is/was developed by armies of developers work on a single product with a strict-ish release cadence?
Have you heard about the horror that is SaFe?
I'm not convinced that true agile works or has ever worked on a project that was bigger than a dozen devs.
In practice, it's just another dishonest way of selling consulting hours, infantilizing and disempowering devs, and putting folks who have zero subject matter knowledge in charge by doing these feelgood rituals.
Agile (scrum) in practice at enterprise-scale projects tends to be a combination of feelgood BS +top-down micromanagement (product owners dicking around with task priorities) +traditional project management.
One of the key ways these agile people are incredibly dishonest, is that Agile at the top level is sold to enterprises as a way of keeping the old high-level project management style, with push-only command-structures, and agile people subsequently try to sugarcoat it as it somehow 'empowering' the devs and giving them autonomy, when the truth couldn't be farther from it.
> Have you heard about the horror that is SaFe?
Yes, and I successfully argued against it by pointing it was was a wolf in sheep's clothing. It has as much to do with agile as waterfall does.
> I'm not convinced that true agile works or has ever worked on a project that was bigger than a dozen devs.
It works fine, large projects are inherently trouble which is why organizations should spend some energy into reducing the scale of an individual team's work rather than piling dozens or hundreds of people into yet another Too Big to Fail megaproject. If Google can build and maintain Bigtable with like a dozen devs then maybe you don't need 200 people and a PMO for your enterprise data warehouse consolidation project.
In fact the biggest issue with SAFe is the size of the project you'd try to use it on, not that it references agile style methods. Waterfall methods were even worse, which is the only reason charlatan consultation manage to keep selling organizations on things like SAFe.
> One of the key ways these agile people are incredibly dishonest, is that Agile at the top level is sold to enterprises as a way of keeping the old high-level project management style, with push-only command-structures, and agile people subsequently try to sugarcoat it as it somehow 'empowering' the devs and giving them autonomy, when the truth couldn't be farther from it.
You're right that this is dishonest and that people try and fail to cargo cult successful efforts they watch from afar. But that doesn't mean these successful teams weren't successful, or that there aren't common attributes of those successes.
That's always been the problem with methodological fixes to the software delivery process for organizations, you usually can't impose the structure from the outside anymore than you can meld your bones with adamantium without having a crazy mutant healing factor...
> If you move this number around the whole estimation will crumble
I still love this sprint where the further into the sprint we went, the further the ‘remaining work’ line went up.
It’s good we could do that without blame, but it looks super funny.
You're describing the opposite of Agile Development. The Agile Manifesto says, "Responding to change over following a plan."
Yes, but the comment is targeting those people that usually would say about themselves that they embrace agile, while actually fighting everything that changes their little schemes...
So should we be talking about how "those democratic people" put everyone who disagrees with them in concentration camps because that's what the Democratic People's Republic of Korea does?
There are, of course, extreme-right-wing reactionaries who make exactly that argument, but I don't think their example is a good one to follow.
That war is over. We lost.
I have noticed the same for a lot of long-running software projects. The estimate is created at the start an never revised.
Projects can and will fail or run late; but heck; a 6-months projects cannot found late after 5 months and 29 days; things must be discovered early, so that the most important issues can be addressed.
> But then your various managers have the ammunition to decide what to do next - allocate more resources to the project, descope the project, push back the release date, etc.
In your view, is this a good thing or a bad thing for developers? In other words, are developers incentivized to update their tickets because it gives their managers "ammunition" to make project changes?
I find that this is usually more evidence of organizational dysfunction.
Sometimes, its developers who don't have enough ownership - someone higher-ranking filled out all the tickets for them and left no room for interpretation, so changes require trashing a bunch of irrelevant tickets and writing a bunch of new ones, making for more communication overhead and less time spent in development. Developers are disincentivized to update tickets since doing so just becomes write-amplification for more bureaucratic overhead.
Sometimes, it's a communication breakdown. Tools like Jira are necessarily asynchronous, and decisions about what to build are necessarily political. Politics and asynchronicity don't mix, such discussions need to be held face-to-face. If they're held face-to-face, then having developers update Jira is pure overhead. Developers are disincentivized from updating Jira because either it's a one-sided conversation (lag in getting notified that your ticket was closed and the work on it should be thrown out, PMs don't always follow through with issue links from the closed issue to the newly opened issues, so developers lack context on what decisions were made on replacements), which is emotionally draining, or the conversation happens outside of Jira, in which case updating the ticket is a formality/not actual communication, in which case it feels like a waste of time.
For most development work, I'd argue that developers shouldn't be making changes in Jira, their managers should be, on the basis of frequent, synchronous communication with the developers they manage.
Exactly. And I’ve found this is incredibly hard to get people to do, developers especially. No one wants to say a task is going over / taking longer than expected but will gladly run over. It’s hard for pms to resource if estimates are not updated as scope and complexity changes.
Also doesn’t help when estimates become due dates.
This is why stuff like Jira is so polarizing for many developers. It is an additional translation tax on me, the developer, to have to go constantly keep this tracking tool thing up to date so that the product owners/managers stay off my nuts. The burden of effort is placed on me to make their tool functional that they can have what they need to provide roll up status. This tool virtually never benefits me, it's always a time consuming sidequest that breaks focus and scatters important information all over a kanban board where tickets disappear and makes it really hard to cohesively work as a unit to get anything done. Like many orgs we've had PM's and stuff try to mandate expectations on when those updates should be provided, somehow completely oblivious to the human nature of development: you should never expect that a developer can simply hop in and out of moments of deep focus and suddenly provide a bunch of updates regularly when it's convenient for you, as if that has zero impact on their work. It takes a toll, full stop. I won't say no to it, but I do expect PM's to know that style of PM'ing is effectively asking me to make your job easier by populating your tool with updates that are useful to you and not me because I'm well aware you aren't involved enough to assess where things are yourself.
We've gone through so many cycles of how to use Jira well at my org where these frustrations are shared and we try a different approach, and we're finally starting to converge on the idea that this has historically been a little too lopsided requiring too much tax on the developer doing the actual work. We agreed on a new approach that has actually been pretty awesome: the product owners or managers that are trying to direct a body of work must be a little more in the trenches with us to have an overall understanding of where the different pieces are in a moving body of work. We don't expect them to understand the nitty gritty work in the trenches, but at the same time no more 30,000 foot view product managers who just ask for status updates at EOD. _Everyone_, developers included, is responsible for keeping documentation up to date as we go. So we have central working-bodies of information to find details without having to cruise thru 100+ jira tickets to find details we're looking for. The expectation is that they're engaged enough with development whether in chat or on meetings that if they were blindsided by an executive asking for an update, they could speak to it with some authority over at the water cooler without having to go check Jira. This has really helped weed out the lazy product owners/managers, has forced them to thoughtfully consider their meeting schedules, and has placed the exceptional ones in the pod of work being done and really added a lot of velocity and together-ness about the things we're pushing along.
This approach we're using now was born out of some hurt feelings from projects that didn't go so well & we had to have some real restrospective convos where everyone aired out their beef. Those are good convos to have, I think a lot of teams would find that people aren't deceptively trying to screw you over. Being encouraged to level set human-to-human genuinely is one of the greatest parts of working where I work. Always walk away from those types of chats learning valuable things: for the most part our product owners really do care. Not just about their career aspirations but also about _us_ nerdy and sometimes socially maladjusted developers. They look forward to working with us, and they want to make this as easy as possible for themselves but also for the developers. In the past they spent a lot of time in planning phases trying to scaffold out a project in Jira and attaching timelines to it so that their needs are met to provide predictable timelines to their bosses... but also with the hope that by plainly outlining work it sort of 2 in 1 satisfies our needs and will make development timelines a breeze. We've had to ask them to cede rigidity on that latter part because even the software architects admit the work being done is often a moving target. And when that target moves, maybe you realized you need to add a pivotal software solution to the stack, you can sometimes throw like 45 planned tickets into the dumpster. New ship dates need to be assessed. This was our reality check that we were all collectively shit at adapting to the dynamic nature of any given project. Now, our product owners decided that the expectation they have of their own role is that they understand this dynamic and are prepared and willing to make the case about why the shipping timeline must change. So there's actually a pain point solved here: don't break your back doing so much up front work to try and guess/capture what the body of work might look like only for it all to possibly get thrown away, involve architecture a bit more in the planning phases, but most importantly let's engage throughout the project and we'll try our best to have shared ownership/interest in making sure where we are in the project is broadly understood by everyone involved.
We're currently in phases of implementing a major platform right now and it's just night and day better and dare I say fun. We're still keeping Jira up to date, but the product owners and PMs are more or less managing this as it is a tool they find useful. Removing the "can you update this ticket please" 24/7 has forced them to be a little more involved and have the right chats, but also makes us developers happier to jump in and get it updated on our own volition because we also want to help them have an easier time. If my PM pings me and says "hey I'm looking at this ticket that's stuck in blocked, I just wanted to make sure we got an update from so-and-so about provisioning this credential so I can follow up if needed" I will likely automagically jump in and be like "still stuck, but let me update that ticket for you there's a couple key details I want to make sure our there for you before you reach out". There's an inherent "we're both interested in seeing this ticket through" here that doesn't strike a nerve with either party. Pretty much everyone involved both developers & non developers has a really solid read on where anything's at and we're all just talking a lot more. And for developers I find that it's really good, even if you've got them committed in one narrow body of work, to understand the larger pieces in motion. When they're in tune with the broader orchestration of a projects timeline, they tend to weigh in during unsuspecting moments that might tie seemingly unrelated pieces together. They might be assigned to work on x, but in a group chat about y they notice y has a dependency on x and they'll speak up and call out the need to test that they both work together. We've had a lot of great callouts materialize like this, and on a human-psyche level I think it snowballs & avalanches encouraging developer participation in a way that is really meaningful for PMs. It's interesting that Jira & the expectation that predicting development time in an arena of uncertainty was previously in the way of forming the group dynamics we have now. Jira, despite just being a tool, can really amplify a lot of bad outcomes when it's used by those who aren't fit to be near development, it sort of devolves into a two dimensional behind-schedule tracker that detrimentally impacts how team members on the ground communicate with each other.
And since we're talking a lot more there's just like... way more memes. And memes are important in any development timeline. We prioritize laughing through the pain together.
This is a fantastic description of why Technical Program Managers (TPMs) can be force multipliers. Imagine involving someone who understands both the needs of Product and Engineering and whose job it is to allow both of them to focus on where they add the most value. We do exist!
It is very much this, your types are insanely valuable to how quickly and effectively we can blast through massive bodies of work that are well beyond the scale of personal-hobby-project. We're in the annoying HR-involved process of revising the role internally to actually include the word 'Technical _' & the bullet points of expectations will demand such capacities. It's been an exciting shift for us and we've reappropriated non-technical product owners/managers to roles they are happier in (and believe it or not we are happy for them).
We didn't get it right the first, second, third, fourth or fifth time. I'd say as an org we are learning lessons that other orgs may have learned a decade ago, but it's just nice to come to these conclusions on our own & really understand how our setup came to be. We hope we remember that things can evolve again in the future, but are grateful that we journeyed together and didn't just throw in the towel and fire people when things didn't work out. We believe sending people through some fires of hell, reassessing, then reattempting is sort of a cycle that levels people up.
We hope to have more technical product guys on board up ahead cause it's a dream setup that really organizes and harnesses velocity in all the right places, but also effectively stops unnecessary side-quests. It's so nice to have a technical product guy step in and say "no" to some absurd executive request because he/she is well aware of what such an implementation would look like. They can actually be vanguards and stewards over development personnel in their own way and it seems to go hand in hand with a lot of mutual respect for each other. Always get a kick out of nerding out over possibilities with our technical product dudes.
The biggest problem I've seen isn't the estimate itself but the telephone game that happens after. You say "probably 2-3 weeks" to your manager, who tells the PM "about 2 weeks", who tells sales "mid-month", who tells the customer "the 15th".
By the time it reaches the customer, your rough guess with explicit uncertainty has become a hard commitment with legal implications. And when you miss it, the blame flows backward.
What's worked for me: always giving estimates in writing with explicit confidence levels, and insisting that any external date includes at least a week of buffer that I don't know about. That way when the inevitable scope creep or surprise dependency shows up, there's room to absorb it without the fire drill.
This is why I push for Kanban whenever I am a PO. If we can ballpark an estimate, I can prioritize it. If we cannot ballpark an estimate, I can prioritize the research to clear out some of the unknowns. But most importantly, we set an expectation of rolling feature rollouts, not inflexible release dates. We communicate both internally and externally the next few things we are working on, but no hard dates. The article correctly identifies that hard release dates communicated to customers are the root cause of problems, so I simply don't give such things out.
Sorry, but how does Kanban come into this?
From your description, SCRUM, could work just as equally.
Don't get me wrong, I'm a fan of Kanban, it's awesome for visually presenting where bottlenecks are for tasks, but estimations aren't a feature of that.
But SCRUM maybe, where people are having a sprint planning meeting maybe more what you're thinking?
Kanban in IT world in my experience implies approach where you focus on the work and tasks as they come based on priority. It doesn't imply what is on the board is finished strictly by some date, as the whole premise is that you can't really know.
SCRUM implies sprints where you agree in advance what will be actually pulled into sprints and delivered by the team so spillovers are not really expected / wanted.
I think that we are agreeing that Kanban and estimates aren't really analogous
When working with kanban I maintained a average number of card done per days. if someone asked when some card woud be done, I just multiplied the number of card ahead of that one by average and get an estimate. You can estimate the cards but usually it doesnt really improve accuracy as tasks are on average the same size.
What the article suggests is basically Kanban. It's the most effective SW development method, and similar scheduling system (dispatch queue) is used by operating systems in computers. However, management doesn't want Kanban, because they want to promise things to customers.
You can make good estimates, but it takes extra time researching and planning. So you will spend cycles estimating instead of maximizing throughput, and to reduce risk, plan is usually padded up so you lose extra time there according to the Parkinson's law. IME a (big) SW company prefers to spend all these cycles, even though technically it is irrational (that's why we don't do it in the operating systems).
Another reason kanban doesn't work for large projects is because you have to coordinate your cycles with multiple dependencies teams roadmaps and releases.
I don't think so, only if they need to have a schedule as well. Most OSS projects operate as Kanban and it's just fine.
Waiting on a dependency is kinda like waiting on a lock held by another process in the operating system. It has little bearing on whether dispatch queue is effective or not; in fact, it shows the solution: Do something else instead of waiting. (This is why the OS analogy is so useful for project management, if only PM's would listen!)
It's again, if you need to plan things ahead (for some reason) when the dependencies become a problem.
But maybe I misunderstand what you mean, if you still disagree provide a more specific example.
> technically it is irrational
Only if your company operates in a vacuum, without investors or customers
As an investor, I don't like a investment that throws away 10-30% of its resources, perpetually lowers morale except among the least creative and misses opportunities because their competition is faster.
Investing or buying into something new and unknown is by itself not a rational act (to the extent the thing is novel).
IMHO rationality (optimization) only makes sense relative to a well-defined goal.
When teams don't need strong estimates, then Kanban works well.
When teams do need strong estimates, then the best way I know is doing a project management ROPE estimate, which uses multiple perspectives to improve the planning.
https://github.com/SixArm/project-management-rope-estimate
R = Realistic estimate. This is based on work being typical, reasonable, plausible, and usual.
O = Optimistic estimate. This is based on work turning out to be notably easy, or fast, or lucky.
P = Pessimistic estimate. This is based on work turning out to be notably hard, or slow, or unlucky.
E = Equilibristic estimate. This is based on success as 50% likely such as for critical chains and simulations.
> E = Equilibristic estimate. This is based on success as 50% likely such as for critical chains and simulations.
I've found giving probabalistic estimates to be hopeless to effectively communicate, even if you assume the possible outcomes are normally distributed, which they aren't.
Critical chain planning has a purpose for the 50%. It's not perfect-- but it's better than other ways, in my experience.
Wikipedia: "The justification for using the 50% estimates is that half of the tasks will finish early and half will finish late, so that the variance over the course of the project should be zero... Because task duration has been planned at the 50% probability duration, there is pressure on resources to complete critical chain tasks as quickly as possible, overcoming student's syndrome and Parkinson's Law."
Three-point or PERT estimates are in the same vein, but are just an old and established business process concept and not a trademarked "project" from a (defunct?) consultancy.
https://en.wikipedia.org/wiki/Three-point_estimation
https://projectmanagementacademy.net/resources/blog/a-three-...
And yeah, they can be useful when you can must put numbers to things and can decompose the work into familiar enough tasks that your estimation points are informed and plausible. Unfortunately, in a field often chasing originality/innovation, with tools that turn over far too often, that can be a near-impossible criteria to meet.
Yes you're correct. PERT 3-point was my starting point and I wanted to add a critical chain 50% estimate.
BTW the SixArm name and ROPE is merely my personal work. Way back when, I had to put on the trademark because someone else said my work was theirs.
This sounds like that if you don't trust a fortune teller, you can mitigate it by going to 4 different fortune tellers, and then somehow combining their predictions into your much more certain future.
Yes except in practice it's worked out more like this... Realistic is favored by tech teams. Optimistic is favored by salespeople and marketers. Pessimistic is favored by lawyers and compliance people. Equilibristic is favored by project managers and military.
Each stakeholder gets the kind of fortune they want. The important aspect (to me) is that the four viewpoints show all the stakeholders that their viewpoints are quite different.
So now you have 4 wrong estimates to work with :) To have some predictability you should have small features. That's the only thing that can give you strong estimates. No industry has solved giving strong and correct estimates - it's in the name, it's estimation!
Yes small features help enormously. I've used ROPE and work breakdown structures (WBS) to estimate multiple projects at thousands of staff hours to under 4% of actual. The ROPE advantage is that it uses multiple perspectives, which gets stakeholders understanding that the estimates are imperfect and depend on many factors.
The trouble with estimation is that few places record the estimates and the actuals for future reference.
As I've pointed out before, the business of film completion bonds has this worked out. For about 3% to 5% of the cost of making a movie, you can buy an insurance policy that guarantees to the investors that they get a movie out or their money back.
What makes this work is that completion bond companies have the data to do good estimations. They have detailed spending data from previous movie productions. So they look at a script, see "car chase in city, 2 minutes screen time", and go to their database for the last thousand car chase scenes and the bell curve of how much they cost. Their estimates are imperfect, but their error is centered around zero. So completion bond companies make money on average.
The software industry buries their actual costs. That's why estimation doesn't work.
Movie people are doing essentially the same thing over and over for 100 years. Everyone in the industry does things the same way. Their projects last only 1-2 years and are totally done once "in the can". In my experience this is almost never the case with technology based projects.
If your database of software projects has a thousand web browsers in it, you have to ask why all those programmers couldn't work together and write just ten or fifteen web browsers; maybe they're hobbyists who are reinventing the wheel for fun? If so, is that a good basis for estimating a new web browser?
The first time I worked on a database-backed web site, my team had written their own ORM, and while I was there we put together a thing so we could use the same view on more than one web page. The second time, we used web.py and no ORM. The third time, we used Ruby on Rails, which solved most of the problems we had on the first project I mentioned out of the box. Most recently, I used Django, which is roughly as convenient as Rails but easier to debug. I haven't tried it, but I'm guessing Claude Code could probably put together a usable Django site in a few minutes.
So achieving equivalent results requires very different activities over time, because each time someone does it, we accumulate not just knowledge but code that makes it easier the next time—thousands of times easier, if the novelty is small enough. This impedes estimation.
Often, too, the things we're trying to estimate are things like "add automatic Dart dependency support to Blaze". The time-consuming part of this is not breaking the rest of Blaze in the process. What reference class do we use for estimating adding new languages to Blaze without breaking it? Is Dart more like C++ or more like Go? These are questions for lawyers, not actuaries.
Maybe the videogames industry is different because they don't share code and don't maintain it? But it seems like an awful lot of games these days are getting written in Godot and Unity where 15 years ago people would have written their own engines... and problems that were major technical challenges for Quake are trivial now.
If someone wants ragdolls in their game, they can just license your code, right? They don't have to write it themselves. A dataset of how long it took people to implement ragdolls from scratch wouldn't provide any insight into how long it should take to integrate your ragdoll library.
My employer does, but only analyses the overage, so when a project comes in under estimate they think it's because it went smoothly, the reality is actually that estimates only ever tell you how good you are at estimating.
The best hack for improving estimation is first never giving a single number. Anyone asking for a single number, without context, doesn't know what they are doing; it's unlikely that their planning process is going to add any value. I think they call this being "not even wrong".
Instead you should be thinking in probability distributions. When someone asks for your P90 or P50 of project completion, you know they are a serious estimator, worth your time to give a good thoughtful answer. What is the date at which you would bet 90:10 that the project is finished? What about 99:1? And 1:99? Just that frameshift alone solves a lot of problems. The numbers actually have agreed-upon meaning, there is a straightforward way to see how bad an estimate really was, etc.
At the start of a project have people give estimates for a few different percentiles, and record them. I usually do it in bits, since there is some research that humans can't handle more than about 3 bits +/- for probabilistic reasoning. That would be 1:1, 2:1, 4:1, 8:1, and their reciprocals. Revisit the recorded estimates during the project retrospective.
You can make this as much of a game as you want. If you have play-money at your company or discretionary bonuses, it can turn into a market. But most of the benefit comes from playing against yourself, and getting out of the cognitive trap imposed by single number/date estimates.
One interesting angle for me is that I am seldom given complete specs or requirements when asked for an estimate. Of course you ask questions to try to determine key information that has not been specified but often the answers are not available or fully reliable.
So any estimate has to include uncertainty about _the scope of the work itself_ as well as the uncertainties involved in delivering the work.
The natural follow on question when you present a range as the answer to an estimate is: what would help you narrow this range? Sometimes it is "find out this thing about the state of the world" (how long will external team take to do their bit) but sometimes it is "provide better specs".
This is what I do but I don't try to make it complicated with too many numbers. "2 weeks but there's a 10% chance something bad happens and it takes longer".
I have no problem if they just hear the "2 weeks" part. If they come complaining in 3 weeks I just say "we hit that 10%".
The other important thing is to update estimates. Update people as soon as you realise you hit the 10%. Or in a better case, in a week I might be able to say it's now "1% chance of taking more than a week".
> The numbers actually have agreed-upon meaning
Theoretically, yes, but some managers go blank when given a hard concept like a probability distribution.
I've found that these incompetent folks (mostly in management as you say) really don't want to reveal their intellectual inferiority; they are desperate to hide it. If you started using this with your team, everyone adding value would benefit from the clarity, and the incompetent folks will just go along with it so as not to reveal their incompetence. They aren't adding value anyways, so it's fine if they fall behind.
I think that executives requiring estimates of time from product owners (PMs, Engineering Managers) is an instrument for putting them into de-facto 'debt' servitude, and provides a constant stream of justification for dismissal with cause. As others have commented, if the ability to time perfectly was there, it would no longer have been an innovative product. Same with requiring sales forecasts from salespeople. There's no way for the salesperson to know, so they are constantly on the chopping block for falling short of forecasts they are forced to generate. I imagine above is more or less tacitly acknowledged in tip-sharing conversations between & among execs & their investors.
[Author here]
A friend told me my post was gaining some momentum on HN and I've read through the comments and found a bunch of good insights.
I especially liked the one from @arach: "this feels like language from another era", which I hope means it's evident my post was written by someone who loves the craft and refrain from prompting an LLM for "a blog post on topic X".
I also try to decide whether I side with the numerous people who want to sort development task into either trivial or novel. Someone wrote that if we break each issue down to its components, the above holds, but idk whether the developers who does not want to estimate are keen on doing such a rigorous breakdown of each feature.
Regardless, super fun to read all the comments. You have already found my blog and feel free to connect in any way as you see fit (or not at all).
One thing frustrating for me is when folk leave $BigCo, with it's methods (ie: estimate time to complete, sprint planning) and try to apply those same methods at a very early company.
Estimates don't work there at all - everything is new.
So, flip it. Use known values to prioritize work. That is: client demand and (potential) revenue. Then allocate known time/budget to the necessary project, see how far you get, iterate. Team can move faster. Looks chaotic.
At some (uncomfortable) point however, need to rotate into the "standard" process.
There’s a well-established Agile technique that in my experience actually succeeds at producing usable estimates.
The PM and team lead write a description of a task, and the whole team reads it together, thinks about it privately, and then votes on its complexity simultaneously using a unitless Fibonacci scale: 1,2,3,5,8,13,21... There's also a 0.5 used for the complexity of literally just fixing a typo.
Because nobody reveals their number until everyone is ready, there's little anchoring, adjustment or conformity bias which are terribly detrimental to estimations.
If the votes cluster tightly, the team settles on the convergent value. If there’s a large spread, the people at the extremes explain their thinking. That’s the real value of the exercise: the outliers surface hidden assumptions, unknowns, and risks. The junior dev might be seeing something the rest of the team missed. That's's great. The team revisits the task with that new information and votes again. The cycle repeats until there’s genuine agreement.
This process works because it forces independent judgment, exposes the model-gap between team members, and prevents anchoring. It’s the only estimation approach I’ve seen that reliably produces numbers the team can stand behind.
It's important that the scores be unitless estimates of complexity, not time. How complex is this task? not How long will this task take?
One team had a rule that if a task had complexity 21, it should be broken down into smaller tasks. And that 8 meant roughly implementing a REST API endpoint of complexity.
A PM can use these complexity estimations + historical team performance to estimate time. The team is happy because they are not responsible for the PM's bad time estimation, and the PM is happy because the numbers are more accurate.
A clear description with background appears in Mike Cohn’s original writeup on Planning Poker: https://www.mountaingoatsoftware.com/agile/planning-poker
I do like the blind estimation aspect, but I don’t like:
* the arbitrary usage of the Fibonnaci sequence
* a non-clear conversion from complexity to time. Complexity and time aren’t always correlated. Some things can be easy but take a long time. Should that be a 1 or a 5?
Let’s just cut the layer of terminology, in this difficult task, and estimate in units of time directly.
The Fibonacci scale isn’t sacred; it’s just a coarse, non-linear scale that keeps people from pretending to have more precision than they actually have. Any non-linear sequence would work as long as it forces you to acknowledge uncertainty as the numbers grow.
As for “just estimate in time,” the problem is that teams are consistently bad at doing that directly. Mixing “how hard is this” with “how long will it take” collapses two separate variables: intrinsic complexity and local throughput. Story points deliberately avoid that conflation. The team’s velocity is what translates points into time at the sprint level, and that translation only stabilizes if the underlying unit is complexity rather than hours.
The whole point of the method is to strip away the illusion of precision. Time estimates look concrete, but they degrade immediately under uncertainty and project pressure. Relative complexity estimates survive discussion, converge reliably, and don’t invite the fallacy that a complex task with a high risk of surprises somehow has an exact hour count.
That’s why the technique exists. Estimating time directly sounds simpler, but in practice it produces worse forecasts because it hides uncertainty instead of exposing it.
One thing that changed my way of thinking about estimates is reading that 86% of engineering projects, regardless of what kind of engineering (chemical, infrastructure, industrial, etc) go over budget (in time or money).
Missing estimates isn't unique to software, but it's common across all engineering fields.
What's wild about software is we don't do 1/10th as much planning as "real" engineering projects do, and also find ways to create problems akin to the ones they run into from interfacing with an unpredictable real-world environment, by inventing ways for all our platforms and libraries and such to be insanely unstable.
If you're not subject to the batshit insanity of the broader software market, and you crank the planning up a little closer to what "real" engineering does, software delivery gets extremely predictable. See: NASA's Space Shuttle software development processes.
(There are actual, not self-inflicted, problems with software development that "real" engineering doesn't see, though—for one thing, you can't on a whim completely restructure a building that's already been built, and for another, you generally don't have to design real-world engineering projects to defend against the case of intentional sabotage by someone who's just e.g. using the bathroom—it may happen, but it's rare and short of wacky Mission Impossible type plans, busting a water pipe in the bathroom isn't going to get you access to super-secret documents or anything like that)
Physical real-world projects include a buffer for this, called "contingencies" or "change orders" so that if a requirement changes or they discover something like previously unknown site geology that will require changes to the foundation they can absorb it. Based on a large history of similar projects their estimates are usually pretty good but occasionally they will run over.
Even accounting for contingencies most of civil engineering projects go way over budget.
Two elements (the first quite obvious, the second not really) seem to be particularly common in overruns:
- the bigger the project the likelier the overrun. Small road projects tend to be over estimated, complex rail projects are virtually always way underestimated, mega projects are never close to the budget.
- the lengthier the planning and pre-construction phase the likelier the overrun. This is particularly interesting because it's counter intuitive: you would expect that the more analysis is done, the more accurate the estimates, but experience tells us the truth is the very opposite.
> but occasionally they will run over
86% is more than "occasionally".
My success rate went way up by only making 4 estimates.
1 day
1 week
1 month
1 year
This communicates the level of uncertainty/complexity. 5 days is way too precise. But saying 1 week is more understandable if it becomes 2 weeks.
I don’t estimate in hours or use any number other than 1
But saying 1 week is more understandable if it becomes 2 weeks.
Amazing to read this… everything wrong with our industry summed up neatly in one sentence :)
Why is that a problem? It is an estimate. Not a hard commitment.
By definition estimate can be wrong
How often you spend a week doing 2-day job?
I’d be fired if that happened without a reasonable explanation why I was so terrible at the task I took on and let my team down by taking that long
thats fantastic as long as you are not the one paying for it
I know it's easy internet points to say we should have accurate estimates, but that's not an actionable comment. That's like saying "cars should be safer" and expecting applause.
Most clients would rather you just give an estimate with 15 minutes of eyeballing it and hold you to that. That's not realistic, and it's not on developers.
As a largely solo dev I found I can't estimate well unless it's a common task, and it's easy to find tasks grow exponentially if it touches too many layers.
Asking "how long do you want me to spend on this?" got better results, because I got more idea how important tasks were to the business and can usually tell if something is going to take longer than they want. (Or know when we need to discuss scoping it back, or just abandoning the feature)
It also helps because software is one of the rare engineering fields where code is non-destrictive and can be changed after the fact.
The carpenters trick of measure many times, cut once, can instead just be cut and re-size if wrong size, which can often be quicker.
Asking how long they want me to spend on it also let's me know how solid it needs to be engineered. Is it something that needs doing right then first time, or do we just want something rough that we can refine later with feedback.
I think that that's how you solve the estimate problem - by having an understanding of how many layers it's going to touch, and how big of a change it's going to be for each layer.
I largely agree.
Another way to say this is that an estimate becomes a commitment to not learn.
Re-planning is seen as failure by [management].
Re-planning is what happens when you learn that a previous assumption was not correct.
You should be encouraging learning, as this is THE cornerstone of software development.
Re-planning in isolation is not a failure. I think re-planning is a failure when nothing is learned from previous assumptions not being correct, which usually leads to serial re-planning. I've seen this pattern form at practically every company I've worked for, in particular whenever there's a merger or a private equity buy-out. More often than not, this is ironically caused by leadership failing to learn from previous assumptions because they're too busy chasing the shiny new bobble of the week, but the blame still gets diffused across the organization.
> I think re-planning is a failure when nothing is learned from previous assumptions not being correct
This is the exact problem, and why it is so extremely difficult to communicate about the subject.
Another way to say "learn from previous [failures]" is "bureaucracy". We did something wrong last time, so we need a rule about that failure.
It SHOULD NOT be seen as failure, we SHOULD NOT add bureaucracy, simple rules CAN NOT fix the issue.
What should happen: Line managers and developers should be able to communicate about what they are learning as they do the work of software engineering. Developers should be able to re-plan their approach to problem solutions.
This simply will not work if there is not strong trust between upper management/[the money] and the engineering/execution side of the organization.
----
The meta-problem is that you have to see things go wrong many times in many ways before you really understand what is going on, and by that point ... well everything gets very tiresome.
When you make a plan, you have the least information you will ever have about the goal/problem.
Re-planning when you gain enough information to know you need to isn’t a failure — not re-planning then would be — but repeatedly finding yourself re-planning when you had or should have had that information before the initial plan is.
> SEEN as failure
emphasis added
Yes, and rightly so, sometimes.
Yes, sometimes, but negotiating this can be the whole ballgame.
I've always seen estimates as trying to guess the highest number the PO will accept, the time or effort involved in actually completing the task is irrelevant. I have never had a PO or anyone else complain that a task was completed more quickly than expected. However I do have to be careful to not tell them it is complete too early, lest they start expecting shorter cycles.
At least in my company we've stopped calling them "estimates". They are deadlines, which everyone has always treated "estimates" as anyway.
Unfortunately in the real world deadlines are necessary. The customer is not just mad that they didn't get the shiny new thing, especially in the case of B2B stuff, the customer is implementing plans and projects based on the availability of X feature on Y date. Back to the initial point, these deadlines often come down to how quickly the customer is going to be able to implement their end of the solution, if they aren't going to be ready to use the feature for six months there's no reason for us to bust our asses trying to get it out in a week.
I can't say I know many engineers who object outright to deadlines. They just get frustrated when an estimate turns into a deadline as those are different questions.
The far more common pattern is being asked to provide such an estimate off hand and those are all about what you mentioned, giving the PM whatever number you think they will accept.
In the 70's a Brazilian programmer told me his method was to make his best guess how long something would take, double it, then promise that amount plus-or-minus 50%.
The one I heard was estimate, double it, and move to the next higher unit. 2 days? Nope, that's 4 weeks.
The thing that sucks is that when I avoid giving estimates, I'm not trying to be difficult, I'm being honest about the unknowns of the project and the inherent uncertainties and messiness of software development. I'm helping protect myself and the rest of the team from making plans based off of bad estimates.
But I get all this pushback when I do that, such that the path of least resistance is to give some bullshit estimate anyway. Or I get asked to make a "rough guesstimate", which inevitably turns itself into some sort of deadline anyway.
Garbage in, garbage out. Inaccurate estimates, unreasonable timelines, stressed devs and upset PMs.
I'm so over working on software teams.
In those cases, you should be asked for estimates to eliminate the unknowns.
A couple of decades back PMs used to look at historical data to guide the estimates for a new project. If a similar coding work took 2 weeks on average in the past, that gives some basis.
So, I think the issue is about whether it is a routine workflow work which has well-tested historical timelines or not.
Nevertheless, estimates are needed at some granularity level. When you order something on Amazon, would like an estimate on when the item would be delivered to you?
Even if coding work can't be estimated, the overall project requires estimation. Someone need to commit to timelines and come under pressure. Distribution of that pressure is only fair.
> A couple of decades back PMs used to look at historical data to guide the estimates for a new project. If a similar coding work took 2 weeks on average in the past, that gives some basis.
Even that doesn’t work because the time taken isn’t just about similarity to other work, it’s about how your new feature interacts with the current state of the codebase which is not the same as when the similar feature was implemented before.
Ultimately, it’s a complexity problem that’s borderline impossible for our feeble human brains to properly understand. And we consistently misunderstand the nature of that complexity
An aircraft engine could be highly complex, but the flight times can be estimated. Complexity is not issue. Reproducibility, and similarity is missing. Every coding work became an adventure into unknown landscape, with no trekking map.
And that cavalier attitude to complexity is literally the cause of so many issues with software engineering. An aircraft engine is almost simple compared to a multimillion line codebase.
The attitude with belief that software work cannot be compared with any other work in other domains, is not going to help. There are far more complex projects out there. Complexity is not feature limited to the work artifacts. It could arise from overall context - funding, worker issues, climate, raw material, political factors, technical issues. But still, estimates can't be avoided.
> The attitude with belief that software work cannot be compared with any other work in other domains, is not going to help.
I didn't say that, at all. But it is an extreme form of abstract complexity which is different to the more tangible and physical complexity that you might have in say a jet engine. That is not to say a jet engine isn't complex, it is and obviously so; but in many ways I think most people don't fully understand the extreme complexity of a medium to large software project because, on the surface, we've apparently captured a lot of it behind reusable libraries and the like. The reality is much more nuanced due to the fact that most software engineering is not following a technique of provably sound composition of components and therefore complexity emerges and is a sum of its parts. That summing of complexity creeps up on a development team and emerges as bugs, security issues, difficulty in planning, and difficulty in maintenance, etc.
My singular argument is: reduce complexity, improve predictability.
> But still, estimates can't be avoided.
I also didn't state that estimates can be avoided. I ran a tech company for 20 years and I needed to be able to tell everyone else in the company when things are likely to be done. That's just a fact of life.
I'll give you random estimate and start working The more I work, the more the estimate can be refined
By the time my work is done, that estimate will be perfect
Wait
Perhaps my true job is to create perfect estimate ? Is coding only a side-effect ?
Businesses have many aspects of their operation that are unpredictable: ask the legal team exactly what the result of a legal action will be, and when it will be completed by, and you'll get a fuzzy answer. Ask the marketing team exactly how many new signups will result from this new ad campaign and you'll get a fuzzy answer. Businesses can easily cope with unpredictability, this is not a problem.
The problem is that software development has not been treated as one of these unpredictable operations. It absolutely is unpredictable, we know this from decades of experience and all the academic research on the subject. But, probably because of history, politics and power imbalances, software dev teams have very rarely managed to convince executive teams that software development must be treated as unpredictable. As a result, software dev as a profession has a bad reputation for not delivering on their promises.
When we do manage to convince executive teams that software development is unpredictable, good things happen. The author's point that "tax software must be released at tax time" still happens, but the planning around the specific feature set takes into account that if the release date is fixed then the feature set in the release will not be. Again, it's not a problem to deal with the unpredictability, as long as everyone accepts that it is unpredictable.
I've found the best estimates are not estimates but non negotiable deadlines with fixed outcome.
"You have six weeks to do X for $$$" or "I'll get it done in six weeks or you don't pay"
Where i work there is no penalty for being late or not hitting a deadline. Life goes on and work continues. I have seen when there are specific dates and metrics and suddenly people work in focused effort and sometimes work weekends or celebrate with finishing early.
I have never been in a project, where estimates were spot-on, and I do this for 15 years now. By now hundreds of features have floated by the river and hundreds of meetings have been held.
Estimations are often complicated, because there are way too many variables involved to give accurate estimates. Politics within companies, restructuring of teams. The customer changes their mind, the reality you've expected is slightly different, your architecture has shortcomings you find out late in a project, the teams your work depends on disband, … and a million other things.
Theoretically you could update your estimates in a SCRUM meeting, sure, but to be honest, this has always been nothing but a fantasy. We rarely do work in a void. Our features have been communicated higher up and have already been pitched to customers. In a fully transparent and open organization you might update your estimates, and try to explain this to your customers. In reality though? I have never seen this.
While this sounds very negative, my take on it is not to waste too much time on estimates. Give a range of time you expect your features to fall into, and go on with getting your work done.
I wonder how different the perception of these projects being late or (massively) over budget would be if we used different words. Bear with me here...
Words mean things. Estimate carries a certain weight. It's almost scientific sounding. Instead, we should use the word "guess".
It's exactly equivalent, but imagine the outcome if everyone in the chain, from the very serious people involved in thinking up the project, to funding the project, to prioritising and then delivering the project, all used the word "guess"
Now, when the project is millions of dollars over budget and many months/years late, no one is under any pretence that it was going to be anything else.
I tried this once. It turns out serious people don't like the idea of spending millions of dollars based on "guessing", or even letting developers "play" in order to better understand the guesses they are forced to make, even when it turns un-educated guesses into educated guesses.
Of course, none of this would improve the outcome, but at least it sets expectations appropriately.
Estimates are difficult, but if you monitor your estimate vs actual, and then adjust your next estimate accordingly, then it becomes easier to have decent estimates.
There is also the adage that if you are late, then 100% of the time the customer will be unhappy, but if you are early, then 100% of the time the customer will be pleased.
So make people more pleased :-)
So always over estimate a bit (25%)
This only works when the next thing is reasonably similar to the last thing. In my 40 years doing this it's happened exactly once.
I think the main issue with time estimates is that they don't follow a normal distribution (like many things we are taught in school) but log-normal distribution, which is very skewed and has a big right tail. This misconception is the reason people fail to understand why time estimates are inherently hard.
All estimates should be accompanied by a "cone of uncertainty". Teams should evolve their ability to describe a cone of uncertainty as well as their ability to estimate. Estimating "accurately" is only possible when understanding the accuracy is elevated to the same importance as the estimate itself.
https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_wha...
https://hubbardresearch.com/product-category/publications/ho...
I don’t agree with the given answer to “Why the PO wants you to estimate”. The author cites external reasons (made commitments etc) that shouldn’t have been there in the first place.
Sure, in some lines of software development you really have a “deadline”, if you are working towards some event or launch date. But most of the time estimates are needed to determine *if it is worth it*. Can it be built in 2 weeks, let’s go. Is it gonna cost 2 months, then let’s prioritise something else. That isn’t to say you always have to go with the shortest te build features (you can apply "CD3" – Cost of Delay Divided by Duration), but you need it to compare alternatives.
I've always found this sound, rational ROI driven approach to product management a little off the mark. Software isn't like real estate or investing in TBills - you don't invest $X in development and get a nice 10% annualized return on investment over the next 10 years or something like that despite how seductive such thinking can be.
It is largely a "hits" business where 1% of the activities that you do result in 99% of the revenues. The returns are non-linear so there should be almost no focus on the input estimation. If your feature only makes sense if it can be done in 3 months but doesn't make economic sense if it takes > 6 months - delete feature.
30 years ago my boss at a large defense/aviation contractor told me estimating software projects was a very valuable skill, but all estimates were always wrong because they are simplifications and to keep that in mind -- his words.
Mainly they are useful to build belief and keep a direction towards the goal.
Models of any kind in whatever domain are necessarily always something less than reality. That is both their value and weakness.
So estimates are models. Less than reality. Therefore we should not expect them to be useful beyond 'plans are useless, but planning is indispensable' -- I think thats' Eisenhower.
I literally filled out TPS reports every week at this place down to the hour, breaking down internal project billing and customer billing. In order get the charge codes from the PM, you need to give them estimates. That was the main driver for estimates - convince the non-coders.
It's also difficult for LLMs it seems. If I forget to add instructions to skip resource estimates, Claude will estimate a week or two, then bang it out in under an hour.
For humans, 2x the original estimate.
If you know your team's velocity, this methodology has worked well for me in the past:
- ask enough questions about the scope of work to do a prototype
- build a prototype to identify constraints / complexity
- ask followup questions
- break the body of work down into a list of tickets (spike/discovery tickets included)
- use the PERT formula to estimate the total number of story points for the body of work
- take the total estimated points and divide by your team velocity for a duration of time (e.g. 100 total points / 40 points per 2 weeks = 2.5 weeks)
I tried to search for the CMMI keyword in the whole discussion to find something relatable. Guess like everything else people happen to loose memory over a cycle and discover it again as if thats something new. Another thing I realize from this is that I am probably getting older.
> Why does the backlog need to be prioritised?
We know _why_ you want to know how long everything will take. It seems to escape these guy's attention that if software development was as predictable as they seem to insist it must be, they could be replaced with a spreadsheet.
"Predictions are hard; especially about the future." –Yogi Berra, MLB HoF catcher & manager
I've tried running teams with and without estimates and I've noticed that when work isn't estimated it tends to drag on forever. But when you put a number on it, even if it's a rough or inaccurate guess, it gets done much faster.
In my experience, most of the time estimates are difficult because prerequisites and constraints are not clear or not fully known.
But, if you are enough experienced, with some fermi-style math, you will do good enough estimates most of the time.
Giving an estimate is not hard and shouldn't be a pain if if was truly an "estimate" PMs were asking for. But after a while you learn that when they say "What's the estimate for feature F?", they're in fact saying "What deadline you wish to impose on yourself for feature F?". Then the next interaction is most likely to be "Are you done yet?" and not "Is the estimate holding up?".
I don’t think I dislike being forced to estimate. I dislike being asked to estimate by the same people that consistently make those estimates worthless by introducing changes at literally every point in the process.
That cuts both ways though. You may hand out an estimate and then once you get into the research/POC weeds need to revise that estimate.
My experience is that I have to basically always overestimate if I can get away with it because otherwise if something goes wrong, I will pushed to do free overtime to complete all the work assigned in a given sprint.
https://youtu.be/QVBlnCTu9Ms?si=k_UolNc2o6UFGS9f
#NoEstimates
Yes, be agile. Yes, measure all the things.
But estimation sets everyone up for disappointment.
It is often not just about the difficulty of estimation the time for specific tasks but also what the assumptions put in were. Sometimes these assumptions werent written out explicitly or other times they were (deliberately?) removed. Just one example of a broken example: a project can be run in many ways. You can have an estimate done based on A-team resources and high priority but the moment the contract or whatever is done, it is decided to outside the whole work to a new team who never worked on the code before and sit thousands of miles away. To compensate 2-3x as many people are assigned. Add in a non technical project manager and scrum master and all kinds of resources that were never envisaged but who will report time in on the project etc. You get the idea. And this was just done type of assumption that could broken!
Maybe software industry should learn from other industries where people don't see estimates as something unthinkable?
I have read about the rule of multiplying any time estimate by 1.57, but ime it should be multiplied by 7.85 rather.
I have a solution but I don't think companies care about this level of meta-analysis on how people work together. They think they do but in reality they just care about optics, and the status quo culture has a huge weight on continuing in the same direction, largely dictated by industry "standards".
In essence, estimates are useless. There should only be deadlines and the confidence of engineers of achieving the deadline. To the extent there are estimates, it should be an external observation on the part of PMs and POs based not only on the past but also on knowledge of how each team member performs. This of course only works if engineers are ONLY focusing on technical tasks, not creating tickets or doing planning. The main point of failure in an abstract sense is making people estimate or analyze their own work, which comes with a bias. This bias needs to be eliminated and at the same time you give engineers the opportunity to optimize their workflows and maximize their output.
TLDR, engineers should only focus on strictly technical because it allows to optimize within the domain, meanwhile other roles (whoever PM, PO or other) should be creating tasks, and estimating. Of course this doesn't work because there are hard biases in the industry they are hard to break.
It's a shame that developers read these articles and not project managers.
https://en.wikipedia.org/wiki/Delphi_method > https://news.ycombinator.com/item?id=26368011
https://blog.pragmaticengineer.com/yes-you-should-estimate/ > https://news.ycombinator.com/item?id=27006853
https://josephmate.github.io/PowersOf2/ Complexity Estimator
https://earthly.dev/blog/thought-leaders/ > https://news.ycombinator.com/item?id=27467999
https://jacobian.org/2021/may/20/estimation/ > https://news.ycombinator.com/item?id=27687265
https://tomrussell.co.uk/writing/2021/07/19/estimating-large... > https://news.ycombinator.com/item?id=27906886
https://www.scalablepath.com/blog/software-project-estimatio...
https://estinator.dk/ > https://news.ycombinator.com/item?id=28104934
https://news.ycombinator.com/item?id=28662856 How do you do estimates in 2021?
https://web.archive.org/web/20170603123809/http://www.tuicoo... Always Multiply Your Estimates by π > https://news.ycombinator.com/item?id=28667174
https://lucasfcosta.com/2021/09/20/monte-carlo-forecasts.htm... > https://news.ycombinator.com/item?id=28769331
https://tinkeredthinking.com/index.php?id=833 > https://news.ycombinator.com/item?id=28955154
https://blog.abhi.se/on-impact-effort-prioritization > https://news.ycombinator.com/item?id=28979210
https://www.shubhro.com/2022/01/30/hacks-engineering-estimat...
https://www.paepper.com/blog/posts/monte-carlo-for-better-ti...
https://drmaciver.substack.com/p/task-estimation-101 > https://news.ycombinator.com/item?id=32177425
https://morris.github.io/e7/#?t=
https://stevemcconnell.com/17-theses-software-estimation-exp...
https://www.doomcheck.com/ > https://news.ycombinator.com/item?id=34440872
https://github.com/kimmobrunfeldt/git-hours
https://pm.stackexchange.com/questions/34768/why-are-develop... > https://news.ycombinator.com/item?id=35316808
https://erikbern.com/2019/04/15/why-software-projects-take-l... > https://news.ycombinator.com/item?id=36720573
https://news.ycombinator.com/item?id=42173575
https://www.thecaringtechie.com/p/8-guaranteed-ways-to-annoy... > https://news.ycombinator.com/item?id=43146871
https://news.ycombinator.com/item?id=43873604
<https://www.joelonsoftware.com/2000/03/29/painless-software-...>
I preffered: https://www.joelonsoftware.com/2007/10/26/evidence-based-sch...
20-25 years later, this is still where we are at :)
"Evidence-based" stats tracking is the only somewhat scientific method i know of that can give you a good range of estimates. But it takes effort. Thus why many don't do it.
But accurate estimate ranges are a super-power for businesses if they can trust them. Never understood why that's not demanded more... oh yeah, it's work :p
I always say, "it's a prediction, not a promise."
Prototyping goes a long way in helping come up with estimates.
estimates are only difficult cause entire industry has convinced everyone that we do something special which we have no idea how long it will take to do :)
An LLM's guess is as good as anyone's.
Estimates are like traveling from point A to point B in completely unpredictable terrain. Sometimes you can just go straight and beat the estimate easy. Sometimes you will find massive valley and you need to build a bridge to continue while your PM is furiously yelling in the radio that estimated deadline was a week ago.
I will often estimate 3x as a rule of thumb. Estimating is indeed difficult especially when dealing with unforeseen task switching.
I was surprised to not see “story points” mentioned in the context of scrum. This is an estimation concept I have found baffling because it is a measure of “complexity”, not time, yet is used directly to estimate how much can be done. At least this is how it is done at my work.
Scrum lingo is silly.
scrum lingo is silly because scrum itself is stupidity squared and hence the lingo also cannot be anything but that
this feels like language from another era
I’m afraid I don’t follow. Would you mind to elaborate a bit?
Companies that need estimates have an Estimations department.
These are usually companies that are led by and perform engineering work.
Software developers aren’t engineers.
Project managers have no authoritative training, certification or skills to manage software development projects.
[dead]