Accurate Task Prediction – the Developer’s Holy Grail

By Dan Y.

Oct 1, 2017

Software task estimation is the process of predicting the time and effort required to complete tasks. Generally, these estimates are used to help customers and managers make better decisions regarding how long it will take to accomplish a given project.

It’s known that programmers often consider task estimation to be one of the most difficult things they do, which means they consistently find themselves with inaccurate or underestimated task predictions. To handle this problem, they pad their estimates but in many cases their rough guesses are too low.

Regrettably, developers think that estimation is literally the equivalent of making a weather forecast. We can’t expect a forecaster to predict rain in a month from now but still, it’s useful when making a shorter term forecast.

Software estimation is a process we make to predict the time and the effort required to complete development tasks. Generally, it helps us to make ongoing decisions and predict the project’s deadline.

From a 10,000-foot bird’s eye view, estimation allows decision makers to draw up a road map and determine the time-to-completion of a given project. Above all of that, it allows for better understanding what is the predicted cost of the project and what is the time frame for it going to market.

Why it’s so hard to estimate?

There are several factors that lead to skewed software estimates. From my own experience, I’ve seen all of the following (but I’m sure there’s more):

  • Tasks granularly and non-specific declarations. Task parameters are too ambiguous which has led to a lot of debate on how it should be best implemented. Sometimes the requirements are not clear enough or there has not been an in-depth analysis. From my point of view, this is the most common problem.
  • The task in question creates new tasks (nested tasks). This is common with research tasks whose outcomes include some sort of solution to be built in the sprint.
  • Estimates made for unfamiliar technologies. This is a trap for developers who inaccurately estimate the time-to-completion based on past experience with disparate platforms.
  • Working on unrelated tasks This is also called context-switching and refers to when the tasks assigned to a developer are unrelated, forcing them to “spin up” for each one.
  • New features that are estimated may break some assumptions a developer made in previous implementations Sometimes the developer is not aware of those assumptions; sometimes a refactor is performed or not considered when estimating.
  • Tasks with dependencies. Developers become immediately blocked because precursor tasks are unfinished. This is especially problematic if the dependencies need to be finished by multiple people.
  • Research tasks with non-tangible outcomes. I’ve seen people take weeks to research/prototype some solution. If you don’t constrain ambiguous tasks, you’re inviting the task-taker to waste time.
  • One person estimates for another This is common when a developer is absent during the planning stage and another does the estimation for them.
  • Peer pressure. Engineers sometimes pressure each other to accomplish a task faster than believed. This generally happens in high-stress environments when a stronger personality bullies a developer into working harder.
  • External pressure Often there is pressure by management to estimate either less or more time-to-completion because of a political or financial agenda.
  • Other tasks You have other things to do from past assignments and you will have to come up with an estimate that takes this other work into account.
  • ** The ‘done’ definition is probably unclear.** When will it be done? ‘Done’ as in just finished coding it, or ‘done’ as in “users are using it”?
  • The programmer’s pride. No matter how conscious you are of all these things, sometimes your “programmer’s pride” makes you give or accept shorter times than you originally suppose it might take, especially when you feel the pressure of deadlines and management expectations.

None of these factors should surprise you. In fact, our software estimation processes are designed to deal with them. Our solution is to apply psychology and mathematical models to develop better estimates. In fact, I feel like we waste more effort (per capita) than any other industry analyzing our own development practices in seek better efficiency.

Are there any ways to deal with estimation?

There are a number of estimation methodologies to choose from—and here we’re going to look at four tried-and-trusted ones that work for all types of projects.

Before listing the methodologies, I would like to emphasize the fact that most of the techniques are based on past estimations and performance. But how do we save and store past data? There are techniques to do it.

At Spectory, in the planning meeting, the developers estimate the required hours to complete a task. The time estimate is saved on a task card using a predefined identity. So now we have the original time estimation. What about the actual time required to complete the task? The relevant and the actual data is collected across the board so we can easily calculate the difference and derive the final estimation for a given feature.

Analogous Estimating

In this method, the estimation is based on past data. Identical tasks that have been implemented in the past can be used as a reference.

For example: you need to implement an unsubscribe mechanism that can be applied on all of the emails sent from your system, It would then send a notification to the super user. In a previous project implementing an unsubscribe mechanism took 25 hours to develop and you estimate it would take about 5 more hours to implement and test the notification feature.

In case you don’t have sufficient past data, you can consult a developer who has experience in this area. In this case, the initial estimate needs to go up by 20-30% depending on the complexity of the feature.

Three-point estimation

In this method, you should set a three-point estimate for a given task.

A – the best case scenario or the most optimistic estimate
M – the most likely estimate
B – The worst-case estimate

In our example, to implement the unsubscribe mechanism, it will take 20 hours for the best-case scenario. The optimistic estimate for completing the task is 22 hours and the worst case is 30 hours.

Now, to get the desired estimation, you can use one of the following formulas:

  • The simplest: ET = (a + m + b) / 3 so we will get (20+22+30)/3 = 24 hours.
  • To get more accurate use: ET = (a + 4m + b) / 6 will give us 23 hours.
  • The most complex that takes the standard deviation into account:
    E = (a + 4m + b) / 6 in our case we get 23 hours
    SD = (b – a) / 6 - we will get 1.6
    ET = E + 2 x SD - we will get 23 + 2*1.6 = 26.2 hours
  • The Three-point estimation is the most accurate technique that reduces the risk of overly optimistic or inflated estimates.

Bottom-Up Estimating

Bottom up estimating is the process of estimating individual sub tasks and rolling them up into an overall task estimate.

This technique is used in conjunction with each or any of the other techniques. The three estimate types are used on individual sub tasks, which are then “rolled up” into the overall task.

Parametric Estimating

This technique is similar to analogous estimating but adds an additional layer of accuracy by dividing the task into smaller, measurable units.

Finally, how do I become a great estimator? Here are some estimation guidelines:

Good estimates can help you determine which proposed set of features might fit a budget or help determine the order of magnitude of the price for a piece of software. The moment a project starts, the estimates and reality of completing a task will start to diverge. Insights and features change, so the estimates should change with them in order to be somewhat accurate.

Use these steps to make accurate estimates:

Understand What’s Required

  • What is the goal of the task and what is the required scenario?
  • Start by identifying all of the work that needs to be done for the estimated task.

Order the sub tasks

  • List all of the activities you identified in the order in which they need to be developed.
  • In this stage you don’t need to provide an estimate.

Decide who you need to involve

  • Brainstorm about the task with others
  • Get help from people who have prior experience with the task.

Make your estimates

  • Choose one of the methods described above.
  • Estimate the time needed for each sub task rather than for the overall task.

Summary

I heard many developers say they don’t need to estimate due to the fact that the project will anyway be off by a factor of two or three.

I’m afraid to say it but they are totally wrong! Time estimation is a major part of professionalism as well as a key component of any software development project.

We should aim to achieve the most accurate time estimation possible. By using the estimation techniques mentioned previously along side past analytical data we can find the most accurate and realistic estimation.

Leave a Reply

Your email address will not be published.