The merging process involves removing duplicate tasks and reaching some reasonable resolution of different estimates for individual tasks. “Reasonable” doesn’t mean replacing the team’s estimates with values the project manager prefers. Large estimate differences for apparently similar tasks might indicate that estimators interpreted that task in different ways. For example, two people might both have a task called “implement a class.” However, one estimator might have included unit testing and code review in the task, while the other meant just the coding effort. All estimators should define their tasks clearly to minimize confusion during this merging step. The merging step should retain the estimate range for the tasks. If one estimator’s task estimate was wildly different from that of the other estimators, understand it and then perhaps discard or modify it.
In the final step, the estimation team reviews the summarized results and reaches agreement on the final outcome. The project manager provides the other estimators with the overall task list, individual estimates, cumulative estimates, assumption list, and any other information. Bring the team back together for a brief review meeting to bring closure to the estimation activity. This meeting also provides an opportunity for the team to contemplate this execution of the Wideband Delphi process—a retrospective—and suggest ways they can improve it for future applications.
The participants should make sure the final task list is as complete as possible. They might have thought of additional tasks since the estimation meeting, which they could add to the task list now. Check to see whether tasks that had wildly different individual estimates have been merged in a sensible way. The ultimate objective is to produce an estimate range that allows the project manager and other key stakeholders to proceed with project planning and execution at an acceptable confidence level.
Completing the Estimation
The estimation process is completed when specified exit criteria are satisfied. Exit criteria help you determine when a process execution is done so you can declare victory and move on with your life. Typical Wideband Delphi exit criteria are that:
The overall task list has been assembled.
You have a summarized list of estimating assumptions.
The estimators have reached consensus on how their individual estimates were synthesized into a single set with an acceptable range.
Now you must decide what to do with the data. You could simply average the final estimates to come up with a single point estimate, which is what the person who requested the estimate probably wants to hear.
However, a simple average is likely to be too low, and there is merit in retaining the estimate range. Estimates are predictions of the future, and the range reflects the inherent fuzziness of gazing into the crystal ball.
Following are several ways to present the results from the Delphi session, using the five estimates from Round 3 (shown in Figure 4 of the second article in this series) as an example.
- Present just the average of the final individual estimates. (example: 1974 hours)
- Present a single estimate calculated as follows (example: 1966 hours):
- Present a range, with the average of the final estimates as the planned case, the minimum value as the best case, and the maximum value as the worst case. (example: 1974 hours planned, 1650 hours best case, 2250 hours worst case)
- Present a range, with the average as the planned case, the upper bound being (maximum value – average) and the lower bound being (average – minimum value). (example: 1974 hours, +276 hours, –324 hours)
Each estimate has a certain probability of coming true, so a set of estimates forms a probability distribution. There are mathematically precise ways to combine multiple estimates and their uncertainties to generate an overall estimate with upper and lower prediction intervals (Watts Humphrey, A Discipline for Software Engineering, Addison-Wesley, 1995). Another sophisticated approach is to perform a Monte Carlo simulation to generate a probability distribution of possible estimate outcomes based on the final estimate values; see Nicholas A. Campanis, “Delphi: Not the Greek Oracle, but Close,” PM Network, vol. 11, no. 2 (Feb. 1997), pp. 46-49.
The results of a Delphi session might not be what the movers and shakers want to hear. But this information helps them determine whether they want to plan their project at a 10 percent confidence level, a 90 percent confidence level or somewhere in-between. Be sure to compare the actual project results to your estimates to improve your future estimating accuracy.
Wideband Delphi Evaluated
No estimation method is perfect; if it were, it would be called prediction, not estimation. However, the Wideband Delphi technique incorporates some solid estimating principles. The team approach acknowledges the value of combining multiple expert perspectives. The range of estimates produced reflects the variability intrinsic to the estimation process. Steve McConnell has found that Wideband Delphi can cut estimation errors by 40% to 60% (Software Estimation, Microsoft Press, 2006). It’s particularly useful in avoiding very large estimation errors.
Although it takes time and requires a panel of experienced estimators, Wideband Delphi removes some of the politics from estimation and filters out extreme initial values. This approach illustrates my philosophy of the correct answer to any request for an estimate: “Let me get back to you on that.”
Don't forget to leave your comments below.