Wednesday, 12 October 2011 10:50

Things Known and Unknown

Written by 
Rate this item
(1 Vote)

On every project there are things we know and things we don’t know – Knowns and Unknowns.  Organizing your thoughts around those concepts can be a constructive approach to understanding a project as shown in the matrix below:

Knowns Unknowns
Knowns Known Knowns

Things in our plan
Known Unknowns

Things we know we don’t know
Unknowns Unknown Knowns

Assumptions
Unknowns Unknowns

??

The Known Knowns you handle via the plan, but what about those various flavors of Unknowns?  How do you normally account for those things in the project?  Often it’s with padding – estimates that include unidentified amounts of time and/or money just in case

Let’s review how padding works: You ask your tech lead, Renee, for an estimate: “Excuse me, I need an estimate for that activity you’ll be doing.”  Now Renee may be thinking to herself “That will take me about 40 hours.”  But Renee doesn’t tell you that.  She may very likely tell you “Um, that’ll take about 60 hours.”

Why would she do this and, more importantly, so what?  First, she’s doing it to cover her Unknowns. And that is absolutely understandable. Stuff happens – Unknowns!  So what’s the problem?

Padding undermines sound project management practice in three ways:

1.   It undermines trust. The notion that it’s a good idea to under promise and over deliver may work once or twice, but over time padding undermines the trust between the PM and the team and the rest of the stakeholders.  Honest questions should inspire honest answers.  That’s how you foster partnerships.

2.   Have closets, will fill.  If Renee says 60 hours, she may very well take 60 hours and that’s opportunity cost.  She isn’t able to be allocated to other efforts. 

3.   Padding undermines the opportunity to learn from our experience, which in many ways is the essence of why we seek to develop good project management habits.  What you want to be able to do at the end of a project is turn around and look at the journey you took to get there and be able to take something away for next time. 

If you pad estimates and quite possibly end up with nonsense for baselines, what do you learn from that?  Certainly not as much as you could have had you started with meaningful estimates to begin with. 

Let’s say Renee is thinking 40 hours, she tells us 60, and then she comes in at 50.  What do you think?  Renee’s a star!   But what really happened?  She was 25% over!  If you are measuring against meaningless numbers, you don’t get much of a chance to make the PM journey a learning experience. 

Importantly, the intention here is not to highlight Renee’s failures.  Rather, the purpose is to identify and understand the Unknowns that derailed the work and consider how to avert those things in the future or plan for them next time.

This is all well and good, but Renee still has her Unknowns to contend with, so what might you do to partner with her to address those? 

The better project management answer to Unknowns is contingency reserves, an amount of money in the budget or time in the schedule seen and approved by management.  It is documented.  It is measured and therefore managed.  You draw from it where and when you need to and then learn from that, as well. 

The purpose of the contingency reserve is not to 100% cover everyone’s Unknowns, but rather to reduce them to a level that is acceptable to the stakeholders.  So maybe it’s 10% of the project total if stakeholders aren’t too risk averse and you feel bullish about what you know and don’t know.  Or maybe it’s 40% on a project with highly risk adverse stakeholders and a big box of Unknown Unknowns.

Knowns we can plan for.  The Unknowns?  We have two choices for how to deal with those.  Under the table with padding which precludes any discussion, measurement, or management of what contributes to the Unknowns.  Or Contingency Reserve in which  case we talk about them, document them, measure and manage them.  Only one way really enables us to learn from them.

Don't forget to leave your comments below.


Andrea Brockmeier is the Client Solutions Director for Project Management at Watermark Learning.  Andrea is a PMP® as well as Certified ScrumMaster.  She has 20+ years of experience in project management practice and training. She writes and teaches courses in project management, including PMP® certification, as well as influencing skills. She has long been involved with the PMI® chapter in Minnesota where she was a member of the certification team for over eight years. She has a master’s degree in cultural anthropology and is particularly interested in the impact of social media and new technologies on organizations and projects.

Read 5617 times Last modified on Wednesday, 12 October 2011 13:13
Andrea Brockmeier

Andrea Brockmeier is the Client Solutions Director for Project Management at Watermark Learning. She has 20+ years of experience in project management practice and training. She writes and teaches courses in project management, including PMP® certification, as well as influencing skills. She has long been involved with the PMI® chapter in Minnesota where she was a member of the certification team for over eight years. She has a master's degree in cultural anthropology and is particularly interested in the impact of social media and new technologies on organizations and projects.

Comments  

 
0 # Yannik Klich 2011-11-13 22:41
Hi, You make it seem that estimating the amount of work for a task is something that can be done accurate and that padding is added as a safety net. But in software projects I noticed that the more experienced the estimator, the more extra time is 'padded' to the estimates. Lets say a senior dev has is making an estimation for a software integration. He thinks that he can do it in 10 days, he adds another 10 days for unforeseen complications (unknown knowns). That's actually adding reserve for risk on a task level. An I think that's better than managing those risks on a project level, isn't it? This leaves room on the risk register for more tangible risks... I find that usually these padded estimates closer to the real executed work! I'm actually interested in finding out why junior people tend to 'underestimate' work... I only have experience in software projects, it might be a different story in construction or any other industry...
Reply | Reply with quote | Quote
 
 
0 # Ton Dekkers 2011-11-13 22:51
What's described is a common practive where estimates are based on activity based expert judgement. It is complaint to what is in PMBoK Chapter 6 (7). However in chapter 6.4.2 - 3 / 4 are mentioned parametric estimation and three point estimates. Combining these options you can mitigate the risk of covering because estimates become fact based instead of judgement based. Based on historical performance a reliable estimate is produced. In conjuction with that also a probality is given. You can use the vaiance between 50% and 80% as a contingency/res erve. Witth that all is transparent. I'm still surprised why this approach is used so limited. I'm very happy that DACE (Dutch association of Cost Engineers) recognised this and started a SIG parametric estimation in collaboration with ISPA (International Society of Parametric Analyst). In the the market there are several tools that support this kind of approach, this to illustrate there are best practises.
Reply | Reply with quote | Quote
 
 
0 # Phil Armour 2011-11-13 23:06
Three of these known and unknown dimensions I characterized as Zero, First and Second Order Ignorance (0OI, 1OI, and 2OI) in The Five Orders of Ignorance Communications of the ACM Vol 43 No 10. October 2000. [PDF of the article is available from a link at http:// http://www.corvusintl.com/CACM002-5OI.htm]. The 5OI identify two other types of ignorance one of which (3OI) is a process level where I don’t know I don’t know something and I don’t know a (suitable efficient) way to find out that I don’t know I don’t know something. This is a really important level in software development and is seen in things like ineffective testing, poor inspections, inappropriate use of lifecycles, etc. Heck, it’s even seen in bad estimation practices. Almost all software methods, particularly things like testing, are 3OI processes whose primary purpose is to expose our lack of knowledge. The bottom left quadrant doesn’t really exist in a practical sense. It implies I know something, but I don’t know I know it. How did this happen? Did I forget? If so, I really don’t know it and I’m going to have to figure it out again. Do I not recognize the problem as something I really already know? If so, there are things I clearly don’t know about my problem recognition process. “Padding” at the low level of organizations and “contingency” at the high level are both attempts to do the same thing: allow for risk. You rightly point out that the first is relatively uncontrolled since it is covert while the second should be overt and should be better controlled. However, to be truly managed we have to empirically and quantitatively determine the risk and then calculate and balance the cost of risk resourcing versus return at risk. This is an actuarial process and it’s pretty much what an insurance company does when they quote an insurance policy. T
Reply | Reply with quote | Quote
 
 
0 # Curtis 2012-04-30 18:06
Phil, I disagree with your statement about the lower left quadrant 'not existing'. Of course it does. Items that would fit into the Known Unknowns field would be: 1) we have been told to use X technology because that is on the product roadmap, but we don't know if it will be compatible, so we have to try it to find out. If it works, we estimate Y hours, if not, we have to find a new solution. Or 2) we have to build X solution but no one on the team has the experience and skills in that area, we have to hire someone, so at this time we don't know how big that effort will be. Or: 3) the previous expert in that area left the company and another developer will have to learn that piece of code, so we are unsure how long it will take him/her to get up to speed.

Those "known unknowns" do arise with some regularity.
Reply | Reply with quote | Quote
 
 
0 # Timour Khachtchevatsky 2011-11-14 00:43
First of all when Renee gives a 60h estimation PM should ask her: is it the most probable value or "safe" (pessimistic) value. If PM is not doing that - it is clearly his fault. And of cause PERT and risks analysis are good practices. And of cause if you don't fully trust one expert you should use panel of experts (for example with Wideband Delphi method). So don't try to improve Renee - improve your process.
Reply | Reply with quote | Quote
 
 
0 # T Mellor 2011-11-14 01:02
This whole padding discussion goes to the absurdity of estimating in the first place. Sure, people want estimates and we are driven to live and die by them. The fact of the matter is that a single number estimate is simply wrong 99% of the time. If we could be honest and estimate a range with a confidence level, then the estimates we provide would have an acceptable level of veracity to them. If the author is really a CSM and understands agile, then she understands that there is no PM role in Scrum. I have heard often that orgs simply cannot do away with the role, but I also know plenty who have and they are operating very effectively. In the case above, the PM simply drives dysfunctional behavior in the programmers and they simply become apathetic about the whole estimation business with good reason. Sizing work through relative estimating rather than time estimating, breaking work down to the smallest possible level, delivering the work incrementally, and measuring the velocity and commensurate cost is a much better method of dealing with this estimating debacle. Using the actual historical data of iterations is a much better way to begin to provide a reasonable level of confidence in delivery of the product.
Reply | Reply with quote | Quote
 
 
0 # Curtis 2012-04-30 18:14
To begin with, agile is not a universal framework applied to project management. So to scoff at a discussion around estimations is simply not helpful, and not realistic. Secondly, I marvel at the Scrum mentality that "we emply Scrum so we don't need a PMO". I've worked in 5 different Scrum environments, and those without a PMO often had difficulties managing the longer term planning for larger sized projects that crossed teams and required very careful planning and synchronization . Teams that integrated PMs over scrum masters had better success at coordinating the work.
But many scrum practitioners are on dedicated Product Teams where the backlog comes from Product Owners deciding what needs to be developed. In entrepreneurial environments, where Sales teams work with external clients to sell enhancements, Statements of Work are developed with a fee for work based upon an estimate. Over-estimating could lose opportunities. Under-estimatin g either cuts into profits or causes losses. Relative sizing only works where a particular team works consistently in the area being estimated and can base their estimate off previous experience. This is not always possible in a matrixed environment where teams are shuffled about as work is prioritized.
Reply | Reply with quote | Quote
 
 
0 # Darrel Raynor 2012-08-31 09:13
Ah, there is a PM role needed in Scrum ;-> The highest usefulness of a PM in Scrum is to remove blocks put up by *the rest* of the organization! The PM may also need to translate burndown and other metrics into those that "feed the machine" rather than take the team's time in status meetings. PMs can also be very helpful with the scores of stakeholders with influence but that are not in the product owner's organization.
Reply | Reply with quote | Quote
 
 
0 # Andrea Brockmeier 2011-11-15 04:11
Yannik, Yes, you can add reserve at the project, phase, or even activity level. What can be problematic is that “padding” implies that there isn’t a discussion about the risks that result in the padded estimate. Without the discussion, we miss the opportunity identify and, possibly, manage the risks that could have been illuminated with a more transparent discussion. Thanks for the comment!
Reply | Reply with quote | Quote
 
 
0 # Andrea Brockmeier 2011-11-15 04:12
Phil, Thank you for the reference. The Unknown Knowns, or assumptions, is just a way to think about those thing we think we know, but may not know. So I “know” it, but not for certain; until I know more or something to the contrary, I am going to say it’s true. It may not stand up to a rigorous logic argument as shown in the diagram, but I've found it helpful to organize the group’s thoughts. Assumptions are good places to start when identifying risks. Thanks for your thoughts.
Reply | Reply with quote | Quote
 
 
0 # Andrea Brockmeier 2011-11-15 04:13
Timour, I completely agree. Let’s not beat up on Renee – let’s engage in a more substantive discussion around what we know and don’t in order to elicit risks lurking in the estimates. And, yes, those other techniques can be very helpful. Thanks so much for the comment.
Reply | Reply with quote | Quote
 
 
0 # Andrea Brockmeier 2011-11-15 04:15
T, Thanks for the comment. Good observation about Scrum. My comment was pertinent to a more traditional approach as opposed to iterative. I think you are right on with the benefits of the latter. Obviously, not all projects are amenable to Scrum. Where a more traditional approach is used, there is usually opportunity for having a better discussion around risk and the implications for estimates. To the extent that organizations want to change the discussion and take advantage of the opportunity to get away from the game of estimating, the quadrants I mention can be a way to frame up the discussion. Finally, as you know, any estimate, by definition, should be a range. Thoughts around what that range needs to be can be teased out in this manner, as well. Thanks for your thoughts.
Reply | Reply with quote | Quote
 
 
0 # Tom 2011-11-22 00:12
I have worked in organizations that rate performance based on meeting estimate numbers, rather than on improvement over time (or some other measure). In such an environment, metrics are used against the team doing the work, so padding is a rational defensive behavior. It does not help develop better estimates over time, but is the way "management" does business.
Reply | Reply with quote | Quote
 
 
0 # Phil Armour 2011-11-26 23:02
USEFUL, NOT "ACCURATE" The phrase "accurate estimate" is an oxymoron. As a few people have acknowledged, an estimate unaccompanied by a probability is simply incomplete and possibly very dangerous. A better criterion for evaluating an estimate is its *usefulness*--d oes it significantly help in making a business decision *right now*? That an estimate will be "better" when we have more information is obvious. A 100% accurate estimate can only be achieved when the project is finished and there is no more uncertainty; it is accurate, but it is not useful--the need for decisions is past. An estimate that states "this project will take between one week and 15 years" is clearly accurate, but it is not useful--it does not help to make a business decision. The same prediction problems occur in the stock market. Post hoc I can tell you exactly what companies you should have invested in; but it is not very useful. The challenge is supporting the business decision-making process *in advance*. Empl oying Scrum is the equivalent of investing small amounts in a range of equities over time rather than one big sum in one stock now. Both mechanisms exist to manage risk.
Reply | Reply with quote | Quote
 
 
0 # Andrea Brockmeier 2011-11-28 00:41
Tom, I would agree that padding is absolutely rational. If the stakeholders, including management, are satisfied with how the estimating works and their definition of success is met at the end, then it sounds like what you're doing is working great. If the missed opportunity to improve the estimates becomes more problematic, then maybe a change in approach is merited. As I say about anything in project management: If what you're doing is delivering your stakeholders' definition of success, even if it flies in the face of best practice, then keep doing it!
Reply | Reply with quote | Quote
 
 
0 # Andrea Brockmeier 2011-11-28 00:51
Phil, I, too, tell students that the idea of an "accurate estimate" is oxymoronic, but I like your discussion to make the point. An estimate as a range implies what you're suggesting - that as you know more, the range narrows - but your comment really makes it clear. Thanks!
Reply | Reply with quote | Quote
 
 
0 # thegrumpyprojectmanager 2011-11-29 03:26
Calculating estimates is simple, when using PERT method. Instead of one time estimate, calculated from: Optimistic time O (1% chance/3 stand. devs) Most likely time M Pessimistic time P (1% chance/3 stand. devs) -->Exp ected execution time = (O + 4xM + P) /6 , Easy! In a smaller job with proper WBS it is possible to have thin pads. However proper change management is needed in all cases. The more vague the task, the thicker paddings (even though we may end up having no paddings if guessing wrong). And somewhere there is a limit, when we need to invoice by hour like suggested in an earlier comment. Peopl e tend to have smaller estimates for the tasks they are interested in doing, and thick paddings for the boring ones or ones they don't favor. Especially in large business projects this gives considerable political power. But this is already kind of out of the subject. Regar ds, Grumpy L:^| ] http://thegrumpyprojectmanager.blogspot.com/
Reply | Reply with quote | Quote
 
 
0 # Curtis 2012-04-30 18:19
Andrea, I enjoyed your article, and chuckled a bit when I used the same "Known Knowns, Known Unknowns, and Unknown Unknowns" once because the phrase elicited a rather cynical reference to Donald Rumsfeld from someone who was not, to say the least, a fan of his!
As for "known unknowns", you are quite right, they not only exist, but are quite common. The best example arises from a project I worked on when chief architect unilaterally decided that X technology was the way of the future and it had to be built in that technology, when no one knew if it would integrate with the existing system, and no one in the team had ever worked with it.
Reply | Reply with quote | Quote
 
 
0 # Andrea Brockmeier 2012-04-30 21:10
Curtis,
I remember the Rumsfeld quote, too. I really like your example of the Known Unknown, which I continue to find helpful.

Thanks so much for all your thoughtful comments today!
Andrea
Reply | Reply with quote | Quote
 
 
0 # BradDet 2012-06-03 18:24
Hi

Some thoughts - to keep good team between the estimator and the pm, I think the pm should always ask for 2 estimates for each task. The first estimate is the "I think it could be done in "40hrs" if everything goes well. They only need to be about 50% confident in this estimate. They should then give the "but it might take as long as 60hrs" if I run into unforeseen difficulties, they should be 90/95% confident in this estimate.

The pm then schedules the plan according to the lower estimate, and use the difference between the two in a buffer near the end of the project, this would be the "contingency reserve".

I use critical chain, and assign buffers that are half the difference of the sum of both estimates, and have found it to be very effective in leading projects to a goal end date.

It is also still possible to compare estimates and actuals, though I am not really satisfied that one's ability to "estimate accurately" means they are more senior at doing the role to which they have been assigned, and therefore I don't think it should be a strong measure of individual performance.
Reply | Reply with quote | Quote
 
 
0 # Andrea Brockmeier 2012-06-12 10:39
Good idea! Thanks for sharing.
Reply | Reply with quote | Quote
 
 
0 # JW Herman 2013-04-03 09:35
A nice twist to the only Donald Rumsfeld quote (when speaking to the Dept of Defense) which is also available as a video clip.

There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we know we don't know. But there are also unknown unknowns. There are things we don't know we don't know.

Donald Rumsfeld


Read more at http://www.brainyquote.com/quotes/quotes/d/donaldrums148142.html#BxPyP5SZ0UF7kzBU.99
Reply | Reply with quote | Quote
 
 
0 # JW Herman 2013-04-05 08:21
One further comment. Known Unknowns are (generally) risks and Unknown unknowns are the basis for contingency within the triple constraint. Many PM's add 20%, however, in a previous job, the PMO added 20% contingency to the budget with the assumption that money could handle unknowns affecting the other constraints - schedule and scope.
Reply | Reply with quote | Quote
 

Add comment