The sailor told Sir Clowdisley that he was keeping track of their position by using his own navigational equipment. He also informed the admiral that the officers were badly mistaken and that the fleet was much closer to English shores than they anticipated. Therefore, according to the sailor it was very dangerous to proceed to England at full speed since they were in danger of wrecking on the Scilly Isles.
Sir Clowdisley ordered the man hanged as a mutineer. Hours later, his flagship and three other boats of his fleet smashed into the rocks. He was swept ashore where, according to one of the legends, he was murdered by a woman who wanted his emerald ring.
By the way, according to British Naval Law, the sailor indeed deserved to be hanged. The danger of mutiny was a real problem on English ships due to bad living conditions, diseases and corporal punishment. Therefore, only the officers were allowed to keep and use navigational equipment. Admiralty's logic was that if the sailors didn't know where they were, they would be less inclined to rebel against the officers.
This story was frequently used by historians to demonstrate the incompetence, stubbornness and, even pompousness, of some military leaders. Nonetheless, I sometimes joke with my students that the first mistake the admiral made was that of wasting his project resources. After all, what project manager in the right mind disposes of his precious project team members in the middle of the project?
On a more serious note, in my opinion this story also serves as a great example of how the project can go terribly wrong if the project manager chooses to ignore the critique and the feedback from the stakeholders in general and from his own "technical" team members in particular.
I have already mentioned in an earlier article (1) Defining the Detailed Scope: How and Where Do You Find Requirements? the interesting paradox with walkthroughs, inspections and peer reviews. I have learned about these techniques fairly early in my career, liked the concept and started applying it on all of my projects fairly consistently and at times covertly, because of the pressure from some of my superiors to "quit wasting technical people's time on 'silly' meetings". I also know that many of my peers in the IT and software development industries utilized such practices on a regular basis as was confirmed by them during our discussions at various conferences and other professional project management events.
Having spent a big chunk of my career in the aforementioned high-tech sector, I naturally assumed that our project management cousins in the fields where project management has taken much deeper roots - engineering, construction and government - are using this tool on a regular basis.
To my great surprise the words "peer reviews" and "walkthroughs" did not generate much interest or recognition/familiarity among my BCIT (British Columbia Institute of Technology) project management students. Over the course of my teaching career at the university I had people from construction, engineering, pharmaceutical, government, utilities, oil and gas, movie and even theatre sectors sign up for my course. Surprisingly enough, very few of them (excluding some of the IT and software people) ever confirmed that they were indeed using or even aware of this powerful technique.
The interesting aspect of this phenomenon is that almost all of them, after a prolonged cheerleading from my end, tried using walkthroughs and peer reviews to validate the scope of the projects, to identify otherwise unexpected risks and to decrease the amount of mistakes and the rework required. The results were, to say the least, staggering. All who tried this methodology reported great results including, but not limited to, projects finished before the deadlines, high quality of final product or service and, most importantly, highly satisfied customers!
Peer Reviews, Inspections and Walkthroughs
Why Conduct Peer Reviews, Inspections and Walkthroughs?
Let us start our discussion on the reviews with a very interesting statistic originating from the IT and software industry. Unfortunately due to lack of reliable information from other sectors we have to rely on the data collected in the high-tech field and then construct generalized extrapolations from them to the rest of the world. So, without further ado, here is a very interesting statistic:
Forty five percent (45%) of project costs industry-wide is rework!
Let's ponder this number for a while. If we choose to believe in the power of extrapolation to one degree or another, it looks like by eliminating the amount of rework completely we can decrease all of our project budgets in half. Durations would probably fall by a significant number as well.
However, upon further analysis this number does not look excessively abnormal if one remembers the "Cost of Mistake" chart discussed in detail in (2) Kick-Starting Your Projects - What Should You Know About Defining Scope? and "Defining The Detailed Scope: How And Where Do You Find Requirements?" articles. In a nutshell, the study conducted by Barry Boehm (one of the leading thinkers in the field of software development) claims that a mistake that would cost you one dollar to fix at the Initiation stage of the project will typically balloon to a $40 -1,000 deficiency by the project closure.
So, what can happen without the walkthroughs, inspections and peer reviews? Here is a very typical scenario (a bit exaggerated for illustrational purposes) witnessed by the author on numerous occasions throughout his career:
Step 1: Project manager, sometimes with the help of other resources (e.g. business analyst in IT field or architect in the construction industry, mechanical engineer in product development, etc.) develops a project plan and a scope document (e.g. Statement of Work).
Step 2: Technical project team members are still working on other "very important" projects and their functional managers frown upon "distracting them from their urgent tasks.
Step 3: Customers and/or end users are not consulted about the final designs, either because they are "fairly technical" or to avoid "annoying" questions, corrections and other "disruptions".
Step 4: Project manager presents the documents - project plan and statement of work - to technical people and possibly does a quick walkthrough during the project kick-off meeting.
Step 5: Project team members discover a great deal of deficiencies, mistakes and risks in the documentation but, unfortunately, major commitments with respect to time, budget and scope have already been made
Step 6: Finally, the project team pulls together and based on sheer will and long hours deliver a product that somewhat resembles the scope described in the documentation. In addition, despite all the heroic efforts, the project is late and over the budget
Step 7: Finally, customers walk in the doors and even after cursory inspection of the product find dozens and dozens of defects, omissions and mistakes ...
The subsequent sections of this article will attempt to explain what should be done and how it should be done with respect to walkthroughs, inspections and peer reviews in order to avoid the depressing situation described earlier. Exhibit 1 provides the reader wit a high-level overview of the documentation review process.
How to Conduct Peer Reviews, Inspections and Walkthroughs
What Documents should be Reviewed?
Typically it is a good idea to review all the major project, scope and other key documents on your projects. The more that documents get reviewed fairly early in the process, the less are the chances of missing scope items, unforeseen risks and hidden constraints. Having said that, in real life, assembling the entire group of project stakeholders or even stakeholder sub-groups is a difficult and time-consuming task that may not be encouraged at some organizations. Therefore, it seems quite logical to at least concentrate on the most important of the project documents.
Primary documents for the project team members and customers would definitely be the Project Plan and the Statement of Work (aka Scope Document, aka Requirements Document) since they contain the key information crucial to project success. It is also very useful to at least briefly familiarize the project team with the Project Charter in order to inform the technical team members about the project itself, key scope items and constraints. However it is absolutely essential that a project manager conduct a full-scale walkthrough of the Project Charter with the customers and key stakeholders.
How to Organize Peer Review and Walkthrough Meetings?
The best way to conduct peer reviews and walkthroughs (especially on larger, more complicated projects) is to arrange for them at regular intervals during the document development stage. At least two or three meetings should be scheduled closer to the end of Project Charter, Project Plan and Scope Document development stages.
Please note that there is a serious chance that the functional managers may tell you that the technical people are too busy to attend such meetings. Sometimes other stakeholders could feign busyness and refuse or ignore your invitations to the walkthrough meetings. Experience tells us that the best way to convince these people is to share the "Cost of Mistake" chart with them and ask them whether they consider spending a couple of hours now in order to avoid spending several hundred to a thousand hours later a worthwhile investment.
It is a good idea to reserve a two-hour block of time in a medium to large-size well-ventilated conference room several days in advance. You will need at least two hours because the documents - especially on larger projects - tend to be fairly long. Thus, even walking a group of, say, twenty or thirty people through a fifty-page document will probably take the entire two-hour meeting.
On the other hand, keep in mind that walkthroughs and reviews of project documents, especially the ones taking place earlier in the process are very vigorous and spirited affairs that could drain all of project manager's energy and even sometimes hope. Thus, allocating more than two hours for the meeting could have some inherent dangers - people will get tired, start losing their collective focus and may even become a bit cranky!
Always remember, walkthroughs and peer reviews can sometimes be a very humbling experience indeed! One advice I give to my students and my fellow project managers is that a couple of hours of humbling experience is a very small price to pay for discovering mistakes, errors and omissions early in the project.
Preparing for the meeting also involves sending the first draft of the document in question to all the meeting participants with an indication that everyone is expected to review the document with a critical eye and come prepared to voice his or her concerns, suggestions and improvements. Mentioning that a document is just a first draft that could change dramatically after the series of reviews and walkthroughs is especially important because sometimes - and this depends on a number of cultural factors, including but not limited to organizational and country-specific - people could be reluctant to voice their criticism of your work.
I remember a situation when our project had to outsource some of the development work to India and, after working in North America for my entire professional life and thus being used to a fair share of peer critique and feedback, I assumed that no adjustments should be made when communicating to the developers in Bangalore. I remember how we sent a Requirements Specification (Scope Document) to them and arranged for a conference call. I spent about thirty minutes walking our colleague through a fifteen-page document and concluded by asking the typical inspection-type question:
Me: "So, guys, now that I gave an overview of the scope, do you see any problems, risks, issues? Anything that would prevent you from doing proper design and development?"
Team Lead in India: "No, everything seems perfect!"
Me: "Guys, there is no such thing as a perfect first draft of the Requirements Specification! Come on, there must be something"
Team Lead in India: "We don't see any problems"
Me: "OK, take a couple of days and review it amongst yourselves; if you have any questions, do not hesitate to contact me. We can discuss this again"
Team Lead in India: "Great! We shall call you if anything comes up"
Needless to say I never received any follow-up phone calls or e-mails. While remaining in a blissful state of denial, although all of my experience was telling me that something was very wrong, I continued to receive progress reports from India stating that they were "four days ahead of schedule". Finally when the product was delivered and inspected by our local team, we discovered that the end-product was so flawed, that we had to redo the entire development phase of the project from scratch. Interestingly enough, we did discover a lot of issues and deficiencies in the original scope document that most definitely contributed to the low quality of the final product.
When I mentioned this story to one of my senior colleagues, a person who, by the way, was born and raised in India, he told me, "You really expected them to criticize your work?! Critique of one's superiors is not something that is welcomed with open hands over there. They probably saw the mistakes and deficiencies in the document right away but did not want to embarrass you by pointing out the mistakes you made. It is very likely that they tried to address these deficiencies on their own and failed miserably. You should have inspected and peer-reviewed it here first".
The next step is to conduct a quick document walkthrough either using the document itself or some kind of presentation software. It is good idea to plug your laptop into the overhead projector so that everyone can see what is being discussed and recorded. I typically opt for a Word document so that all of the comments, critique, feedback and follow-up items can be recorded right there. It is a good idea to let people word their comments by themselves rather than rephrase them in format acceptable to you. This way you can avoid future misunderstandings about the content of the feedback.
Whatever form you decide to choose - comments in the word processor document or remarks in your notebook - it is worthwhile to re-record all of the action items from these meetings into some kind of Issues List and disseminate it to all of the meeting participants and other project stakeholders. Personal experience shows that incorporating this Issues List into the meeting minutes that should be created and sent out to all the stakeholders is an effective and time-saving technique.
Running Peer Reviews and Inspections
Before getting into the peer review discussion I would like to point out that there are actually two very distinct types of peer reviews. One can call them proper format review and correct technical context review.
Proper Format Review. This is typically done by an experienced project manager external to the project. Accordingly this person is usually not familiar with the detailed technical scope of the project. Actually in my experience it has been even better if that person hasn't heard about the project in question before!
The problem is that someone who has been involved in the project extensively and has participated in many of the project discussions is less likely to be as alert and vigilant when reviewing project documentation. In addition, he or she may be subject to the group's assumptions, either consciously or unconsciously, dominating the project team. A colleague of mine actually came up with a very accurate description of these phenomena; he referred to them as "soaping of the eyes".
For instance the technical people on the project team may think that they have a very good idea what a "seamless integration" means. An experienced peer reviewer who hasn't been burdened by team influence will flag this phrase as inappropriate right away.
Therefore the proper format reviewer's mission is not to validate the technical scope of the project or schedule and budget estimates, or to ask questions like, "What problem were you trying to solve?" His job is to make sure that the document follows the best practices of project management.
The documentation guru's primary responsibility would be to capture deficiencies and mistakes such as ambiguous language (see Exhibit 4), missing measurability, completeness, consistency, prioritization, proper constructs, etc. (see article "Defining The Detailed Scope: How And Where Do You Find Requirements?" for an in-depth discussion of quality project documentation).
Unfortunately these kinds of reviews can not answer questions like:
- Can this scope item be delivered at all?
- Can this scope item be delivered with the budget and timeline provided?
- Are there any technical inconsistencies in the scope?
Correct Technical Context Reviews or inspections come in handy here. The technical team members probably would focus on the correctness of technical information, unidentified risks, overlooked resource requirements, etc. in the documents rather than the way in which documentation is written.
For example, a statement like
"Build a large four-bedroom three-bathroom Tuscan-style home on a budget of one hundred thousand dollars by the end of next month"
will probably generate different kinds of feedback from the guru and from the technical project team members. The documentation guru will almost certainly notice an inappropriate use of the word "large" and will ask the project manager to associate a measurable attribute with that particular term. As a result the word "large" could be replaced with statements like "5,000 square feet" or "three-level, 6,000 square feet."
On the other hand, technical team members may have an idea (not necessarily a correct one) of what "large" means or they may just miss it because of the lack of training in documentation reviews. However, they will probably come back to you with a comment that buildings of that size can not be built on a budget of hundred thousand dollars and be completed in month and a half.
Who are the people you should invite to peer reviews and inspections? With proper format reviews it is fairly straightforward - it's the project manager and the documentation guru, most likely one of the most experienced project leaders in the company who has been properly trained in the fine art of critical reading.
The technical document inspections will require the presence of the project manager, team leads of each functional group involved in the project, all technical team members, quality assurance people, subject matter experts in each of the functional areas and several key representatives from the customer side.
Running Customer Walkthroughs
For a stakeholder walkthrough you probably want to consider inviting your client or customer (whichever is applicable) and if possible the end users of the product. Note that each one of these groups could be in turn subdivided into several clusters; some of these clusters could be primary and some of them could be secondary but both are important.
Also, it would be a good idea to invite key representatives from the technical team since some of the questions posed by the customers and users could only be answered by them. You also need technical team reps in order to explain and justify your estimates.
If you are working on a multi-departmental project at a fairly functional organization inviting other department heads could also be valuable because they could end up providing resources for your project.
Peer Review and Walkthrough Challenges
Some of the issues encountered by the project teams during the peer reviews, walkthroughs and inspections include large requirements documents, large inspection teams and geographical separation.
So, what are the tools at your disposal in battling the "walkthrough fatigue"? First, if dealing with larger documents, attempt to schedule several reviews in stages. This way you can reduce the duration of reviews and maintain the focus of the participants. Also, try not to exceed a two hour time limit on meetings. And one final suggestion: when booking a conference room, pick the largest and the best ventilated room possible. Nothing kills the thought process like the lack of oxygen!
Sometimes the review teams can get very large, especially on key project documents like project plan or project scope definition. Having fifty or more people in the room commenting and providing critique on your documentation can turn into chaos very quickly. Therefore, it could be useful to break the inspection teams into groups for initial inspections, and once the major issues discovered in these separate meetings have been analyzed and addressed properly, the final joint technical team and customer walkthrough involving all the project stakeholders can take place.
Geographical separation can also present certain challenges. Try using tools like video conferencing, teleconferencing or tracking and comments functions in word processing software.
What Kinds Of Questions Should You Expect (And Ask)?
So what kinds of questions can one expect from the walkthroughs with the customers and the technical team members? Exhibit 2 lists some of the questions the project manager should expect when engaging in the walkthroughs, inspections and peer reviews.
|Question||Document||What Should The Project Manager Do?|
|"Why are your estimate ranges so wide?"||Project Charter and Project Plan||Explain that the appropriate ranges for the Initiation stage estimates are +75; -25% for regular projects and +300%; -75% for software development and R&D ventures|
|"Why will it take so long?||Project Charter and Project Plan||Go over the project schedule and explain how the estimates were obtained|
|"Why will it cost so much?"||Project Charter and Project Plan||Go over the project budget and explain how the estimates were obtained|
|"Can you finish the project faster?"||Project Charter and Project Plan||Explain the concept of project management triangle (scope, time, budget) or pentagon (scope, time, budget, effort and quality) and find out which areas can be manipulated to deliver the project faster|
|"Can you find some ways to do it cheaper?"||Project Charter and Project Plan||Explain the concept of project management triangle or pentagon and find out which areas can be manipulated to deliver the project cheaper|
|"We have to add another high-level feature"||Project Charter||Try to assess which problem the requested feature will solve and if, necessary, add to the "Issues List". If required, schedule an "offline" meeting|
|"We have to add another detailed scope item"||Scope Document||Try to assess which problem the requested requirement will solve and, if necessary, add to the "Issues List". If required, schedule an "offline" meeting|
|"With respect to project feasibility, who came up with revenue projections?"||Project Charter||Explain the underlying logic in arriving at future revenue forecasts|
|"You should also mention regulatory projects, competitive advantage, etc. in the project feasibility section"||Project Charter||Have a discussion with the participants and add to the document if necessary|
|"Have you considered this risk?"||Project Charter and Project Plan||Quickly assess the risk and flag for incorporation into the documentation. Arrange for an offline meeting and/or follow up if necessary. Add to the "Issues List"|
|"You need to communicate to person X from Department Y; he should be involved in this discussion"||All documents||Add to the "Issues List". Schedule a follow-up meeting with that person|
|"Director of Department Z will have to assign resources to this project||All documents||Add to the "Issues List". Schedule a follow-up meeting with that person|
|"Hey, that is not what I needed!"||Scope Document||Briefly discuss the problem with the stakeholder. Add to the "Issues List". Arrange for an offline meeting and/or follow up if necessary|
|"Oh, I have just realized that we will also be needing this
|Scope Document||Briefly discuss the new scope item with the stakeholder. Add to the "Issues List". Arrange for an offline meeting and/or follow up if necessary|
|"You forgot to include this..."||Scope Document||Briefly discuss the new scope item with the stakeholder. Add to the "Issues List". Arrange for an offline meeting and/or follow up if necessary|
|"There is conflict between these two (or more) scope items"||Scope Document||Add to the "Issues List". Arrange for an offline meeting and/or follow up if necessary|
|"We do not have the capability to make that happen"||Scope Document||Discuss why this can't be done and what are the alternative ways of reaching project objectives. Add to the "Issues List". Arrange for an offline meeting and/or follow up if necessary|
|"I can interpret this statement in several ways"||All documents||Rephrase the statement in a proper format to remove ambiguity|
What to Watch Out For: Defect Checklists
So what are the things all the reviewers should be on the watch for during walkthroughs, peer reviews and inspections? When reviewing scope and other project management documents the stakeholders should be asking themselves the following questions under the following categories:
Organization and completeness of the documentation:
- Are all parts of the documents written at a consistent and appropriate level of detail?
- Does the scope definition document provide an adequate basis for design?
- Do we have priorities assigned to each scope item?
- Is there any information missing in the document? Are there any "TBDs" in the documents?
- Did we cover all possible alternatives, exceptions, risks and constraints?
- Is every requirement in scope?
- Can all the scope items be implemented with all the known constraints?
- Do all of the scope items have appropriate measurability attributes associated with them?
- Do all the key statements in the document have only one possible meaning?
- Can they be misinterpreted by any of the stakeholders?
- Exhibit 3 provides a list of potentially troublesome words and phrases that tend to frequently appear in project management documentation in basically all industries and types of organizations
|Words||What To Do About Them?|
|"Acceptable", "adequate"||Define acceptability and how the product and stakeholders can decide what is acceptable and what is not|
|"Efficient"||Explain how efficiently the product performs operations or how easy it is to use|
|"Fast", "rapid"||Specify minimum, maximum and desired speed|
|"Flexible"||What specifically should the product do in response to specific changes in the environment or business objectives|
|"Improved", "better", "faster", "superior"||Quantify how much faster constitutes adequate improvement|
|"Maximize", "minimize", "optimize"||Provide maximum and minimum acceptable parameters for each value. You can also provide desired range too|
|"Seamless", "transparent", "graceful"||Translate into observable product characteristics|
|"Several"||How many? Provide a specific number or maximum and minimum acceptable parameters for each value|
|"State-of-the-art"||Define what this means|
|"Sufficient"||How much is sufficient?|
|"Support", "enable"||Define exactly what functions would constitute support or enabling|
|"User-friendly", "simple", "easy"||Translate into observable product characteristics|
Document walkthroughs, inspections and peer reviews are surprisingly underutilized in many industries despite strong evidence that these practices have a very positive effect on the quality of project results.
Considering the fact that close to forty-five percent of project costs are rework and that the cost of fixing a mistake grows a thousand-fold as one progresses from the project initiation to the close-out, utilization of these techniques is a very worthwhile and rewarding investment.
In a perfect world most of the project management documentation would be properly reviewed by the proper audiences; however, in the "worst-case" scenario at least review the project charter, the project plan and the scope documents.
When conducting reviews and inspections, you should reserve ample time (but not too long, as people will get tired) in a large, well-ventilated room. Email the document in question form ahead of time and inform the reviewers that this is a first draft that may change considerably as feedback is received, analyzed and incorporated. Be aware of cultural and organizational obstacles to conducting candid and open reviews.
Understand the difference between customer walkthroughs, "proper format" reviews done by the experts in your organization and "correct context" technical team inspections and utilize them all at proper times in the project.
Be aware of the types of questions you may be asked and know how to act in each scenario. Maintain an Issues List during each meeting to have a list of all the problems and questions raised for future follow-up. .Also, use a "defect checklist" as a guide in assessing the quality of your documentation.
One final piece of advice: when walking into the review conference room, leave your ego at the door!
Don't forget to post your comments below
- "Software Requirements, Second Edition (Pro-Best Practices)" by Karl E. Wiegers
- "Effective Requirements Practices" (Addison-Wesley Information Technology Series) by Ralph R. Young
- "Exploring Requirements: Quality Before Design" by Donald C. Gause and Gerald M. Weinberg
- "Requirements by Collaboration: Workshops for Defining Needs" by Ellen Gottesdiener (Kindle Edition - Feb 17, 2009)
- Mastering the Requirements Process (2nd Edition) (Hardcover) by Suzanne Robertson and James C. Robertson
- "Project Management Body of Knowledge (PMBOK)" by Project Management Institute (PMI)
- "Project Management" by Robert Santarossa
- "Economic Principles: Seven Ideas for Thinking ... About Almost Anything" by Douglas Allen
- "Software Engineering Economics by Barry Boehm,", Prentice-Hall, 1981
Jamal Moustafaev, MBA, PMP – president and founder of Thinktank Consulting, is an internationally acclaimed expert in the areas of project/portfolio management, scope definition, requirements analysis, process improvement and corporate training. Mr. Moustafaev is author of “Delivering Exceptional Project Results: A Practical Guide to Project Selection, Scoping, Estimation and Management” (released by J. Ross Publishing in September 2010). He is also the author of various project management and business analysis webinars delivered in partnership with Project Times:
- Why and How Should You Manage Scope and Customer Expectations?
- Are We Supposed To Negotiate On Projects?
- Top Ten Things You (Probably) Did Not Know About Estimation
In addition to teaching a highly acclaimed “Project Management Essentials” course at British Columbia Institute of Technology, Mr. Moustafaev also offers the following corporate seminars through his company:
“Practical Portfolio Management - Selecting & Managing The Right Projects”
“Successful Hands-On Management of IT and Software Projects”
"Successful Hands-On Management of Modern-Day Projects”
“From Waterfall to Agile - Practical Requirements Engineering”