“Projects are a means of organizing activities that cannot be addressed within the organization’s normal operational limits. Projects are, therefore, often utilized as a means of achieving an organization’s strategic plan.” A Guide to the Project Management Body of Knowledge.
Projects are a significant investment for an organization. Projects are key to an organization’s growth and improvement. The decisions about which major projects go forward are usually made during an annual planning cycle. After executives approve them, they are budgeted, resourced and slated for delivery.
How do the executives know that what is being delivered is what they originally agreed to? How does management make decisions during delivery? How does the team remain aligned on the priority items? Project status is the key.
Some example questions that project status should answer are:
1. Will the project be delivered when we expect it?
2. Do we have the budget to complete the project?
3. Will it deliver what the users expect?
4. Will the quality of the final product be sufficient?
Some example decisions that project status supports are:
- Decide the project isn’t going to meet expected ROI and thus cancel it
- Decide the project needs additional investment and can still meet anticipated ROI
- Change resources on the project
- Decide what scope to keep or change
The challenge with project status is to know what information should be collected, consolidated and reported to best support management and executives. This article discusses a framework and considerations for measuring project status. Objectives and metrics are suggested. Each organization would modify these objective and metrics to apply to the characteristics of their own organization.
Why Measure Status?
Why bother measuring status at all? It’s an additional cost to the project to collect status information. It is a distraction for team members delivering the work. It needs to be monitored by someone to ensure consistent measurement across projects. It can lead to disagreements about whether a project is actually in “green” or “yellow”.
To answer this question we first draw on a widely quoted statement from Management Guru Peter Drucker, “What gets measured, gets done”. A simple example of this is demonstrated by an effective weight loss technique of recording what you eat actually helps reduce the amount (and type of food) you each. With respect to projects, the fact that we are measuring status drives activities to get done. It serves as a reminder to team members and management what their focus should be. So, although it can be more work for team members, with the proper measurements, paradoxically, more work gets done.
A longer statement that applies comes from the famous scientist and engineer, Lord Kelvin:
“When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the stage of science.”
Applying a scientific method permits an organization to acquire new knowledge and integrate and correct previous knowledge. In this way an organization can improve with how they deliver projects. It enables an organization to make fact-based decisions rather than relying on opinions.
What does it mean to measure? Defining what measurement means helps decide what we need to measure. For the purposes of this discussion measurement is a set of observations that reduce uncertainty where the result is expressed as a quantity. Different measurements have different information values. The value of information related to the chance of being wrong, and the cost of being wrong.
With respect to measuring project status, we want to reduce the uncertainly about what is going on in the project. By reducing uncertainty, executives gain confidence in the decisions they make on how best to continue to allocate their resources.
How to Measure Status?
Status, as defined by dictionary.com is “state or condition of affairs”. What we need to track then is the condition of affairs of various dimensions of a project (to be discussed next). The implication is we need to define the desired condition of affairs (our objectives) and measure against them.
The other area to consider is how often we want to sample status. Sampling status means to make a measurement. The measurement frequency may be different for each status dimension.
The decision about the measurement frequency is a trade-off between the cost of taking the measurement and the value of the information. For instance, if we asked team members to provide updates to how much time they have spend on a task every day it incurs overhead and thus cost. The value of having that information daily rather than weekly may be no different because it would not impact any of the decisions made.
In order to know our condition of the project, we need to know what we expect it to be compared to what it actually is. Therefore, for each measurement, we need to have an expected or target value.
Status should be driven by facts about the project rather than opinions. This removes controversy about the ‘colour’ of the status from the discussion. For instance, consider a rule in place that states a project is in ‘yellow’ if the forecasted cost is between 10%-15%. If the forecasted cost is 12% there cannot be an argument about a project being yellow or not. The rule itself cannot be in disagreement as it would be one imposed by the Project Control Office across the organization. That would only lead to a disagreement about the forecasted costs being correct or not. That type of problem should be addressed quickly since forecasted costs are core to the management of the project.
There may be multiple metrics supporting an objective. Ideally, we would have metrics that should show results in the context of the objective and leading indicators that provide a hint as to where the objective is heading.
What to Measure?
Schedule, budget and scope (also known as the project triple) are the fundamental constraints set out at the beginning of a project and agreed upon by all stakeholders. The business case to proceed with the project is based on these elements. They set the expectation of what will be delivered according to the resources allocated to the project. For this reason, the project triple are represent important items to monitor, as they are the major contributing factors to stakeholder satisfaction and to meeting the business case.
Once a project is underway the questions most often asked are: “Are we on time?” and “Are we on budget?” A project can easily be on time and on budget and still be a failure. A particularly dramatic example is the Tacoma Narrows Bridge which collapsed in 1940. The bridge was completed close to budget and on schedule. However, due to some design flaws, the investment was lost (thankfully no lives were lost except for Tubby, a cocker spaniel of a driver who narrowly escaped).
For this reason, we will add quality as a dimension we want to measure.
The following is a framework with details of the objectives and metrics to use across these different status dimensions.
Schedule
Schedule objective::Meet milestone dates agreed upon by stakeholders.
Schedule metric: Variance between target date and forecast date.
Measurement: Weekly
This helps us determine if we are going to meet our goal. Perhaps there are other indicators that we should be measuring that might suggest that the schedule is at risk. Components that make up the schedule are:
- Work estimates
- External dependencies (such as equipment orders)
- Speed of risk and issue resolution
Metrics to address the components include:
- Accuracy of work effort estimates for tasks completed
- Mean time to resolving issues
- Mean time of open issues
- Longest current open issue
Budget
Objective: Meet the financial project cost agreed upon by stakeholders.
Metric: Variance between target cost and forecast cost.
Measurement: Weekly
Again, this helps us understand if we are going to meet our goals. Let’s consider if we can determine some leading indicators based on budget components. As an example, an IT project budget is composed of:
Hardware and software change in size depending on the load placed on them. Examples include number of users and number of transactions. Budget issues would arise for not capturing the load correctly. Change requests related to increasing loads is an indicator of increased budgets.
Labour cost is usually tied to work effort. Labour cost and schedule are correlated in that often an extension to the schedule results in an increase of labour cost. We already have schedule metrics so that will help us with labour costs and thus budget overall.
The other area of increased budget would be requests for new functionality. Monitoring change requests related to functionality is an indicator that the budget may go up.
Scope
Objective: Deliver the agreed upon functionality.
Metric: Variance in schedule and budget due to functionality change requests.
The measurement mechanism for scope is project change requests. Schedule and budget show the impact of that change to the project.
Measurement : Weekly
Change in functionality can be observed by the number of change requests related to functionality. As well, having requirements completion running late could mean that there is difficulty obtaining requirements and, thus, defining functionality. Measuring the variation in time between target sign-off date for requirements and the actual sign off date can be a leading indicator that there may be scope challenges.
Quality
Objective: Deliver a product with no unresolved defects.
This is a rather strong statement. Note that the term used is “unresolved defects” rather than “zero defects”. All defects need to have a resolution in place. This may be correcting the defect or deciding that the product can be realized with the defect with a mitigation solution in place.
Metric: Percent of unresolved defects.
Measurement: Weekly (although may be daily for a QA Manager)
Defects may be latent in the product just waiting for a test case to uncover it. Test coverage across all functionality is important to meet our specified objective. There should be at least one test case per requirement. Therefore, a metric to use is percent of requirements with test cases.
What about Overall Status?
The overall objective for a project is:
Deliver a project that meets an expected level of quality within the stakeholder agreed upon constraints.
Measuring overall status is tricky because, suddenly, you have lost sight of the context of the status. If the overall status is ‘yellow’ it is not clear what needs to be addressed. Rather it is a sign that the project needs attention. Overall status would be a logical combination of the other status items. Let’s look at a couple different ways to calculate this.
Worst Child
In this situation, the overall status takes on the value of the worst child status. For instance, if the financial status is yellow and all others are green, the overall status would be yellow.
This approach drives attention to pushing projects to be “all green”. It is the most thorough approach, however it may be difficult to distinguish relative priorities between projects. For example, a project with only one child status of red and a project with all children status red will both have and an overall status of red.
Weighted roll-up
In this case, each sub-status is provided a relative weighting of its importance. For example, you could place greater relative importance on budget over scope, schedule and quality. Of course, you may have more thresholds than the traditional red, yellow and green. The three thresholds are used here as an example.
Status Trends
If a project is off track as indicated by the status, management needs to make some decisions and take action to correct the problems associated with the status. Once the action has been executed management needs to know if it is making a difference. To do this we need to capture and report on the status trend.
The trend is simply how the status is changing from one reporting period to another. The principal consideration is the level of granularity of reporting period. Multiple levels of reporting periods (and thus multiple trends) may be required.
The guiding principle for the reporting period for trends is how quickly you anticipate actions will make an impact on the status. For example, a decision and action to improve the financial status may take a month or two to take effect. This won’t be reflected in a weekly trend indicator. In fact, taking too small a granularity will be misleading if there are weekly variations that do not contribute to the overall long term trend.
The following chart shows an extreme example of how measuring in the short term can overlook the longer term trend.
As you can see on a small scale, the trend oscillates up and down providing misleading information. Overall, the trend is increasing. Some example trend periods are described below.
Schedule
Schedule changes are only indicated as frequently as tasks are updated and the schedule is recalculated. In many organizations, this happens on a weekly basis. This would be the minimum period to show trends. However, there may be no material change in a project during a week. It may be better to show trends on a bi-weekly basis.
Budget
Similar to schedule, a budget change will only be seen as tasks are updated and the schedule recalculated. For some organizations, project costs are tracked through what work is invoiced. In this case the minimum frequency would be the billing cycle. Again bi-weekly is likely a good period to monitor trends.
Scope
In a controlled environment, scope only changes through Change Requests. In most organizations, change requests can be submitted at any time. However, there may be regularly scheduled change request review meetings after which the status is updated. The trend period should be aligned with these meetings in order to capture the change in information.
Quality
Quality is most measurable during the testing phase of the project. Different people may want to see different trends. A QA Manager may want to see a daily trend over an extended time period in order to track whether defects are converging or not. A manager may only want to see a weekly trend to determine if the product quality is heading in the right direction.
Ongoing Evaluation
Measuring status will require a number of iterations to refine the status items. After a project delivery cycle, reflect on the measurements in place and the end result of the project delivery. Based on the results, the measurements and trending should be tweaked to focus on the measurements that convey the information that most strongly represents project status.
Other Considerations
The status items discussed cover delivering a quality project within agreed upon constraints. These are the main concerns of the delivery team and should be their principle focus.
What it does not address is whether the business case for a project is still valid. A business case is composed of two parts: the financial opportunity (additional revenue or reduced costs) and the delivery costs. The metrics discussed cover the delivery costs. The other side piece of the equation is the financial opportunity. This should be monitored through the project lifecycle by the original project business sponsor to ensure the overall benefits to the company remain in place. Some questions the business sponsor should be asking:
– Is the original business case still valid?
– Do we still expect to capture the same market share?
– Has pricing changed?
– Have other better cost saving opportunities come up?
A similar process of defining objectives and assigned metrics can be used to answer these questions.
First Steps
The first step towards understanding project status through measurement is to identify owners of the project status definition. Typically this would be the Project Control Office or a Project Management Office. Once the area of responsibility is assigned, they can take the concepts in this document and apply them to their organization. Each organization is different with slightly different metrics. The status owners will need to draw on their experience of the business to determine the specific objectives and metrics.
Don’t forget to post your comments below!
Craig McQueen, MSc, CBIP, leads the Business Intelligence Practice for Agora Consulting Partners (www.agorainc.com). A key part of his role is assisting companies with implementing a performance management framework which ties metrics to strategic objectives. Craig has worked with various enterprise customers in the Financial, Telecom and Manufacturing industries. Through his career he has been a contributing author to 5 books and has authored over 20 articles. Craig can be reached at [email protected]
