Author: Fernando Santiago

Good things come in “threes”: from the divine Holy Trinity to the mundane, and here the list goes on and on: the three little pigs, caballeros, stooges, amigos, sisters and, of course, the musketeers.

Three is a “sacred” number, which for many represents perfection; the stability of a tripod is an example.

Earned Value Management (EVM) started with a trilogy: PV, EV and AC. However, the relationships between three elements are two: n x (n-1)/2 = 2 (as we all recall from high school, well, maybe). The two relationships are, obviously, SPI and CPI, the two great indicators that come from EVM.

However, EVM has received strong criticism in recent years. With the advent of agile and other approaches that talk about delivering “value”, EV has been discredited, and for good reason: the term is a misnomer. Earned Value should be called Earned Scope, and this is what creates the confusion particularly in those that do not understand EVM and don’t understand what value means either but, isn’t that always the case? Ignorance fuels criticism.

In capital goods projects and programs, EVM is religion. Industries like defense, aerospace, nuclear, etc. have been using EVM for fifty years and have perfected the technique to an art. Conversely, in IT, EVM is shunned and perceived by many as overly complicated, a relic from the industrial revolution in project management. Very few organizations, outside of capital goods, use EVM, and those that do usually do it poorly. To make things worse, tools like MS Project and JIRA provide “automatic” calculation of EV at the task level which creates even more confusion in people who don’t have an understanding of EVM. Recently a friend of mine who runs one of the main PPM solutions for a major Canadian bank told me they were implementing EVM and like all the other tools, it builds from the task level, a conceptual mistake but easier for the tool. With fifty thousand projects, when they run the calculation it crashed after 18 hours.

No wonder why organizations out of capital goods shun EVM: they don’t understand it.

Despite all these shortcomings, and despite the cries from agile practitioners that EVM is a step back in the evolution of project management, this framework remains healthy and is in for the long run. This past May I presented a topic on strategy execution at the EVM conference in Fort Lauderdale, Florida, with close to 500 participants, and will present another one in November in Washington DC. Two conferences per year is not bad for EVM, not bad at all.

Today everyone is talking about managing benefits, and very few people turn to Earned Value as a framework that could be used for this purpose. There are two reasons for this: EVM has been discredited, as explained above and, second, it is about delivery performance, which happens during execution. However, Crispin Piney, the author of the book Earned Benefit Program Management, CRC Press 2017, developed a framework to extend EVM in a way that it covers not only delivery but also benefits and value. In his book Crispin mentions how by pure chance, he and I connected on LinkedIn and he found in my work a piece that was missing in his framework. We started to collaborate and we co-wrote Chapter 4 of the book. In May 2018 Crispin was the keynote speaker at the Earned Value Management Conference in Fort Lauderdale, Florida, where I also presented a topic on strategy execution.

Crispin’s book explains an approach to calculate Planned Benefits, an equivalent to Planned Value in EVM. Planned Benefits, or PB, can be compared to forecasted benefits during the execution of the project, allowing for the calculation of an indicator, Realization Performance Indicator, RPI, the third musketeer. EVM now has the trinity of indicators: SPI, CPI and RPI.

When we approach the topic of benefits, we get into the same “challenge” that generates the need for EVM: “A yearlong project is six months into execution and has spent 60% of the budget: is this project on schedule, on budget?” The answer is, of course, not enough information. Similarly, a project that has an estimate of $2.5M in benefits in the next five years, should it be done? The answer is, of course, not enough information. Until we know the amount of the investment required, we cannot say. If investment is $1.5M and happens in the first year of the project, the project gives a rate of return of 20%. This can then be compared to the company’s hurdle rate of, say, 12%.

Table 1 – Example of a project IRR

However, when risk is considered, the situation could change:

• If delivery risk is high (i.e. new technology, multiple and conflicting stakeholders, unclear requirements, etc.) the amount of investment and the duration of the project is likely to increase.
• If benefit risk is high (dependent on industry/market forces, time to market critical, etc.) the amount of benefits and the time to realize could also change.

In a high delivery and high benefit risk scenario, the situation looks bleak: an IRR of only 2%, way below the hurdle rate of 12% and not enough to justify this project.

Table 2 – Example considering risk

Based on this, we can say that it is value what matters, and not benefits alone. “Realization Value” (RV) can be defined as Benefits minus Investment and adjusted to risk and the time value of money. It is this definition of value that should be used when defining a portfolio management performance framework.

Going back to our example, people who would highlight the impact of risk would be called fear mongers, so the sponsor would likely throw a pep rally where s/he asks the commitment from the team to deliver as planned, so the $1.5M is allocated and the $2.5M is forecast; let the games begin.

As the project progresses, EVM would allow us to track schedule and cost performance, so we can produce reports with SPI and CPI (schedule and cost performance indicators) that clearly show how bad a decision we made approving the project.

The concept of Earned Benefit (EB) is a natural extension to Earned Value Management. EB is tied to components, deliverables that have the potential to generate benefits. Planned Benefit is based on the plan to deliver the components, measured in Business Case benefit dollars. This is analogous to Planned

Value (or BCWS for those old enough to remember), which is the scope planned to be delivered at a given time, measured in Budget dollars.

Before we tackle the numbers, let’s explore some differences between EV and EB:

• They are units of scope measured in dollars, but different dollars: PV in budget dollars, PB in benefit dollars from the business case
• Earned Value recognizes scope as it gets completed, while Earned Benefit recognizes benefit both as a potential benefit as the component projects progress, and as accruing benefit once components complete and are ready to generate benefit.

The calculation of EVM indicators is well known; well, it should be. Anyway, let’s assume that we all understand what SPI and CPI mean and, most importantly, how to interpret these metrics:

• Indicator = 1 means right on target. Basically, it means that the actual execution is on track
• Indicator above 1 is good: either ahead of schedule or under budget
• Indicator below 1 is bad, either behind schedule or over budget

The terms good and bad have to be taken with a grain of salt. There is another reason why my Catholic upbringing conjures images of sin and guilt when words like good or bad are used. In reality, a kiss is just a kiss, like in Casablanca, and a plan is just a plan, and the only thing that we knew for sure, when we started the project, is that the plan was wrong, and that we would finally get it right when we finished the project. So, we are comparing reality to a plan that is wrong, so tolerance for variation against that plan is needed.

A very graphic way to illustrate tolerance is the “bull’s eye” chart, presented in Fig. 1. This chart is based on the tolerances defined by the organization for schedule and cost performance. Some points to consider:

• The organization can choose to define these tolerances at different levels: corporate, business unit, etc. and/or consider other factors like type of project, approach to delivery, etc.
• Tolerances are not symmetrical. In the example, SPI for a project behind schedule remains green down to 0.9 or 10% of variance while on the positive side, a project ahead of schedule remains green up to 1.4 or 40%.

SPI< 1Behind schedule  SPI= 1  SPI> 1 Ahead of schedule

Fig 1: Bull’s Eye Chart

The obvious question that jumps off this chart is how does a project that is ahead of go yellow, and even red? What these colours mean is that this project requires management attention, as one or more of the following could have happened:

• The estimate was too conservative, “sand bagged”
• Coordination with other projects in a program or a portfolio for deliverables or transition of resources could be disrupted

The same logic applies to cost performance, as funds and resources tied to a project could have been allocated to other projects or invested in the market.

At this point the question is: is there an indicator for benefit, let’s call it RPI, which could work in a similar way as SPI and CPI?

There are two answers to this: the calculation of RPI is covered in Piney’s Earned Benefit Program Management book. For Piney, RPI is based on the ratio of contribution and allocation, terms associated to the model used to calculate benefits for a program or for a portfolio. Allocation is defined as the investment to date in the component projects; Contribution is the potential benefit “earned” from the current progress on the same projects. The full details of the calculation of this Earned Benefit are out of the scope of the current article.

The second option, which I call “poor man’s RPI” is a simple ratio of Planned Benefits to Forecasted Benefits, which works as follows:

---

---

• At the time of approval of a project during the initial stages, a business case is a given and it is safe to assume that it provides the calculation of Net Present Value, which is based on the series of cash flows described earlier in this article at a given discount rate set by the organization. NPV is a financial quantity, not a rate or an indicator, so it can be used to compare future values to a baseline.
• At subsequent gates, depending on governance, or following a time cycle (e.g., quarterly), projects are required to review their business cases. This includes both investment (cost) and benefits. This raises the following concerns:
  • If NPV is used to calculate RPI, it is including the impact of changes in cost so, in a way, it is double counting costs. While this is true, EVM has the same situation with schedule and cost, as a project that takes longer will likely cost more money, likely the burn rate, for a longer period.
  • As RPI represents benefits, we should not measure value, only benefits.
• Because of these considerations, the simple solution is to discount only the benefits, the positive flows, and not the investments, or negative flows, as seen on Table 3 below:

Table 3 – Discounted Benefits

As much as someone may have thought that sunglasses should have been included with this article to look at the bull’s eye chart, it does provide in a very simple and graphic way the state of a project or a portfolio, or the comparison of portfolios, etc. The problem is that the two axi of this matrix are already taken with SPI and CPI, and now we need to insert RPI. To solve the problem, we will consider an element of EVM called a conservative estimate of the budget = Budget / (SPI x CPI), as displayed in Table 4

Table 4 – Conservative Estimate

Now that we can call this indicator DPI, to represent delivery performance and RPI for benefits, we can use the bull’s eye chart as in Figure 2

Fig 2: Bull’s eye chart – Delivery VS Benefit

When the portfolio has dozens or hundreds of projects, a bubble chart looks more like a bubble bath. An alternative is to display three sets of information, one at a time, with a Business Intelligence tool like Power BI, or using an application that has this functionality, like ValueViewer from P3M Solutions:

• Number of projects
• Investment
• Internal Rate of Return (IRR)

Fig 3 – Number of Projects in Bull’s Eye Chart

As shown in Fig 3, out of 35 projects, only 1 is under budget and ahead of schedule but 20 are “green”, 12 are yellow and 2 are red, from which 1 is unrealistically ahead of schedule and under budget.

The same chart can be used to display investment, as shown in Fig 4, as well as Internal Rate of Return (IRR) or NPV. This information allows executives to assess “where they are putting their chips” which, analogous to the decisions made at a casino, it is based on expected return, investment and risk. Unfortunately, in this case, the table is not all green, like at the casino, but this is reality.

Fig 4 – Total investment in bull’s eye chart

A simple case study will help to illustrate all these concepts:

An online retailer is developing a web application to let customers know when products that are related to the things they are buying are on sale. As an example, if you are buying paint, brushes and frog tape, it may be nice to find out that rollers are on sale. Customers can either shop online or through a call centre. The project has identified three major deliverables:

• Web app for customer shopping on-line
• Call centre application to provide the customer service rep with the products on sale as the customer places the order
• Engine that manages the relationships between products: (i.e. paint, solvent, brushes, rollers, tape, ladder, etc.) and presents the information to the other two components

Looking at these deliverables we can identify the web app and the call centre application as business capabilities that will allow the business to do something it couldn’t do before and generate incremental revenue (benefits). The engine is a technical capability, or enabler, that contributes to the business capabilities but doesn’t generate business value by itself. When all these capabilities are implemented, they will contribute to business outcomes, revenue increased in this case. We can capture all these relationships in a Results Chain Diagram, presented in Fig 5

Fig 5 – Results Chain Diagram for the case study

The company has estimated that incremental sales due to these capabilities will be $3M/year for on-line sales app and $2M/year for the customer service application.

The results chain (RCh) also captures the contribution each element has on the forward nodes of the chain. In the example, the contribution of the engine to the business capabilities is different, as it reflects how dependent on the engine the outcome is, and in the case of the web app it is more dependent, as there is no human interaction. Fig 6 presents these contributions in a percentage basis. When there is only one contribution it is 100%.

Fig 6 – Contributions in the example

The next step is to estimate benefits. The sponsor of this program has calculated the expected increase on line items per order, the Key Performance Indicator (KPIs), from the current 3 to 5, and that this will generate $400k/year. For the call centre KPI, the expected increase is from 4 to 6, with incremental revenue of $500k/year. These benefits are then propagated:

• Forward, left to right, to the financial outcome of revenue increased
• Backwards, to capabilities and initiatives, based on the relative contributions (percentages)

Fig 7 – Propagation of inflows/benefits

Finally, we add the investment needed by each initiative (in blue under the symbol) and propagate it forward (from left to right). From this diagram, shown in Fig 8, we can conclude:

• The overall return of this program is positive, with a total investment of $780 that is recovered in the first year, with at least four more years of inflows.
• Each one of the nodes has a positive return.

Fig 8 – Propagation of outflows/investments

Fig 8 captures the baselined plan described in the business case. The Results Chain diagram (RCh) captures all the relationships and the contributions and allocations in a static way. In this diagram we can see that return on investment (ratio of benefits to investment) is overall positive, and also positive at every node except for the initiative Develop CS App, where investment is slightly higher than the benefit.

The time component is not represented in this diagram, as it belongs to another venerable PPM tool, the portfolio roadmap, captured in Fig 9. This is where dependencies and constraints for execution are captured and managed. The RCh captures relationships for realization of benefits, not dependencies for execution.

Fig 9 – Portfolio Roadmap

What a plan! Now, let’s execute

At the end of the first calendar quarter, March X

• Engine started late, in March, still “hoping to deliver on time” (haven’t we heard this before).
• SPI is 0.46 as it shows 1 month of progress against two months. CPI is 0.8.
• Management decides not to revise forecasted benefits yet, so RPI = 1 for both business outcomes

Fig 10 – Example Q1 200X

At the end of Q2 the situation is as follows:

• Customer Service App did not start in June as planned
• Engine has recovered to SPI = 0.88 through overtime and contract resources, CPI is at 0.86.
• Management decides to adjust down the forecast to no benefits in the first year, RPI down to 0.94 for call-centre app, as benefits for the first half of the year have been removed. Benefits for the online app also went down to 0.90.

Fig 11 – Example Q2 200X

At the end of Q3 the situation is as follows:

• On-line sales app did not start in September as planned, due to project resources temporarily assigned to support the Call Centre App.
• Call Centre App also hit some new issues; the project is behind schedule with an SPI = 0.77 and CPI of 0.83.
• Engine has not been able to recover and still around 0.80 SPI. Management has decided to approve an extension to the end of June 20X1.
• Benefits forecast has been adjusted, as projects will not be able to realize benefits until the engine is in place. The engine is what is called an “essential contributor”; this means that whether the on-line sales or the call centre app are delivered in Feb 20×1, benefits will not be realized until later. The adjustment of the forecast is based on the following considerations:
  • Call centre app does require training and a ramp up time as there is human intervention. No benefits will be realized until Q3 20X2, for a total of $150 in the year. RPI goes down to 0.71.
  • On-line sales do not require human intervention and the customer should be able to pick up this functionality very quickly. Benefits start to ramp up in Q2 (half of the $100k per quarter and $100 in Q3 and Q4, or a total of $250 in 20X1. This is closer to the original estimate of $300k, RPI goes up to 0.97

Fig 11 – Example Q3 200X

You get the picture; no? Let’s try the Bull’s Eye Chart, which shows the three projects in this program

• On-line sales have not started, so it remains at 1-1
• Call centre app is in the lower left corner, with DPI of 0.68 and RPI of 0.71
• The engine delivery is behind, at DPI = 0.68. In terms of benefits, the engine delivers a technical capability, and enabler, that contributes to the benefits of the other two projects but doesn’t have benefits on itself, which is why we represented it at RPI = 1. Alternatively, the engine could have the RPI of the overall benefits of the “program”, but the calculation of this is beyond the scope of this article.

Fig 12 – Example Bull’s Eye Chart

As seen in the example, Earned Benefit and RPI provide that metric that was not available before. Since RPI is based on planned and forecasted benefits, it can be measured at any time during delivery, as it will reflect both the impact of changes in the schedule, and adjustment based on changes in external conditions (i.e. state of the economy, or competitor entering the market, etc.).

SPI, CPI can be presented in grids against delivery risk, as RPI can be presented against benefit risk. Grids can also be built for each one of the three indicators against project stage (i.e. on-hold, planning, executing, etc.). What this means is that the three indicators that come from an expanded EVM system can be used as the foundation to put together indicators that relate to stage, risk, etc. This provides a very solid foundation for a framework destined to optimize the investment, as it covers both delivery (SPI and CPI), benefits, risk (delivery and benefit) and incorporates the time value of money. This is, by definition, a framework that focuses on optimizing return on investment from projects.

The three musketeers are complete: RPI comes to complement two venerable performance metrics that have been used for fifty years to measure delivery. The adoption of Earned Benefit and RPI are a natural progression to EVM and will likely become part of this comprehensive framework.

We have our trinity.