Simplified Agile EVM – The Art of Managing Triple Constraint
In the last few years, the project management community has garnered significant attention by the constant dilemma of choosing between Traditional Proven Ways and Agile Based Approaches.
On one side there is choice of extensive planning predictability and controls necessary to manage the projects. On the other side there has been an increasing push to drive innovation and adapt to changes. This constant conflict is often over simplified leading to bifurcated opinion with less guidance on how to strike the right balance between two extremes.
This paper introduces the SAEVM model that focuses on keys to de-risking the delivery capabilities of the project/organisation by adherence to some of the traditional project management fundamentals and practices and at the same time embracing the agile practices for true value creation that can change the project outcome.
Challenges
We often notice that middle to large size programs often run into trouble due to their inability to change with the dynamic and ever-changing market requirements. The need to remain relevant and to keep up with the competitive pressures is enormous. Our customers are now more informed and they demand more specific or personalized services, which means that to gain a competitive advantage the Project Management group must constantly look for innovation, latest tools and techniques, to gain newness, add value and customer benefits.
The Agile Project management community constantly faces the challenge of cost overrun and schedule delays. Many Agile practitioners will agree that there is also a suspicion that some organisations have taken major implementation decisions without understanding the heart of Agile methodologies, practices and how best to implement them within the organization. Also, the reduced need of project / program governance and documentation may lead to less clarity around the requirements and monitoring of the triple constraint.
A need to support project managers for enabling them to manage triple constraint or project management triangle effectively
In addition, there is an increased awareness and expectation from stakeholders and customers towards moving to the agile based models. Even to the extent that both customers and stakeholders do no longer accept any traditional models as a possibility, even in situations where the organisational environment or project context is not supportive to an agile culture. The effect of the same is that the contracts/commitments still are made from traditional mindset with limited or zero flexibility on Scope, Budget, Schedule, or Quality.
Although the context is slightly different in product and services organisations but there is limited difference in the challenges project managers face and hence there is a need to support project managers for enabling them to manage triple constraint or project management triangle effectively.
Facts and Figures
Some of the facts and figures below will state that an alarming number of projects that have failed at some point or the other stage of the project due to – (Source: PMI)
- Change in the organization’s priorities (39%)
- Change in project objectives (37%)
- Inaccurate requirements gathering (35%)
- Inadequate vision (29%)
- Poor communication (29%)
Some other trends show us that –
IT projects, failure rate corresponds heavily to project size. An IT project with a budget over $1M is 50% more likely to fail than one with a budget below $350,000. For such large IT projects, functionality issues and schedule overruns are the top two causes of failure (at 22% and 28% respectively) – (Source: Gartner)
A PwC study of over 10,640 projects found that a tiny, tiny portion of companies – 2.5% – completed 100% of their projects successfully. The rest either failed to meet some of their original targets or missed the original budget or deadlines. These failures extract a heavy cost – failed IT projects alone cost the United States $50-$150B in lost revenue and productivity – (Source: Gallup)
Investing in proven PM processes and methodologies pays off. According to CIO, organizations that use proven PM practices waste 28x less money than their more haphazard counterparts – (Source: CIO)
The Solution
Tackling the above challenges requires shared effort from both Agile believers and the traditionalist EVM practitioners.
Our paper will explain that the best practices of both agile and traditional EVM models that can be encompassed to effectively manage the triple constraint.
The traditional project management community uses an approach to handle the triple constraint that further translates to work breakdown structure, cost breakdown structure and organization breakdown structure with the objective to give emphasis to provide the project teams with capabilities to manage, monitor and control them. The traditional EVM Implementation works well here to help the command and control style of project management.
can be the new way of working for project delivery teams that can be a solution such that “one size can fit all”
Agile teams are self-organised and the approach towards managing deliveries is very different. Agile teams play an active role in managing triple constraint by focusing on the progress with support from agile metrics and artifacts like Velocity, Burndown, Burnup charts. The traditional EVM concepts do not directly fit here.
To bring in the best of both worlds together SAEVM can be the new way of working for project delivery teams that can be a solution such that “one size can fit all”. While staying true to the spirit of Agile it can provide support to the organisations for bringing much higher success and effectiveness in handling wide variety of projects.
The Model
The SAEVM is based on the traditional EVM applied to the projects with an agile context. Although the projects can be very different and the agile maturity and application also varied, but the model can still be applicable in most scenarios.
Following are the four core components of the model –
• Cost Management using SACPI
• Schedule Management using SASPI
• Scope Management using SARPI
• Quality Management using SAQPI
SACPI and SASPI are derived from traditional EVM, SARPI is derived from Agile Metrics and SAQPI is derived from General Software Engineering Practices. These KPIs are described in detail subsequently and are the key drivers of the model itself.
Variations
The current IT world is very complex with a huge variety of projects.
Following are the project type variations based on type of agreement with customers –
1. Fixed Price Projects – There are several sub types within this, but for simplicity we would consider these as the projects with fixed cost, scope, and timelines. The customer takes a commitment up front on the project and there is no possibility of missing on any of the parameters. Although these types are not recommended for agile based projects, but it is a reality that these are still in abundance and customers are generally not comfortable to get into other contract type for a new engagement.
2. Time and Material Project Managed – These are customer engagements were project management responsibility lies with service provider and billing is based on Time and Material. Although these are better than the above type for agile projects, but here also a focus at project level is required and there is an implicit expectation to comply with the project budget and timelines and initially understood high level scope.
The model can be applied with ease and equivalent effectiveness to a wide variety of project category and types
3. Time and Material Sprint Managed – These are customer managed projects with Sprint responsibility expected from the team. Budget is fixed, Sprint Duration is fixed, and a constant velocity and sprint goal compliance is expected from the team.
4. Time and Material Pure Staffing – These are customer managed projects with resources from the service provider participating in customer agile teams.
There are many more project types including combinations and variations of the above types, but for simplicity we have tried to cover the most common types only.
The above context is based on services organisations. Although the context for product organisations might be different as it generally would involve internal funding but expectations from internal stakeholders /customers would be similar and hence same approach should work.
Categorisation
In the previous section we described the project types based on agreement with customers. In terms of applicability of the SAEVM model we would like to categorise, projects into the following broad categories –
1. Project Focused – These are projects where the expectation is to manage the project delivery at overall project level. Type 1 and 2 above fall in this category.
2. Time Bucket Focussed – These are projects where the expectation is to manage the delivery at sprint level or time bucket level. Type 3 and 4 above fall under this category.
The application of the SAEVM model is similar in the above two categories. The model can be applied with ease and equivalent effectiveness to a wide variety of project category and types.
Calculations
Base Metrics
These are the metrics that are required to be captured for all projects at the Project or Time Bucket level, all other metrics described could be derived from these metrics only.
These can be captured at a desired project tracking frequency, but we would recommend a weekly frequency.
Re-Planned Size (PSP)
The total size of the project can be measured in any size unit e.g. FPs, Use Case Points, Feature Points, High Level Story Points. In the absence of a better unit for size estimation, size can be in measured as Ideal Hours/Original Estimated Hours. In case, there is a change of scope (requested and approved by customer), the size can be revised based on new estimates, otherwise it should not change during project execution for Project/Time Bucket.
Completed Size (ASP)
The Sum of size satisfying DoD till date, the unit used should be same as the PSP. This gives a measure of work done till date upto a good level of accuracy. This should be cumulative of the entire Project/Time Bucket scope and partially completed stories/work should be avoided.
Total Duration (TST)
Total duration of the Project/Time Bucket from start date to end date. Calculated as total working Days between Start and End Date of the Project/Time Bucket. For Scrum Based projects this can be taken as the Sprint Count (no. of sprints) for the project. The time duration taken is with the assumption that the team would be working with a uniform pace. Initial preparation period/Sprint 0 or post development period should be avoided.
Completed Duration (CST)
Total duration of the Project/Time Bucket from start date to till date. Calculated as Total Working Days between Start and Date of reporting of the Project/Time Bucket. For Scrum Based projects this can be taken as the total Sprints completed Count for the project. The unit or basis for this metrics should be in line with TST.
Total Budgeted Cost (TBC)
Total cost or Efforts budgeted for the project/Time Bucket. This parameter is an indicator of the target cost for the project/Time Bucket.
Total Incurred Cost (TIC)
Total cost or Efforts consumed for the project till date for the project/Time Bucket. This parameter is an indicator of the incurred cost for the project/Time Bucket. The unit or basis for this metrics should be in line with TBC.
Scope Change (RSP)
Total scope changed within sprints/time bucket. This indicates the cumulative Size of the scope changes during sprints/time bucket. This is important for agile based projects where we do not expect scope changes within the sprint/time buckets. This is to be cumulated for Project/Time Bucket.
Delivered Defect Count (UDC)
Total Defects reported post-delivery to the customer. This would be usually taken as defects delivered to User Acceptance Testing and Production.
It is important to note that there is a complex relation between all the above factors and care should be taken to align them to similar context. E.g. we cannot take PSP and ASP at Project Level and TST and CST at a Time Bucket Level.
Derived Metrics
Planned Value (PV)
This indicates the work expected to be completed till date of reporting in % value, based on the schedule. With the assumption of uniform sprints/time buckets this can be taken as the ratio of Completed Duration to Total Duration –
PV = CST/TST
Earned Value (EV)
This Indicates the total value earned till date of reporting. We consider this as an indicator of the actual work completed till date in % value.
EV = ASP/PSP
Actual Cost (AC)
This indicates the actual cost incurred till date of reporting w.r.t. total cost budgeted, expressed in % value. This is taken as the ratio of Total Incurred Cost to Total Budgeted Cost
AC= TIC/TBC
Advertisement
[widget id=”custom_html-68″]
[widget id=”custom_html-68″]
Key Performance Indicators
Cost Performance Index (SACPI)
SACPI reflects how well are we in control of our budget for the project. This is a leading indicator and is fairly accurate in providing insight about the potential budget compliance at project closure. SACPI, if calculated accurately can provide correct indications as early as in the first Sprint/Initial Stages on cost performance for the project. This gives the project manager adequate time and opportunity to take steps to bring the project back on track.
It is considered as the ratio of Earned Value to Actual Cost
SACPI = EV/AC
Schedule Performance Index (SASPI)
SASPI reflects how well are we in control of our schedule for the project. This is a leading indicator and is fairly accurate in providing insight about the potential time compliance at the project closure.
SASPI if calculated accurately can give indications as early as in the first Sprint/Initial Stages on the projects Schedule performance, which in turn gives adequate time and opportunity to the project manager to take adequate steps to bring the project back on track.
It is considered as the ratio of Earned Value to Planned Value
SASPI = EV/PV
Scope Performance Index (SARPI)
Defining performance on scope management is tricky for agile based projects. In traditional models scope management is directly aligned to controlling scope creep and effectively applying the change management process for the same.
In agile based projects changes are accepted as a norm and might still require to be managed, but we do not consider it as a right measurement for scope performance.
Instead what we would like to establish is that the requirements are managed effectively with right level of details and compliance to timelines for backlog refinement, so that there is no scope creep within sprint of the project.
The requirement or scope changes within a sprint might be a sign of ineffective scope management and might have energy losses and have an impact on project commitments.
In view of the above, we would like to define the scope performance index as the ratio of cumulative stable requirements in the sprints to the total scope delivered in the project.
SARPI = 1- RSP/PSP
Quality Performance Index (SAQPI)
Quality of delivery is at the centre of the project management triangle and arguably the most important parameter in the IT world.
The right measure must be around delivered quality to the customer and defects can be a good indication of the same.
This metrics is considered as the
SAQPI = 1 – [(UDC/PSP)/(Baseline Defect Density)]
Where Baseline Defect Density is the organisation defined baseline. In the absence of the same it can be defined at the project level based on historical data.
The above four KPIs can be utilised in the following manner to assess project health at any point of time in the project execution to derive inferences and take actions to keep project in control and lead towards successful closure
Case studies
Introduction
The SAEVM model has been successfully applied to projects for a period of more than a year with huge success. The model was used in all different categories of projects.
The improvement in project execution in terms of core delivery KPIs has been massive on the scale of 30% to 50% within short span of time.
Following are some example projects to share the experience of applying the model. The actual names of the project has been taken off for confidentiality reasons.
Case Study – Project 1 – Project Focused
Case Insight
This project was “Type 2 – Time and Material – Project Managed” with an expectation of Project Level commitments for Scope, Time and Budget.
This project was for one of our European clients. After a few initial discussions with the customer, it was understood that the requirements were very fluid and non-structured.
We started applying SAEVM to the project with two week sprint cycle and annual release roadmap with releases spread throughout the year.
The project started with Quality slightly below par, Scope Performance also slightly below par, other core KPIs completely offset to really low value with SACPI at 0.72 and SASPI at 0.55.
Implementation
The project can be considered in three stages of performance.
Stage I: The initial stage of the project involved a lot of struggle. The SASPI and SACPI clearly indicated the productivity of the team was low and it impacted both on cost and schedule and created a lag. We looked internally towards improving the effectiveness of the team. With continuous focus and steps both SACPI and SASPI could be brought well above healthy levels.
Stage II: At mid stage our project started showing satisfactory cost and schedule indicators yet quality parameters showed no signs of improvement. The team was augmented to support our delivery quality. As a result, the SASPI started to further incline but the SACPI declined. With continuous focus on quality and reduce rework, by the mid of this period quality returned to healthy levels and additional members could also be released from the project and the SACPI started to incline to a reasonable satisfactory level.
Stage III: This was the final stage of the project. This stage was relatively smooth as the core project performance was gradually settling down. However, the key challenge that we faced at this stage was a sudden peak in attrition of important resources. With mature functioning of our improved agile team the new member were brought to speed and by the end of the project, all essential project performance parameters reached an healthy levels with SACPI at 1.02 and SASPI at 1.0.
The above stages were very typical to this project but with the help of the SAEVM model different project situations can be handled with the insights provided by the core KPIs.
Case Study: Project 2 – Sprint/Time Bucket Focused
This project was “Type 3 – Time and Material Sprint Managed” with an expectation of Sprint Level commitments for Scope, Time and Budget.
Case Insight
This project followed two week sprint cycle and continuous periodic releases spread throughout the year.
The project was Driven by a competitive and ever-changing business landscape, the requirements coming from the customer were extremely unclear at the start.
The project started with quality maintained at healthy level with one spike at the initial stage. The scope performance remained at a healthy level with one spike. All other core KPIs completely offset with high variations on sprint on sprint basis. SACPI remained around 0.5 and SASPI around 1 with variation from 0.5 to 2.0.
The data points are on sprint level and hence the variation is understandably much higher as compared to long duration projects as the previous sample project.
Implementation
The project can be considered in three stages of performance.
Stage I: At the initial stage the project suffered, high variation in SASPI and SACPI which clearly indicated that the predictability of the sprints were low. The low average value of SACPI indicated that the productivity of the team was low and clearly team was incurring more efforts than budget to meet their time commitments.
With continuous focus and steps both SACPI and SASPI could be brought well above healthy levels by the end of this stage with moderate variation.
Stage II: At the mid stage, project showed overall improvement in SACPI and SASPI variations but started getting some spikes on SARPI and SAQPI. Upon further investigation and with certain root cause analysis of the spikes and process adjustments, it was brought back to the healthy levels for all KPIs by the end of this stage.
Stage III: This was the final stage which was relatively smooth, previous process improvements seem to have paid off. The variation reduced and the average value of core KPIs very close to desired value 1.0. Project got completed with both SACPI and SASPI closing at an average around desired 1.0 with variation much better improved, both other KPIs closed at near 1.0.
The above stages were very typical to this project but with the help of the SAEVM model different project situations can be handled with the insights provided by the core KPIs.
Inferences and Takeaways
- Although the execution of the model was very different in two different categories but we could establish effectiveness and overall performance improvement in both scenarios.
- The model’s effectiveness lies in the actions taken based on the measures provided, as with any other data driven models.
- Since model is heavily dependent on data it would be crucial to have a good data quality and care should be taken for the same for effective implementation and positive results from the model.
Conclusion
It is undeniable that customer expectations are changing every day. To create a winning proposition for our customer is the key to earn the competitive edge. At the same time, we need to minimize the unknown unknowns. Setting up proper control mechanism and gain right visibility, we need to have the correct indicators in place for our project work; that will be essential to achieve the right customer value within the existing triple constraint of the project.
Whether a project is delivered via traditional or agile methods, only the outcome verifies whether the intended business value is achieved or not. Requirements will change rapidly as per the changing environment. In a fast altering environment, the risk is high, and projects and programs are complex.
SAEVM is a low risk of implementation and it coincides with the ethos of Agility. On the other hand it acts as a control mechanism for project deliverables. In the process it brings in excellence from both worlds of Agility and Project Management.
With the model being applied and its effectiveness established with a wide variety of projects, it can be taken further to be applied into all different contexts with ease and in the process derive success on projects helping the business.
About the Authors
Dinesh Sharma
Over 22 years of experience in IT Industry with more than 15 years in Project management and more than 7 years into handling agile based deliveries. Extensive Project, Program and Delivery Management Experience comprising of end-to-end execution of complex high risk projects.
Excellent track record with different type of projects – Development, Testing, Support, Transition, IT Setup; all with 100% success rate.
Demonstrated Exceptional Team Management and Leadership capabilities – handled large teams, managed an excellent team motivation and a low attrition rate, applied leading through Innovation and Leadership by example.
Multicultural Customer handling experience – worked for customers and customer locations across Europe and US. Not limited by any Technology or Business domain – Managed Projects in varied technologies and business domains with equal ease, maintained hands-on experience in core technical skill areas.
Conducted training and seminar organisation wide, written articles/whitepapers and delivered talks on global platforms and conferences on different topics related to Domain, Technology and Process.
linkedIn: https://www.linkedin.com/in/hidineshsharma
Jayeeta Dutta
A Project Management Professional, with extensive experience in Project Management, Business Analytics and Program Governance in IT Consulting. HealthCare domain, Finance Global Markets, Insurance in Techno-functional domain.
Strong business acumen, ensuring strategies and projects are aligned to the business objectives that yield high ROI and impact.
A total of 12+ years of professional experience with 9+ years in Project Management carries a strong understanding of Project Management methodologies, Program Integration, Resource management
Program governance and PMO Practices across industries.
Successfully carried out several techno-functional projects from end to end in Finance systems, Program Integration, system migrations, Business transitions that involved Solution designing, Resource Management, Planning and Implementation etc.
Strong acumen in reporting and governance skills and have developed strong business relations with business partners and teams located in various geographical locations. A green belt certified Project manager with several successful implementation of Quality projects.
linkedIn: https://www.linkedin.com/in/jayeeta-dutta-9108281a/
References:
The Agile Manifesto: http://agilemanifesto.org/
Ken Schwaber and Jeff Sutherland-The Scrum GuideTM: https://www.projecttimes.com/wp-content/uploads/attachments/2017-Scrum-Guide-US.pdf
Gallup Survey: https://www.gallup-international.com/survey-results/
