Skip to main content

Author: Robert Galen

Agile Risk Management—Viewed Through a Different Lens

I often find myself teaching various classes on agile methods. One of my more popular sessions surrounds the mapping of traditional project management techniques towards the agile methods. I do this quite often in a wide variety of venues. One metric I’ve noticed is that the more PMPs in the audience the more spirited the discussions become.

One of the core translation areas between traditional and agile project management relates to risk management. I often get a lot of pushback from the PMs telling me that the agile approaches to risk management simply won’t work. Their arguments are more based on traditional thinking, PMBOK guidance, and “we’ve always done it this way” pattern; rather than a situational response to individual project needs. I usually just leave it that agile risk management is “different” and move onto safer topic areas. But I usually want to add more flavor and this post seemed like a good opportunity to do so.

Traditional Risk Management

In this view, the project manager is managing the risk. The premise is that you need a highly skilled and experienced PM to adequately control and manage project risks. That risk can be anticipated, planning, triggered, and mitigated.

One of the first activities with any project is to begin the compilation of a risk list so one can manage the project risks. These risks can be gleaned in a wide variety of methods—team brainstorming, speaking directly to key stakeholders, analyzing previous projects, and from the technology and business climate.

Once identified, a common method is to evaluate the size of each risk. A common mechanism is to gather likelihood and impact from the project team, then multiply the likelihood of the risk occurring against the impact the risk would have.

Potential Risk Likelihood of Occurrence: 0 – minimal / none, 1, 2, 3 – Certain! Impact of Risk: 0 – minimal / none, 1, 2, 3 – Project Failure! Risk Priority
50% of technical team will leave for a competitive position 0 3 0
The open source LAMP stack will be acquired by Oracle 1 2 2
Chance we will deliver 100% of the agreed functionality? 3 1 3
System Performance will not meet requirement levels; only 50% 2 2 4
Business stakeholders will be confused on required features 3 2 6

Once you’ve accumulated a “complete” list of risks and analyzed their “priority”, then a plan is put in place to detect and mitigate the risks that are most likely to occur. Quite often, this is a very detailed plan that is formulated with all of the project’s functional teams—expending quite a bit of time and effort towards these hypothetical risks both in upfront planning and in ongoing monitoring and discussion.

A Quick Point of Context

Oh, and I need to get this point out of the way. My primary context for this post is technology projects. If you’ve managed a technology-driven project, IT project, development effort, integration project, etc., you will certainly agree that these beasties are “different” than other types of projects. And yes, their risks are different too. Rarely can you predict risks early on in these projects. Why? Because you simply haven’t done it before—so you have little to no experience with this specific type of project or technology within the team chartered with implementing it.

The variability in complex software projects is what undermines traditional risk management in my view. We try to expend all of our effort in up-front risk management. Rather, we should expend most of our efforts towards figuring out what we do and do not know. We should decide how to design and code this particular widget I.e. do some of the project work before trying to predict risk.  Let the risks emerge from our efforts rather than trying to predict them.

Agile Risk Management

This nicely leads into the essence of agile risk management being an emergent approach. First of all, you rarely hear the agile methods focus on risk at all. Why? Because we flip the notion of planning risk on its ear a bit. How is that? Well, instead of guessing or planning for risk, one of the central themes of the agile methodologies is to deliver real working software in very thin slices of functionality in time-boxed iterations. Realization of risk occurs quickly and abruptly. It hits you in the face as a team at the earliest possible moment. Quite often it surfaces in a daily stand-up meeting.

Is it solely the responsibility of the project manager? No, in the agile case the self-directed team is primarily responsible for detecting, acting on, and mitigating risk…and here’s the important part, in real-time, as each risk occurs. It’s a whole-team and reactive view towards risk.

The team often engages in strategies surrounding risk by how they approach iteration planning. They have the choice of what to build first, so very often the development strategy is to load risky items first—

  • To do early experimentation to see how the team will respond to technical challenges and the unknown.
  • To measure the velocity of the team to understand their capacity to meet business milestones.
  • To engage stakeholders in assessing requirements as they evolve…in real-time and on working software.

Conclusions

Now all of that being said, I don’t think we throw out all of the traditional approaches in agile contexts. For example, I think it a very healthy exercise for larger-scale agile projects to assess and understand the Top 10 risks they might be facing. In addition, to also look for more systemic or organizational risks and do a bit of up-front planning.

But don’t spend too much time there. Nor in exhaustive detection strategies or mitigation plans. Setup a straw man risk structure and then start leveraging the emergent nature of agile iterations to surface risks quickly.

Now for you traditional project managers listening in, I wonder if some of these agile approaches might be applicable in your current contexts! 

Don’t forget to leave your comments below

The ‘Essence’ of Agile Metrics. Part 2.

Let’s go back to the example I gave at the end of my last post. What is wrong with measuring story volatility? On the surface it appears fine. We can see how well the product owner staged the sprint and also how well the team is interpreting stories for execution. For example, if they “signed up” for 10 stories, but only delivered seven because of story (requirement) volatility, then they have a lot of improvement to do…right? Well, that could be the case.

But it could also be the case that the team trimmed those stories because they were superfluous to the customer’s needs. Meaning they were essentially waste. Or another positive interpretation of it could be the team became very creative and figured out a way to meet the spirit of the sprint goal with fewer stories. In fact, they may have even trimmed out some previous code in order to simply the application – so the overall related LOC count might be quite low too.

Can you see the point I’m making? I find that leading indicator metrics can sometimes be more prone to metrics dysfunction than trailing, results oriented metrics.

In his book, Measuring and Managing Performance in Organizations, Robert Austin coined and explored the term metrics dysfunction. In simple terms, it’s a metric whose collection influences the very behavior being measured, leading to metrics dysfunction. One of the classic examples of it is measuring software testers by the number of bugs they find. In this case, testers would report more trivial, non-germane bugs simply to hit the metric. In the meantime, they’d miss truly important bugs. Clearly not what the measurer intended!

In our example, the leading metrics might influence the team towards not changing stories that need to be changed, simply to hit the metric of stories being consistent with the views from planning. I can see nothing more destructive to the agile mindset of the team than having metrics that drive negative behavior within the team simply to meet or game the metrics.

So, what are some unhealthy leading metrics to avoid within agile teams?

  • Planning quality – Measuring how well you planned. An important part of this is change control, i.e. measuring how little or much change is ongoing.
  • Requirement quality – Measuring whether each requirement meets some sort of baseline definition of completeness. Or that each has a specific number of acceptance tests.
  • Estimation quality – I’ve sort of beaten this down in the article. Effectively it’s anything that tries to measure estimation variance with actual effort.
  • Arbitrary results – Counting LOC produced, or Bugs found, or Requirements written, virtually anything that is materially produced by the team that negates the quality of the result and the application of common sense, since not all results need the same level of attention and thinking.

Conversely, what are some healthier trailing metrics to concentrate on within agile teams?

  • Team agitation levels – capturing multi-tasking events, # of committed hours per sprint and variance from expected levels, recruiting support for team members
  • Team velocity levels – trending over time, improvement trending (implying improved team dynamics), paying attention when team composition changes
  • Impediment handling – #’s per team, avg. time to resolve, # that impacted the Sprint Goals
  • Retrospective actions – is the team improving themselves based on retrospective results, how many improvements per sprint, average time to resolve
  • Escapes per sprint – bugs found post-sprint, adherence levels to Done-ness Criteria
  • Sprint Success / Failure – from the perspective of the Sprint Goal. Not so much focused at a Story or Task completeness level, but at an overall work delivered level

One of the things you’ll notice with trailing indicator metrics is that there is an inherent real-time nature to them. We want to be sampling them (observing, discussing, driving action) each and every day during each iteration or sprint.

It’s this level of team awareness of their performance (metrics) and active engagement towards continuous improvement that is a key for functional metrics in an agile context.

The ‘Essence’ of Agile Metrics. Part 1.

AgileMetricsThis is my initial blog for Project Times and I’m certainly happy to be joining the community. So, a little about me.

My name is Bob Galen and I’m from Cary, North Carolina. I’ve spent the greater part of 25 years doing software development—in roles such as Developer, Software Director/ Manager, Test and QA Director/Manager, and Project Manager.

For the past 10 years, I’ve been moving my thinking and practices towards those espoused by the Agile Methodologies. I’ve found them to be the most congruent, humane, and effective approaches for delivering software and I try to share my insights and excitement with everyone I meet.

Perspectives – you’ll see a strong skew in my posts related to the following topic areas:

  • Implications of agility for the project manager
  • Implications of software testing for the project manager
  • And leadership and soft skills topics for the project manager

and I also blog for Business Analyst Times on similar topic areas. So that gives you a feeling for coming attractions. But, enough of that…

I’ve been struggling for quite some time figuring out the “essence” of agile metrics. What are good ones versus bad ones? Are there related sets that are relevant and how many are needed? And how much change do we need to make from our traditional metrics thinking towards agile metrics?

Another thought I’ve had is whether they should represent a trailing indicator or a leading indicator? Meaning, should we measure the processes (at the beginning or leading edge) or should we more so focus on the results and learning (at the end or from the trailing edge)?

For example—

I think of measuring estimate accuracy (estimate + actual) as a leading indicator. Why is that since you measure actuals at the end of your work?

I’m thinking of it from the point of view of what you’re trying to improve. In this case, you’re measuring estimates and actuals to improve your overall planning and estimation processes. These activities are typically front-loaded actions and are often static…meaning they are completed once at the beginning of the project.

The other challenge with this metric is taking corrective action for improvement. Effectively, you can only effect a change with the next planning interval for a project. Depending on the length of that interval, improvements could be infrequent and delayed. Or depending on project success, they might not be useful at all.

So I’ve seen some agile teams that try to measure various things. Let’s take a typical example.

How about if we measure user story volatility in the sprint? What we’re trying to focus on is how many times do stories change (get added to, become reframed or reduced in scope, or removed entirely) within any particular sprint. To what end? Well, we’d like to improve our story grooming process and monitor how the team is delivering on their sprint planning commitments.

Ah, there’s a key! Sprint planning is sort of a leading indicator, in this case you’re measuring the quality of story entry. What if we turn this into a trailing, more results oriented, measurement? Let’s measure the velocity of the team as they exit the sprint. We can measure velocity in stories produced and in story points delivered.

An even better trailing metric is to measure how the team delivered value towards their sprint goals. Meaning, did they deliver what they committed to? Not in tasks or in stories, but in content that exactly met the spirit of their customers needs and the level of their commitment.

You notice that I reframed the notion of a trailing metric to a results oriented metric. Another way of thinking about it is measuring at the “outbound side” of the team’s work. That way, you’re leaving the doing of the work up to the team and simply measuring results or outcomes. To be continued…

Don’t forget to leave your comments below