Author: Robert Galen

I was talking to a fellow coach the other day and she was venting a bit about one of her teams and their Product Owner.

Bob, she said, I have an outstanding agile team. We’ve been working within our product organization for nearly two years. In that time, we’ve delivered an application upgrade that everyone has viewed as simply fantastic. Now we’re onto a building a critical piece of new system functionality for them—so we’ve earned everyone’s confidence in our abilities.

We work hard, we work well together, we deliver high-quality working code, and we have fun doing it.

Ok, I asked. That sounds like a fantastic situation. To be honest, I’m a bit jealous.

Well, she said, don’t be. I’m incredibly frustrated with our Product Owner. He:

• always seems to be ill-prepared with stories;
• hardly ever emphasizes or attends Backlog Refinement meetings;
• brings new or under prepared stories directly to Sprint Planning;
• in general is always undecided and under prepared.

We’ve gotten into the habit of being able to work around him and still deliver sufficient customer value at each sprint review…but it’s incredibly challenging. The team has to “bend over backwards” to help develop a view to our backlog.

And he constantly changes his mind, which is the cherry on top – increasing our rework and driving the entire team crazy.

Wow. I guess everything isn’t so great. Then I asked, have you made him aware of the behavior and your (and the teams’) concerns?

Of course, she said. This comes up in virtually every retrospective. He listens, agrees, and then promises to improve. There might be a slight change for a sprint or two, but he always falls back into the same old patterns.

I’ve even tried to escalate it (carefully) to his supervisor. And even she agrees that it’s a problem, but again, nothing seems to change.

I’m at a loss. What do you think I should do?

Taking a step back

On the surface, this is sort of an intractable situation. It reminds me that every team, agile teams included, is only as good as their “weakest link”. Sure, teams can collaborate and work around internal challenges such as this. And in some cases that’s the right thing to do. But when the behavior is affecting the entire team’s morale and performance, and when there is no consistent improvement; then I say something needs to be done.

What do you think? And what are the options here?

1. It seems clear to me that the Product Owner’s manager is dropping the ball. There is a performance issue and (apparently) they’re ignoring it. One obvious and immediate action would be for his manager to begin taking it seriously and start actively coaching the PO, with a possible performance improvement plan as an outcome.
2. Could the team “vote the Product Owner” off the island (team)? I’ve seen this happen in extreme cases with team members. It’s a very messy situation and somewhat of a last resort. However, if the team really is being negatively affected, then it might be the most congruent thing to do.
3. Under the banner of “good team play”, should they just persevere along and keep dealing with it? That’s what this team has been doing. Sure, they’re talking about it. But it’s been happening for nearly TWO YEARS. What makes anyone think it will improve without a major intervention?

Overreact or Underreact

I normally see two reactions to similar situations in my agile coaching travels. Either the team grows impatient too quickly and overreacts to remove someone from the team, many times without having a conversation with the person. It’s just easier for them to complain behind their back and then look to eject them.

Or

Like the team in this example, tolerate the underperforming behavior in perpetuity, working around it within the team. Sure, they bring it up occasionally, but the team essentially accepts it.

Is there somewhere in “the middle”?

I sure hope so!

You might be asking yourself, what counsel did I provide to my coaching colleague? Well, I’ll share it with you. But I’m still thinking about the situation and whether my advice was sound.

I told her that the team was “working around” an issue and that they weren’t transparent enough with the effects of the under performance. You see, they were working overtime to cover for the Product Owner and from a results perspective; the team was perceived to be high performing.

I basically said to stop doing that.

My advice was to allow the Product Owner to fail more often and to make those failures more transparent.

For example, the Product Owner was essentially bringing empty stories to the team for execution. The team would then meet with stakeholders and analysts across the organization to fill in the blanks. This would take tremendous time that wasn’t theirs to offer. In the end, they got very good at writing stories FOR the Product Owner.

I told her to stop that, to put in place Readiness Criteria for stories that would enter the teams’ sprints and to not accept “under-cooked” stories. If the Product Backlog became empty as a result of this, then so be it. Raise that as an impediment and wait until the Product Owner did their job and worked with the team to deliver a proper backlog.

I was careful not imply that the team sabotage or not help the Product Owner as a healthy team would. But I was firm in that they needed to stop doing the PO’s job for them AND allow the dysfunction, under performance and honest results to become transparent to all.

I suggested that IF they did this well, then the organization would reach out to the entire team to discover “what was wrong” and to help them “fix it”.

Wrapping Up

Now at this point, we broke the conversation off and went on our way. But I could tell that my colleague was very uncomfortable with my advice. To be honest, so was I.

But I keep thinking that it was a congruent suggestion for handling a “weak link” that is affecting team health AND not improving. That if teams have the option of helping / masking the problem vs. making it transparent, they might want to reflect whether their masking it IS part of the problem.

That’s what I intended here, but I do agree that it’s a fine line to walk. My biggest fear is that teams read this and start throwing struggling team members under the bus. That’s certainly not my intention.

I wonder what you think of my story and my advice? And if you’ve had a similar situation, what have you done?

Stay agile my friends,
Bob.

Don’t forget to leave your comments below.

In a recent recording of our Meta-Cast, Josh and I went through a couple of questions from the audience. One of the questions surrounded what to measure for individual developers. To be honest, I was taken aback by the question.

You see, I’ve been preaching for years that when you move to agile metrics, you want to do three things:

(One) Move from individual team member metrics to a more holistic, whole-team view.

(Two) Move from measuring functional teams, for example test team progress, again towards holistic, executing agile team view.

(Three) That you’ll want to collect fewer metrics and focus them in four distinct areas of interest. They are:

• Predictability
• Value
• Quality
• Team Health

(And Four) That you want the metrics to be more output based or results based, than input based. For example, instead of being interested in “planned velocity”, be more interested in “resulting velocity” from your teams.

An addendum here, is that trending is much more important than any specific data point. Then recently I ran into the following thoughts around agile metrics.

Outcome Metrics

Gabrielle Benefield wrote about outcome as being what truly matters in this blog post. She makes the case for “outcome” metrics that focus slightly elsewhere than are traditional views. Here’s an excerpt:

Throughput tells us how fast we go, output tells us how much we delivered. But why are we doing any of this work? What are the problems we are trying to solve? What are the business results we desire?

By measuring throughputs and outputs, we are incentivising people to deliver more of them. More creates more waste. While some people may believe if they throw enough features out it may increase the probability of hitting their target, it feels eerily similar to the belief that if you put enough monkeys in a room typing for long enough they will produce the complete works of Shakespeare. This not only misguided, but is wasteful for our products and out planet. Yes you heard it right. Think of the amount of resources being used for all of those wasted features, let alone all the power they end up consuming as they linger around on servers for years. When the glaciers melt away, you know who to blame.

The one thing very few organisations appear to be measuring are Outcomes. Outcomes are the value we create. They range from wanting to increase revenue for a company, to improving the usability and learnability of a product.
In spite of all of this, we continue to measure what is easy rather than what is important. I think the reason is that it is not as simple as measuring throughput or output. You have to think, understand and test relentlessly.

Type of Metric

Based on her article and my own thinking, I’ll try to categorize each of the metrics as either:

Input – something defined or measures at the “beginning” of the pipeline. A typical measure might be planned values, for example, planned test cases.

Output – something that results from the pipeline being executed. Velocity is good example of an output metric.

Outcome – and to Gabrielle’s point, these are metrics focusing on experimentation and customer testing. For example, measuring actual customer satisfaction with a new releases feature set by leveraging a survey.

Metrics Brainstorming List

Most of these are wild and crazy ideas. However, I thought it might be useful to share some of them around what to measure in agile teams.

What might be even more interesting is what I haven’t listed. For example, individual metrics (test cases run per tester per day) or functional metrics test case run coverage per day per plan.

1. Features removed from the Product Backlog—over releases, quarterly, or perhaps as a percentage? Features de-scoped or simplified in the Backlog; same as above. These two are trying to show if we’re as willing to subtract as we are to add. (Value, Output)
2. Stop the line events for a release or across an organization; could include Continuous Integration / Continuous Deployment stops and/or other “process” stops. Clearly a “quality centric” sort of look into the organization. (Quality, Outcome)
3. Root Cause discovery sessions conducted per team – per Release; could view types of corrective actions and correlate patterns across teams? Again a quality-centric metric—are we truly focusing on continuous improvement? We could also consider the sheer number of issues and their trending over time. (Quality, Outcome)
4. Number of Retrospective Items resolved per team. I’ve often wanted a way to look at the retrospective and still maintain the integrity/confidentiality of it for the team. I wonder if this would work? (Quality, Output)
5. Number of stories delivered with zero bugs – per Sprint, per Release? And by zero bugs, I’m implying newly introduced bugs. Is the team holding to their agile quality values? (Quality, Output)
6. Number of User Stories that were reworked based on PO / Customer review; perhaps even maintain a “Cost of Rework” factor. There is a range here where this is very healthy. But it can also become repetitive and unhealthy. How would we determine that? (Value, Output)
7. The percentage of technical debt addressed; targets > 20%. You’d need to be clear about what fell into the “technical debt” bucket. (Value or Quality, Outcome)
8. Trending of velocity per team, perhaps a rolling average in story points. What’s interesting here is release-level predictability per team in points. Avoid aggregation of results across teams (organization-level) and beware of the affect of team turbulence. NO individual team member velocity measurement! (Predictability, Output)
9. Time-stamp types of work as they move through your teams—capturing throughput per story size. Then you’ll have a range and average/mean for story throughput. You can choose from this to forecast your release-level commitments. Avoid aggregation of results across teams (organization-level) and beware of the affect of team turbulence. (Predictability, Output)
10. Delivery predictability per sized user story; average variance across teams and is the trending improving? This is the non-specific variant of the above—simply looking at raw story delivery variance. (Predictability, Output)
11. The percentage of test automation (including UI level, component / middle tier, and Unit level) coverage. Or instead of coverage, you could show the ratio of planned vs. running automation. Trending here would be the most interesting bit. (Quality, Output)
12. The percentage of each sprint spent on automation investments. The percentage of each sprint spent on Continuous Integration and Continuous Deployment investments. Again, trending over time would be the most interesting. (Quality, Output)
13. Team agile health survey data; monitor trending & improvements. Happiness Factor of the team 
14. Training budget per agile team member. Simple. Perhaps look at year-over-year trending. (Team Health, Output)
15. Agile Maturity Survey – there are a wide variety of these sorts of tools developed. I lean towards the Agile Journey Index developed by Bill Krebs. The AJI strikes a very nice balance between measuring but not being too heavy handed in reacting to team-based maturity and evolution. (Team Health, Output & Outcomes)
16. Customer usage of delivered features (actual usage). Somehow instrument your application or product so that usage can be collected and analyzed. For example, Google Analytics for web pages. Establish Product-driven Business Case values and then actual values. The delta might be interesting. (Value, Outcome)
17. Customer survey’s to identify value delivered (actual, not wishful) Perhaps use some sort of Net Promoter Score? (Value, Outcome)
18. Dedicated Scrum Masters vs. Multi-tasking Scrum Masters (Dual or more roles). Create some sort of ratio that represents your investment in the core roles for agile transformation. Could extend this to coaches and Product Owners as well. (Team Health, Output)
19. Number of failures due to teams experimenting, taking risks, stretching, etc. I might be looking for something greater than zero. Could also make this cross-team or organizational. (Value or Team Health, Output)
20. Measure organizational commit levels to date driven targets/expectations. This is an indicator of the level of planning within your organization and who is “signing up” for the credibility and feasibility of the plan.
    1. 1, senior leadership only
    2. 5, plus – directors & managers
    3. 10, plus – team leads
    4. 15, plus – whole team
    5. 25 plus – integrated across ALL contributing teams
    6. 50, EVERYONE (contributing to the release) thumbs up…
       for any delivered release
       (Predictability, Output)
21. New test cases added per Release or per Quarter. Retired test cases removed per Release or per Quarter. Both focus towards the effort to make our testing relevant in real-time. (Quality, Outcome)
22. Stakeholder & Senior Leadership attendance at Team Sprint Reviews. Could measure distinct feedback from this group, rework driven by feedback, and pure attendance. Point is, how often are the “right people” in the room for the review? And were they engaged? (Value, Outcome)
23. Team Churn – measuring internal and external changes (impacts) to a team sprint over sprint. At iContact we had a formula for this that created an index of sorts. I simply can’t recall. But it’s a powerful metric because it’s basically “waste” and slowing the team down. OR you could correlate with velocity. (Predictability, Output)
24. Backlog grooming and look-ahead is an incredibly important team activity. There are also milestones where you want to have groomed the work for the next release – within the current release. Track backlog grooming frequency and the pace of story analysis through it. I often use an egg timer to reinforce “crisp” grooming of stories at a 5-8 minute pace per story. (Value, Output)

Wrapping Up

Remember the title of the article. These are potentially weird, outrageous, or downright silly metrics ideas. I’m not sure if any of them deserve serious consideration in your agile projects, teams, and work.

Now I wouldn’t have written them down and shared them if I didn’t think they had “potential”. Still, I’d like to hear back from you on their viability as high quality, agile-centric measures.

• Which ones are viable or do you like?
• Which ones aren’t?
• Do you have any “pet” metrics that you’d like to add to my list? Please do so.
• What do you think about the notion of: input, output, and outcome metrics? Does it matter?
• And if you had to “boil it down” to 5 simple metrics to measure High-Performance agile teams, what would you look at?

I’m looking for any and all feedback, so please give it to me.

Stay agile my friends,
Bob.

Don’t forget to leave your comments below.