Agile Processes Go Lean
Two years ago, the company where Charles Suscheck worked purchased the rights to a third-party software platform and launched a project to test, debug, and enhance it to fit into its business lines. But developers at the company hit a roadblock: Requirements for the system were fuzzy, continually changing, and not based on a solid understanding of the product, says Suscheck.
At that point, the company turned to agile development, a methodology based on iterative development and team collaboration. Initially, the entire team — developers and business stakeholders alike — faced a steep learning curve for both agile development and the management and billing product. But once everyone understood agile, a process that breaks big projects into smaller segments and allows software features and plans to be adjusted as development projects progress, the new methodology proved its worth.
“Agile is perfect when you’re not sure what you’re getting into,” Suscheck says. “If you run into a big roadblock, you’re only derailed, on average, a week’s worth of work.”
From the outset of the project, meandering requirements were a problem for the team before implementing agile development, Suscheck says. But once it adopted the methodology, the group found it could add new requirements as it learned about the product, and the process of developing requirements became more flexible. Fleshing out requirements became a just-in-time process, meaning the development team learned about the system from one iteration, and then addressed requirements that would be pulled into the next iteration, Suscheck says.
Suscheck sees a viral effect from his group’s initial agile adoption, as others in the company pick up the methodology for projects from ongoing maintenance to greenfield development. Suscheck expects to see this increase in agile adoption continue.
Not Trouble Free
Experiences such as this are increasing agile development’s popularity, with almost a third of companies using Agile. In a recent Forrester Research poll, 30% of respondents were following an agile method, 32% were using an iterative approach, and 38% a more traditional waterfall method.
The agile approach was created out of the need to build software faster and at a higher-quality level. It did away with development processes built around specialized roles for the people involved, departmentalization of the process, and an emphasis on customer contracts and extensive documentation. Instead, agile champions team-oriented, customer-focused, collaborative, and iterative practices.
But agile methodology isn’t without its problems: Many companies have found it difficult to use and error prone. The problems, broadly, tend to be technical, cultural, or organizational. Technical problems relate to infrastructure, tools, and architecture, and are often the most visible of the changes involved. But many companies find the cultural and organizational issues are the hardest to resolve. When a development organization uses the approach to transform itself, it often runs into problems with other parts of the company that haven’t undergone similar transformations. Clashes come with the business operations, governance, and organizational structures, among other areas. For example, hierarchical organizations may struggle with agile development’s quick, iterative review cycles and decision making.
This is where a lean approach to management and processes comes into play. At companies where agile methods have been successfully adopted, Forrester has found that the application of lean principles — the same thinking that has helped manufacturers change how they work to improve efficiency — has helped solve many cultural and organizational problems tied to software development.
The lean approach emphasizes challenging traditional techniques and replacing them with ones that reduce waste and increase value for the customer. That creates an environment that encourages and supports the use of agile development methods.
Lean principles change how businesses plan and measure projects. The lean method replaces departmentalization with team structures, measuring their progress in more effective ways. It ensures that only the right amount of planning is done, and done at the right time.
Among the businesses Forrester surveyed, the application of lean principles across the company provided the foundation for more effective use of agile methods within the development teams. The resulting organization had the following characteristics:
- Processes were simpler.
- Customer involvement was more natural.
- Organizations were flatter.
Simpler Processes
Agile teams frequently bemoan wasteful steps or having to create documents for the sake of compliance and governance. The friction between traditional software development life cycles and agile methods is particularly evident in the areas of planning, documentation, and progress reporting. By looking at waste throughout the value chain, lean approaches challenge the status quo of traditional software development, letting teams redesign the process in a way that’s defined by the process itself. The resulting life cycle is simpler and more effective because the development team focuses on what the process is trying to achieve instead of mechanisms, artifacts, and process steps set by a development methodology.
The lean approach also teaches organizations to plan according to demand, rather than forecasts. It encourages a planning cycle that’s much closer to delivery: You plan a small chunk of work, deliver it, and then plan the next chunk. It closes the gap between “demand” (what business people say they need) and “supply” (the software the IT group ends up building). That, in turn, improves value by delivering just what’s needed rather than everything that could possibly be thrown into a software requirements document. It reduces feature bloat and waste.
Another way of looking at the lean approach is that it combines a high-level overview plan that provides focus for the work with complementary micro-level plans. The key benefits of this approach are that plans improve over time as the data from the micro plans are fed back into the macro plan. And because the macro plan is simple, it’s easily changed. The lean approach teaches organizations to plan in response to demand; agile methods provide clear techniques to deliver on that approach.
Customer Involvement
Traditional development approaches have often led to an almost confrontational relationship with the business partner or customer, and this has driven the development of complex requirement processes.
Agile methods turn this contractual orientation into a collaborative one that requires close working relationships with the customer. But just saying “we must collaborate” doesn’t ensure collaboration, which points to another advantage of the lean approach. Lean organizations have cross-functional teams that include both the customer and IT, and the partnership is fostered by the ongoing development of requirements and regular team meetings. Because being lean focuses on reducing the number of hand-offs that occur in the development process, documents often are replaced with collaborative technology such as wikis, which are great for visualizing and collaborating, or kanbans, which require some action and thus trigger communication flow through the team. Both drive discussion.
A lean organization provides an ideal environment for agile customer collaboration practices; it encourages clients’ input that defines their needs, frequent project reviews, and daily meetings. Lean doesn’t let a business unit and IT go off in separate groups.
Flatter Organizations
To make that iterative, collaborative process work, agile development needs a flat development organization, with teams capable of managing themselves and allocating work as necessary, without the overhead of complex demand and resource management. Traditional organizations are hierarchical, and power lies within specialized departments. The number of people managed determines a person’s status, and a person’s position and rewards are individual — rather than team — based.
With a lean approach, organizations put the emphasis on teams. Entire teams are measured in terms of value delivered to the business, with interim deliverables measured only to track status of the project and provide early warnings of potential problems. Communities of practice support these cross-functional teams, providing mentoring, training, and technical skills. The result is a fully functional, but flatter organization.
The Value of Lean and Agile Approaches
Organizations that have taken advantage of both agile and lean approaches together have seen benefits such as reduced costs, improved time to market, and higher quality. But perhaps the most surprising result is in the areas of innovation and staff motivation. These organizations are highly motivating places to work, with team members feeling that they’re contributing to the company’s bottom line. Internal and external innovation is the natural result of empowering such teams because the processes they’re following are streamlined and effective, and the products they’re producing are being improved.
Over the past 30 years, development organizations have striven to improve their processes, tools, and measurements. The application of agile methods changed the way those improvements were sought, discarding complexity and focusing on the customer. But agile methods will always create friction and stress if there aren’t simultaneous changes to the company-wide environment. Lean approaches retool the entire enterprise, providing the organizational changes needed to let agile development methods function without the friction.
This article first appeared in Dr. Dobb’s Journal in April 2009
Don’t forget to leave your comments below
Dave West is a Senior Analyst with Forrester Research and works with Application Development and Program Management professionals. He is a leading expert on software development process, agile development, l ean thinking, process improvement, project management, and requirements management. His interest in software modeling and application development led to his authoring the book Head First Object-Oriented Analysis and Design. He also covers the area of product development and its relationship with software systems development. Dave earned a B.A. in computing and business from Huddersfield University in the UK and an M.Sc. in computer science from Southbank University (UK).