In this case, the company was responsible for a vital interface between a financial services provider and its corporate clients. The company needed to move the servers that managed thousands of transactions a day and still ensure they provided 24x7x365 coverage to their clients. They planned, they studied, they consulted, they configured, they tested and tested again. They thought they were ready and so flipped the switch. And then the fun began.
Thanks to S.B. for the details on this case.
This company managed millions of transactions daily across their various services. One of the services provided a bridge between their financial services clients and a remote exchange. That service was hosted on three parallel servers located across the continent and managed by a local firm on their behalf. The service provided an attractive revenue stream and so the company decided to invest in upgrades to the hardware, software and communications services to provide faster throughput, greater stability and security. However, when they contacted the local firm to plan the upgrades, they discovered the local firm was no longer interested in managing the service. So, what to do?
The company decided to move the service to their data center and host it locally. They had the facilities. They had the capacities. They had the skills for the new target environment. The biggest challenge would be to acquire the expertise to work with the existing configuration and manage the transition. So they launched the project.
The project would be handled in two phases. Phase I would move all equipment and functionality to the company’s data center and offer the same levels of service as provided by the remote facility. Phase II would upgrade the servers to company standards, move the operating system version to current levels, convert to the database technology used internally and implement a disaster recovery capability. There were also a number of functional enhancements planned to address client needs.
Phase I was to be completed in seven months at a cost of $720,000. Phase II would be sized and planned on the successful completion of Phase I.
The sponsor, the business director of the service, worked with the CIO to line up the key decision makers including the VP of Computer Services, the VP of Software Development and a project manager with suitable knowledge and skills, with an initial focus on Phase I. They also contracted with a local technology support organization to manage the move. These four individuals plus the assigned manager from the contractor made up the steering committee that would oversee the project.
With the steering committee in place, the project manager got to work. He put together a project charter and had it approved by the members of the steering committee. He then worked with the contractor and the firm that currently managed the service to consider alternatives for the move. There were three options that made the cut for further consideration:
- Replicate the current configuration with three new, identically configured servers in the company’s data center. Once the switchover was complete, the old servers would be decommissioned.
- Move one of the existing servers to the data center to handle the initial switchover and ship and configure the remaining two servers after the initial switchover was complete.
- Move one of the existing servers to the data center and add a new, identically configured server to handle the initial switchover. Ship and configure one of the remaining two servers after the initial switchover was complete and redeploy the remaining server.
The PM put together an assessment matrix to manage the evaluation of the alternatives. Factors considered included capital costs, one-time and ongoing expenses, resource and skill availability, conversion and operating risks and the impact on schedule, quality and test plans.
As a result of the analysis, the PM and contractor both felt option 2 was the preferred approach. It was by far the lowest cost in terms of capital, one-time and ongoing expense. It was the simplest approach and therefore lower risk on some dimensions although it would not offer bullet proof reliability during the switch over period.
With the analysis complete, the PM presented his findings and recommendation to the steering committee. It was a very contentious affair. The VP of Computer Services was adamant that Option 1 was the only viable approach. He pointed out that the performance penalties contained in their contract with the exchange would wipe out any possible savings and result in considerable incremental costs if there were any significant issues. The sponsor and the VP Software Development liked the recommended approach for the lower cost and apparent simplicity. As the key stakeholders were unable to agree, the matter was escalated to the CIO. After considerable heated debate, the CIO sided with the recommended approach – Option 2.
The PM got to work. He developed a detailed work schedule and expense plan. He drafted a risk plan. He prepared a master test plan strategy and a quality plan. All were reviewed and approved by the steering committee members. Work progressed ahead of schedule and well below budget until they were ready to do the switchover. And then the fun began!
The initial cutover at midnight, almost six months after the start of the project, failed. There was a communication problem. The service had to be switched back to the original site. Outage – 2 hours. The next attempt three days later also failed. There was a date problem. Back to the original site. Outage – 3 hours. This pattern reoccurred over the next three weeks, with six more failed attempts and outages totalling more than 16 hours. After a number of particularly nasty conversations with the exchange, his CEO and his clients, the CIO finally pulled the plug on the project.
After a bitter post mortem, the company decided to relaunch the project with Option 1 as recommended by the exchange and supported by his clients. The project was finally delivered successfully, ten months late and, including exchange penalties, almost three times the original budget.
How a Great Leader Could Have Helped
This project got off on the wrong foot and never recovered. It was almost entirely internally focused. The stakeholder group included only managers within the company whose organizations would be affected and the hired contractor. They forgot about the exchange the service supported. They didn’t consider the needs of their clients or give them a voice at the table. Had they included the exchange in their stakeholder group, they would have discovered that using a single, standalone server to support the feed from the company’s clients to the exchange was unacceptable. Had they included their clients in the discussion, they would have heard the same message. In addition, the exchange could have provided a very explicit road map they had used with other intermediaries in similar circumstances to manage the transition successfully.
Getting the right decision-makers actively engaged in a change initiative is the paramount project success factor. In this case, the company spent too little time identifying, engaging with and leveraging external influencers. While the devil was certainly in the details for this project, salvation lay with those external stakeholders. So, put these points on your checklist of things to do in future endeavours so you too can be a Great Leader. And remember, use Project Pre-Check’s three building blocks covering stakeholder, process and decision area best practices right up front so you don’t overlook these key success factors.
Finally, if you have a project experience, either good or bad, past or present, that you’d like to have examined through the Project Pre-Check lens and published in this blog, don’t be shy! Send me the details and we’ll chat. I’ll write it up and, when you’re happy with the results, Project Times will post it so others can learn from your insights. Thanks
Don't forget to leave your comments below.