Archive for January, 2012

Lessons Learned from a successful catch-up

January 16, 2012 Leave a comment

I previously noted some lessons learned from issues encountered during releases on my last project. To redress the balance I am now going to make some observations of an aspect that went well – a managed data catch-up following implementation.

The project included setting up a service to collect and collate customer usage data, before passing it on to my client’s billing system. The usage data was to be supplied by each customer on a daily basis via an agreed interface, but before go-live each customer had technical or data collection problems which meant that none was ready in time. Rather than postpone the start altogether, we agreed to let the customers to collect their data and store locally for a couple of months and submit it late in a controlled way. This managed catch-up was planned and executed very well.

Contingencies and scenarios. Once the possibility of a delayed start arose, we held a planning session to identify all the combinations of factors that had to be considered. For example, what if each of three separate systems involved was not available in time? What if it spanned more than one financial period? How long a delay could we absorb before catching up became unfeasible?

Clear steps. Once the likely scenario became clear, we defined a set of very clear steps to follow, including anticipated error scenarios. This was important as processing depended on everything running in the correct sequence. We verified the steps on a test environment – the chance to rehearse is something I am keen on to improve the chances of success. We minimised the risk to the production system by starting with a small volume of and gradually building up the volumes after that once we were confident everything was running smoothly.

Active Monitoring. As well as running the manual steps, every day we actively checked that automatic processes had completed successfully by querying the data rather than relying on error logging (in case of bugs).

Daily progress call. There were five separate groups of people involved – the business, development project, customers’ IT technical teams and two support teams (one for each main application). Each day we held a conference call with all parties to confirm progress, resolve any issues and to agree next steps.

Dedicated co-ordinator. One of the team took responsibility to oversee the process and make sure everything happened as planned. He was also the single point of contact for any issues, leading to clear communication. While activities were being carried out he made sure all interested parties were kept in the loop via email.

Management status updates. Senior management were kept informed of significant progress via email, usually following the conference call, and when to expect the next update. The project was high profile and this proactivity reassured management that everything was under control.

Overall the catchup worked very well – the sole error was when the regular person in one team was on holiday and his replacement didn’t follow the steps correctly. Otherwise, everything ran smoothly, all issues encountered had been foreseen and we caught up ahead of plan.