Home > Work > Release Management Lessons Learned – some problems and what we did about them

Release Management Lessons Learned – some problems and what we did about them

Last year, I was QA and Release Manager for the project I was working on at the time. Now that I’ve moved on, I’m making some notes on things to remember for another time.

I am always interested to read about other people’s experiences, what worked for them etc, so in the same spirit I am offering up some observations of issues that we found while forming releases, and what we did about it. We aren’t always very good at recording lessons learned, either formally as part of our job or even privately, which is a shame as it can be a very worthwhile exercise.

During the year our internal processes eventually became quite tight. I have mentioned previously that I created simple checklists to ensure we carried out all the steps needed to prepare a release. Despite this, new issues still arose nearly every time. Those described below occurred during one of the later production releases and are typical of the sort of thing we found.

Development process issues

We had two significant issues with our code that surfaced immediately following a release into production, which – embarrassingly – required an immediate Hotfix:

1. Missed branch – lack of formal QA process

We reintroduced a critical defect that had been fixed in Production by a Hotfix to the previous release. It turned out that the Hotfix code branch hadn’t been incorporated back into the main code base. The development team leader was aware of it and thought it was done, but in fact had confused it with a different Hotfix.

The immediate failing in this case was with the Hotfix release process. What should have happened when the code branch took place (a couple of months before) was to raise a defect to merge it back into the main code base. Then, even if the development team forgot about it, the QA checks would have spotted it.

On this occasion, the ultimate cause was lack of familiarity with the process; I was on annual leave and my role was performed by someone else. At the time the original Hotfix was applied, we didn’t have a standard checklist for Hotfixes, and a defect wasn’t raised because my stand-in didn’t think to do it. By the time this issue surfaced, we had already closed this gap; the Hotfix checklist included a specific check to prevent this very issue.

2. UI usability issue – failure of testing process

A bug fixed at this release introduced serious usability issues, which really should have been spotted during testing. It went all the way through the test process including User Acceptance Testing without detection.

The problem here was that the original defect description was incorrect, and subsequent attempts at clarification served only to muddy the waters. It was unclear exactly what was being tested, making it harder for a review to identify that the test was inadequate.

You could argue this might be symptomatic of a more widespread process failure, within both my project team and the client’s; fortunately this was an isolated incident. Following this the test team started ensuring that all defects contained a clear functional statement, and were not afraid to challenge them if they didn’t.

At heart, this reflected a very common cause of project delivery problems – poorly specified requirements.

Our existing processes should have prevented both of these issues, but we didn’t quite have the right checks in place to ensure they were being followed.

Production release process issue

During the release database build the client DBA didn’t carry out instructions as documented and set up some tables incorrectly. He hadn’t been involved in the Production Support rehearsal of the implementation; the action for the next release was to ensure the same personnel are included in the dry run.

A second point – instructions are notoriously difficult to write well, and can easily be misunderstood by a new reader even if everyone else thinks they are crystal-clear. If the dry run involves the same team who will actually carry it out, any confusion can be clarified (and if necessary, the instructions improved) when it doesn’t matter.

Self-improving process

The end result was some embarrassment, it didn’t do much harm to the team’s reputation in the long term but we can always do without it.

This was why I added a step towards the end of the release checklist to “write up notes of release including any issues encountered for later dissemination”. By reminding myself to do it, we were able to prevent silly mistakes from happening a second time.

Even when you think you have everything under control, there is always scope for improvement.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: