Archive for March, 2012

Testing approach notes

March 27, 2012 Leave a comment

Recently I delivered a project where testing was severely impacted due to the difficulty of setting up test environments in time.

The client has a detailed test policy setting out all the stages of testing that must be undertaken and when. It follows a classic ‘V’ Model, in practice we have four main stages of testing each carried out in its own environment

  • Unit test by developer
  • System Test to prove functionality
  • FAT/UAT with realistic volumes of data
  • Operational Acceptance and Pre-Production testing to prove build and performance in an environment that is supposed to match production

The full set of environments for our main development phase was not provided in time. Eventually the situation stabilised with the FAT/UAT one missing so we had to use the Pre-Production environment for that purpose, as it was the only environment large enough to test with realistic volumes of data copied over from the Live system.

As a result we were never able to do the final set of testing which led to a number of issues on release particularly:

  • the Production environment required some configuration settings that were not needed in any of the test environments
  • stricter permissions on Production meant we couldn’t run some of the migration scripts without modification
  • we couldn’t rehearse the release as it was intended to be run, so instructions had minor errors leading to confusion e.g. where we’d merged test releases into a single production release

One of the key lessons learned from earlier development phases was that having a chance to do a dry run of the release was invaluable in identifying missing, inaccurate or ambiguous steps in the instructions. It was a retrograde step and led to complaints about “Failed Releases” from other client managers in roles like “Service Delivery”, all demanding absolute perfection. Consequently both the client PM and I had to expend too much effort trying to set expectations and dealing with the inevitable political fallout, which was largely out of proportion as in fact all the application code ran perfectly because we had been able to test it adequately.

Having said that, setting high expectations about the ease of implementing a release is a good thing. In previous roles I have been the gatekeeper – it is hard enough keeping production systems running uninterruptedly just due to the vagaries of hardware and operating system software. Past experience tells everyone that introducing application software upgrades almost never runs smoothly. As well as the application in question, there are impacts on shared infrastructure or dependent systems to consider. It is not surprising that the default position of the people running the infrastructure is to deny any change and accept it only when absolutely necessary. This is especially true when there has been a past history of failure.

The benefits of this approach are clear – to minimise the risk of any system downtime, either due to a botched release or subsequent poor performance bringing the system (or others) to a halt.

On the other hand, setting a very high bar for every system change in all systems is expensive incurring additional costs by

  • providing infrastructure for additional test environments and managing it – which gets ever more expensive as the production specification is reproduced more closely.
  • carrying out an extra round of testing, setting up the environment and the data, and testing the installation process and carrying out a regression test as a minimum. This environment might only be used for one week every three months.
  • lengthening the project duration to fit this extra step in.

Is it worth it? The impact of system outage needs to be assessed.

Some applications demand high availability, where any downtime has an immediate and measurable cost to the business e.g. banking systems or online retailers. There is also the cost to a company’s reputation if a service is unavailable or perceived as unreliable.

For less critical systems, the case is less clear cut; it may actually be more cost-effective to dispense with the final round of testing and then deal with any subsequent issues as they arise. In our case, the system is business-critical but in fact t needs to be fully available only once per week – on any other day the business can cope with a half day outage or more once the main processing overnight has completed. Knowing this, we always schedule releases for the day after to give us time to deal with any issues before the next week-end – by then issues remaining are likely to be irritants rather than critical. It is much cheaper for a support analyst to spend half a day with a developer to fix any odd issues with the release as they arise. From a commercial perspective this is even more so if the developer’s time is covered by a Fixed Price contract and paid for anyway.

All this needs to be considered at the start of the project when the test strategy is being agreed; in my experience this isn’t done often enough.

Categories: Work Tags: ,