Testing approach notes

March 27, 2012 Leave a comment

Recently I delivered a project where testing was severely impacted due to the difficulty of setting up test environments in time.

The client has a detailed test policy setting out all the stages of testing that must be undertaken and when. It follows a classic ‘V’ Model, in practice we have four main stages of testing each carried out in its own environment

  • Unit test by developer
  • System Test to prove functionality
  • FAT/UAT with realistic volumes of data
  • Operational Acceptance and Pre-Production testing to prove build and performance in an environment that is supposed to match production

The full set of environments for our main development phase was not provided in time. Eventually the situation stabilised with the FAT/UAT one missing so we had to use the Pre-Production environment for that purpose, as it was the only environment large enough to test with realistic volumes of data copied over from the Live system.

As a result we were never able to do the final set of testing which led to a number of issues on release particularly:

  • the Production environment required some configuration settings that were not needed in any of the test environments
  • stricter permissions on Production meant we couldn’t run some of the migration scripts without modification
  • we couldn’t rehearse the release as it was intended to be run, so instructions had minor errors leading to confusion e.g. where we’d merged test releases into a single production release

One of the key lessons learned from earlier development phases was that having a chance to do a dry run of the release was invaluable in identifying missing, inaccurate or ambiguous steps in the instructions. It was a retrograde step and led to complaints about “Failed Releases” from other client managers in roles like “Service Delivery”, all demanding absolute perfection. Consequently both the client PM and I had to expend too much effort trying to set expectations and dealing with the inevitable political fallout, which was largely out of proportion as in fact all the application code ran perfectly because we had been able to test it adequately.

Having said that, setting high expectations about the ease of implementing a release is a good thing. In previous roles I have been the gatekeeper – it is hard enough keeping production systems running uninterruptedly just due to the vagaries of hardware and operating system software. Past experience tells everyone that introducing application software upgrades almost never runs smoothly. As well as the application in question, there are impacts on shared infrastructure or dependent systems to consider. It is not surprising that the default position of the people running the infrastructure is to deny any change and accept it only when absolutely necessary. This is especially true when there has been a past history of failure.

The benefits of this approach are clear – to minimise the risk of any system downtime, either due to a botched release or subsequent poor performance bringing the system (or others) to a halt.

On the other hand, setting a very high bar for every system change in all systems is expensive incurring additional costs by

  • providing infrastructure for additional test environments and managing it – which gets ever more expensive as the production specification is reproduced more closely.
  • carrying out an extra round of testing, setting up the environment and the data, and testing the installation process and carrying out a regression test as a minimum. This environment might only be used for one week every three months.
  • lengthening the project duration to fit this extra step in.

Is it worth it? The impact of system outage needs to be assessed.

Some applications demand high availability, where any downtime has an immediate and measurable cost to the business e.g. banking systems or online retailers. There is also the cost to a company’s reputation if a service is unavailable or perceived as unreliable.

For less critical systems, the case is less clear cut; it may actually be more cost-effective to dispense with the final round of testing and then deal with any subsequent issues as they arise. In our case, the system is business-critical but in fact t needs to be fully available only once per week – on any other day the business can cope with a half day outage or more once the main processing overnight has completed. Knowing this, we always schedule releases for the day after to give us time to deal with any issues before the next week-end – by then issues remaining are likely to be irritants rather than critical. It is much cheaper for a support analyst to spend half a day with a developer to fix any odd issues with the release as they arise. From a commercial perspective this is even more so if the developer’s time is covered by a Fixed Price contract and paid for anyway.

All this needs to be considered at the start of the project when the test strategy is being agreed; in my experience this isn’t done often enough.

Categories: Work Tags: ,

East of Bloomsbury, Part 4

February 23, 2012 Leave a comment

This is the fourth part of a walking tour following a Pevsner Perambulation in part of Bloomsbury, London; the previous part is here. See the introduction for fuller details; page references are to the Pevsner Buildings of England series volume London 4: North.

St George's Gardens

St George's Gardens

This part of the tour starts on the corner of Cartwright Gardens and Hastings Street and covers a section where some of the original early 19the Century development has survived amid later apartment blocks. We turn left into Hastings Street and immediately right into Sandwich Street. In the centre on the left hand side is the Lutheran Church and hostel.

Sandwich Street, Lutheran Hostel

Sandwich Street, Lutheran Hostel

The church is in the basement; if it were not for the cross outside the door it would not be apparent that this is a place of worship.

German Lutheran Church, Sandwich Street

German Lutheran Church, Sandwich Street


Sandwich Street

Sandwich Street

An original 4-storey terrace (nos 1 to 9) is immediately beyond the church on the same side.


Leigh Street

Leigh Street

We continue to the end to Leigh Street. Facing us is another significant terrace (nos 1 to 19), mostly with shops on the ground floor.

We turn left into Leigh Street, where just beyond the corner of Thanet Street and contrasting with the original houses opposite is Medway Court of 1949-55, which is better seen from this side rather than from Judd St where Pevsner mentions it.

Medway Court

Medway Court


Thanet Street

Thanet Street

We now enter Thanet Street, where we find another original terrace (nos 8 to 17), this time (unusually) with only two storeys.


87-103 Judd Street

87-103 Judd Street

At the end we turn right (into Hastings Street again) and then immediately right again into Judd Street. Around here, although not especially noticed by Pevsner, are several large red brick Edwardian mansion blocks. Another long original terrace (nos 87 to 103) is on the right hand side. Number 95 is one of a handful in this area with an original shop front.

95 Judd St

95 Judd St


Medical Centre, Hunter Street

Medical Centre, Hunter Street

We continue on past Medway House as Judd Street becomes Hunter Street, heading back towards the Brunswick Centre. On the corner with Handel Street is the Health Centre. This is best viewed from the steps of the Brunswick Centre, as are the late Georgian houses at 3-4 Hunter Street on the same side.

3-4 Hunter Street

3-4 Hunter Street


Handel Street

Handel Street

We cross over into Handel Street where nos 4-7 are on the right hand side, sandwiched between Edwardian mansion blocks.


A short deviation now from the tour as given to take in St George’s Gardens (see p. 263), which we enter through the gateway at the end of Handel Street.

St George's Gardens

St George's Gardens

This is one of several similar small gardens in this part of London, laid out in what were originally overflow cemeteries for neighbouring parish churches, an early attempt to solve the problem of lack of space inside overcrowded churchyards. A line of stone slabs along the centre marks the boundary between the former burial grounds for the parishes of St George, Bloomsbury and St George the Martyr, Holborn. At the time it was created (1713) it lay outside the built-up area. Although unpopular at first, it soon filled up and eventually became gardens in 1882. The gardens are landscaped around the large remaining 18th Century chest tombs and obelisk.

St George's Gardens Lodge and Chapel

St George's Gardens Lodge and Chapel

Just on the right inside the gate as we enter are the early 19th Century former mortuary chapel and the Lodge beside it.

Immediately in front of us as we enter is the statue of Euterpe removed from the Apollo Inn, Tottenham Court Road when it was demolished in 1961.

Euterpe, St George's Gardens

Euterpe

The earliest notable memorial is to Robert Nelson of 1715, who was one of the promoters of the cemetery and who by being buried here himself made it more acceptable. It is easily identified by the urn on top, on the right hand side.

Nelson Memorial

Robert Nelson Memorial

We pass out through the exit at the far left hand corner into Regent Square, where the tour continues.

Lessons Learned from a successful catch-up

January 16, 2012 Leave a comment

I previously noted some lessons learned from issues encountered during releases on my last project. To redress the balance I am now going to make some observations of an aspect that went well – a managed data catch-up following implementation.

The project included setting up a service to collect and collate customer usage data, before passing it on to my client’s billing system. The usage data was to be supplied by each customer on a daily basis via an agreed interface, but before go-live each customer had technical or data collection problems which meant that none was ready in time. Rather than postpone the start altogether, we agreed to let the customers to collect their data and store locally for a couple of months and submit it late in a controlled way. This managed catch-up was planned and executed very well.

Contingencies and scenarios. Once the possibility of a delayed start arose, we held a planning session to identify all the combinations of factors that had to be considered. For example, what if each of three separate systems involved was not available in time? What if it spanned more than one financial period? How long a delay could we absorb before catching up became unfeasible?

Clear steps. Once the likely scenario became clear, we defined a set of very clear steps to follow, including anticipated error scenarios. This was important as processing depended on everything running in the correct sequence. We verified the steps on a test environment – the chance to rehearse is something I am keen on to improve the chances of success. We minimised the risk to the production system by starting with a small volume of and gradually building up the volumes after that once we were confident everything was running smoothly.

Active Monitoring. As well as running the manual steps, every day we actively checked that automatic processes had completed successfully by querying the data rather than relying on error logging (in case of bugs).

Daily progress call. There were five separate groups of people involved – the business, development project, customers’ IT technical teams and two support teams (one for each main application). Each day we held a conference call with all parties to confirm progress, resolve any issues and to agree next steps.

Dedicated co-ordinator. One of the team took responsibility to oversee the process and make sure everything happened as planned. He was also the single point of contact for any issues, leading to clear communication. While activities were being carried out he made sure all interested parties were kept in the loop via email.

Management status updates. Senior management were kept informed of significant progress via email, usually following the conference call, and when to expect the next update. The project was high profile and this proactivity reassured management that everything was under control.

Overall the catchup worked very well – the sole error was when the regular person in one team was on holiday and his replacement didn’t follow the steps correctly. Otherwise, everything ran smoothly, all issues encountered had been foreseen and we caught up ahead of plan.

East of Bloomsbury Part 3: Around Cartwright Gardens

December 1, 2011 2 comments

This is the third part of a walking tour following a Pevsner Perambulation in part of Bloomsbury, London; the previous part is here. See the introduction for fuller details; page references are to the Pevsner Buildings of England series volume London 4: North.

We start at the south end of Cartwright Gardens, which forms a crescent on the left with gardens in the centre, but first we look at right hand (straight) side, which is filled by three uninspiring University of London Halls of Residence (see pp. 281), collectively known as the Garden Halls.

Commonwealth Hall

Commonwealth Hall, Cartwright Gardens, London WC1

Commonwealth Hall of 1960-3 (planned 1947) is the first block on the right on the corner of Leigh St.

Hughes Parry Hall

Hughes Parry Hall, Cartwright Gardens

Hughes Parry Hall of 1967-9 is furthest on the corner of Sandwich Street. Between the two and not noticed by Pevsner is Canterbury Hall, of 1937 according to its website, which is no less worthy of notice than it neighbours and has some interesting period detail at least around the entrance.

Canterbury Hall

Canterbury Hall, Cartwright Gardens


John Cartwright statue

John Cartwright statue

Opposite in centre of the east (straight) side is a statue of John Cartwright, after whom the Crescent is named. The gardens are dominated by tennis courts and are not normally open.


The main crescent is formed of two sets of substantial original (c1820) terraces. In Trollope‘s day, this was known as Burton Crescent and he used it as the setting for the boarding house where Johnny Eames lodged while in London, in his Barset Chronicle The Small House at Allington. The boarding houses of Trollope’s day have now become hotels in the south quadrant, and the bulk of the north quadrant, which was refurbished during 2008, looks like it is University accommodation.

Cartwright Gardens, northern quadrant

Cartwright Gardens, northern quadrant


4-7 Burton Place

4-7 Burton Place

In the centre between the two parts of the crescent is Burton Place, which we enter. Nos 4-7 are on the right (North side) and were built as four houses disguised as one.


Burton Street, rear of BMA building

Burton Street, rear of BMA building

Burton Place leads into Burton Street. Immediately in front and stretching northwards is the rear of British Medical Association building (1911-25, see p 265-6) of which the prominent part is Lutyens’s’ great hall. The rest is more recent and plainer.

Original terraces are to the left, which is a cul-de-sac, so there is no need to go in that direction.

Burton Street, London WC1

Burton Street


We turn right along Burton Street along the length of the BMA building, then left at the end. Almost immediately in front are Woburn Walk and Duke’s Road.

Woburn Walk

Woburn Walk


Woburn Walk is to the left with the original (1822) bay-fronted shops on both sides, Duke’s Road is to the right with similar buildings on one side only (the one facing us). They form quite a picturesque group and can occasionally be seen as a backdrop in period TV dramas or films.

Duke's Rd

Duke's Rd


Dukes's Rd, The Place

Duke's Rd, The Place

Opposite in Duke’s Road is the former Drill Hall (“The Place”) of 1888-9. This is now used by a Dance School, more modern buildings stretching out of sight behind it. It is used as a café and retains its original terracotta embellishments revealing its original identity for the Middlesex (Artists’) Rifle Volunteers.

Duke's Rd, The Place, terracotta

Terracotta decoration on The Place, Dukes's Rd


Flaxman Terrace, lodge

Flaxman Terrace, lodge

We retrace our steps back to Flaxman Terrace. The right hand side consists of an early (1907-8) public housing development by St Pancras Borough Council. Immediately on the corner is the caretaker’s lodge, and behind it on the right are the flats of 1908. Note the iron railings in front which include the St Pancras coat of arms.

Flaxman Terrace

Flaxman Terrace


Hamilton House

Hamilton House, Mabledon Place

We continue to the end of Flaxman Terrace to Mabledon Place. The Headquarters of the National Union of Teachers, Hamilton House occupies the block opposite and to the right, between Bidborough Street and Hastings Street. This is one of many current or former Trade Union Headquarters buildings, which have congregated up and down the Euston Road because of the good rail links to the rest of the country.

Hamilton House, doorway

Hamilton House, doorway

At the right hand corner of Hamilton House we find ourselves at the North end of Cartwright Gardens once again and we turn left into Hastings Street where the next section of the tour continues.

Lessons learned from a recent implementation

November 17, 2011 3 comments

For much of the past year I have been very busy as the development and implementation phases of my project have been in full swing. A quieter period following delivery has given me a chance to reflect how things went before the next phase which has now started up and will keep us all busy through the coming winter.

The project was overwhelmingly successful as we implemented two new applications and a series of significant enhancements to an existing large billing system, all on time. Unfortunately some of the gloss was taken off by a series of problems with each of the major billing system release implementations which bears looking at again to see what we can improve next time.

The problems we encountered were varied and were often quite peripheral to the system’s core functionality – in practice we had very few bugs. One of the main sources of grief throughout the project was the timely provision of test environments – a long story, the short version of which is that the outsourced infrastructure providers couldn’t provide anything in time and we had to test as best we could with what we had. Ultimately the root cause of implementation problems usually stemmed from this; generally something hadn’t quite been fully tested because the lack of a test environment. We tried to compensate by add extra manual checks, but inevitably some things slipped through. This was all understood by the client Project Manager and the decision was taken to go ahead knowing this (correctly – we had to meet a regulatory deadline).

Each of these events created a lot of excitement among the many client managers in Change Management and Service Delivery who as stewards are naturally suspicious of any change to their lovely shiny production systems. They demand perfection so when we failed to provide it, even though they knew the true situation before we went live, they all made a big fuss. The time spent dealing with the politics was disproportionate, given the actual impact.

I wrote earlier about the similarities between Conducting and Project Management – here is another one. A common joke in musical circles is that as long as you start a performance well and end well nobody will notice the mistakes in between, and this is just as true with an IT project. Each release was viewed as a “failure” even though by any dispassionate evaluation, it was emphatically not.

Also in every case, each time my team has been blamed whether it is technically our fault or not. The contract clearly spelled out a dependency on the client to provide adequate environments. Most of the testing that was not carried out was the client’s responsibility, not ours, and we always received formal client signoff. Despite that the finger of blame has been pointed at us, not elsewhere within the client’s organisation

Summary of the lessons of our experience:

  • An outsourced infrastructure management hinders progress (and I am quite sanguine about whether my employer would have been any different)
  • Visual checks, however careful, are no substitute for rehearsals
  • As the supplier, my project team sustains reputational damage when things go wrong whatever the contract says
  • I should have paid more attention to managing internal client stakeholders, especially those with no particular interest in the overall success of the project – even if it’s really the client PM’s responsibility

There are some positives too; for another time, maybe.

East of Bloomsbury Part 2: Tavistock Place

October 17, 2011 1 comment

This is the second part of a walking tour following a Pevsner Perambulation in part of Bloomsbury, London; the first part is here. See the introduction for fuller details; page references are to the Pevsner Buildings of England series volume London 4: North.

Tavistock Place N side

Tavistock Place North side looking West

We start in Herbrand Street, where we find the striking white art deco building now used as offices by the marketing firm McCann Erickson. I like the curve of the ramp between floors (betraying its original use as a garage for Daimler) and there are some other nice details surrounding windows and doorways.

McCann Erickson office, former garage

Herbrand St, McCann Erickson office


We retrace our steps and cross back over Coram St to the northern continuation of Herbrand St where there are two contrasting sets of public housing dating from a century ago, one on either side.

Herbrand St former LCC flats

Herbrand St former LCC flats

On the left (west) side is a former London County Council (LCC) development of three blocks of flats. They are rather plain red brick blocks of five storeys without any embellishment. It looks as if they may have been built with balconies that were later filled in.


Opposite is the more substantial Peabody estate, built around a courtyard.

Peabody Estate, Herbrand St

Peabody Estate, Herbrand St

Its striped brick appearance is archetypal of Peabody housing of this period. It didn’t take much lunchtime wandering around in central London before I found I could correctly identify Peabody flats from a distance. It is interesting to note, a hundred years later, that (as here) the Peabody flats often look both more solid and better maintained than their former LCC contemporaries.


We continue to the end of the road, turn right into Tavistock Place and turn right again into Kenton St. Around here, there are quite a few red brick Edwardian mansion blocks, the one Pevsner has picked out (occupying most of the right hand side) fits the description, having five storeys, but is quite plain and unexceptional even compared to some of the other blocks in neighbouring streets.

Aberdeen House, Kenton St

Aberdeen House

Aberdeen House is an example – it has clearly been truncated (possibly by wartime bombing) to make room for the Brunswick Centre.

I think the converted workshop next door to it (not mentioned, possibly refurbished since 1998) is of more interest, retaining its original sign and hoist, and is typical of the type of the small workshops that once were common in this area.

Kenton Street

Kenton Street old workshop


We retrace our steps and return to Tavistock Place and turn left to the far (Woburn Place) end. Tavistock Place is quite busy, used especially by taxis cutting through between Tottenham Court Road and Gray’s Inn Road.

Mary Ward House, Tavistock Place

Mary Ward House, Tavistock Place


Mary Ward House is the rather odd-looking hall on the right hand side at the far end. It is not mentioned in the perambulation (but see p.267-8) as is 9 Tavistock Place next to it, partly hidden behind a high brick wall.


2-14 Tavistock Place

2-14 Tavistock Place


Opposite are 2-14 Tavistock Place, modern facsimiles in the style of the original houses in the vicinity, including arched lights above the doors and sash windows. The sharp and new brickwork (in yellow brick rather than red) gives it away as being more than just an expensive restoration.


18-46 Tavistock Place

18-46 Tavistock Place


Continuing eastwards, immediately beyond Herbrand St, 18-48 Tavistock Place are original and provide a nice contrast, being very similar apart from the stuccoed ground floors.

We continue along Tavistock Place as far as Marchmont St and turn left into Cartwright Gardens where the next section of the tour continues.

Link updated 01/12/2011.

Conducting and Project Management

October 5, 2011 2 comments

Some more thoughts on similarities between conductors (recently, for me, this means choral) and IT Project Managers.

Strategy. The conductor shapes the whole piece as a performance by setting style and expression, and has a clear idea how he wants it to sound. (It is striking how performances of the same piece can be very different). In the same way, the PM sets out the strategy for the project and holds in mind a vision for how it is to be done.

Rehearsals / training. In many ways, the whole choral rehearsal process is a parallel to the repeated iterations of build-test-refine that any system undergoes before it is finally released. Rehearsing a choir is not just about teaching the notes, but also about how to sing together as a unit. As well as coaching individuals, a project manager ensures that the team knows how to work together – what the internal team processes are, who is expert in what, and so on.

Detail. While holding the whole piece together and the choir in time by keeping the beat steady, the conductor also brings in different parts at the right times, and indicates changes in tempo or expression. To do this he must have an intimate knowledge of how each individual part fits with the others to make up the whole piece. A PM uses his detailed understanding of the business background and of how all the various planned tasks fit together to decide which are the critical tasks to bring to the fore at different points during the project.

Constantly changing priority. I’m struck how my current choirmaster will sometimes sing along with a part when it falters for a little while until it is secure again. Perhaps at that point he may not pay attention to some of the other parts at all. I once sang in a concert where the conductor never brought my part in at all, because he knew he could safely ignore us. This is very much the way a PM will focus on a hot issue until it is resolved maybe to the exclusion of other things for a short while. And of course the choirmaster can only sing one part at once, just as at any point in time at work we can do only one thing at a time.

Contribution of team members themselves. In an ensemble, listening to each other is critical, to keep together – in time, in tune, and expression. In a project this means discussing issues, being aware of what colleagues are doing and chipping in sometimes when needed. The PM should encourage an environment that enables this to happen.

Finally, both have ultimate authority – there may be some discussion at first but ultimately what you say, goes, and there is nobody else to stop you. Which means you need to get it right!

Categories: Work Tags: