Release problems: zero

A Scrum team has been developing the Saxo Bank community website TradingFloor.com for three years. Along the way, the team has gone from a slow release process to zero faults and as many releases as operations desire.

One of the major challenges in Scrum is to figure out what to set up for testing.
Development of the TradingFloor.com site was no exception. Scrum Master Brian Fischer explains:

’Back in the day, the development team handed over the final deliverable for testing, but once the test team had done their work, it took too long to get the results back to the development team and correct the faults.’

This was partly connected to the less than agile mindset of classical testers. In a Scrum process, it is not unusual for requirements to change somewhat along the way. Therewith, the product changes before the testing gets started, but classic testers have to have very precise specifications to do their job. So, Fischer and Head of IT Web Fehrend needed a new kind of tester.

’Not just a tester, but one who coordinates things so that the prerequisites for quality are in order, so that everything is in place. One who goes in and talks to the developer about how the task will be accomplished, so that he knows how it should be tested,’ says Fehrend.

He used to think that testers should be ’fresh eyes’ with no preconceptions, whose only job was to perform the test. But that did not work. Experience has shown that it is very important that the tester is able to write code so that they understand how the product was created.

’Getting to that point was something of an epiphany,’ says Fehrend.

Four and a half hours of testing in 15 minutes

Today, Fehrend’s testers are the people who know the most about the applications. This new approach to testing means that the 10-person team needs only one tester – a team of that size would normally have three or four testers. However, the developers were also made responsible for ensuring that the quality they deliver is at production-ready levels.

‘The developers have to test their features themselves and make sure everything is in order. They cannot expect someone after them to verify that their work is good enough,’ says Fischer.

At the moment, the team is about to reach a major milestone. In that connection, the test coordinator sets up a few test sessions where he asks the entire development team to thoroughly test a specific feature in 15 minutes.

‘We have been able to substantially raise quality this way,’ says Fischer.
All together, this equates to four and a half hours of testing done and dusted in 15 minutes. Developers who are unfamiliar with the feature push all the wrong buttons. The results are written into a document so the test coordinator can evaluate the outcome. Fehrend adds:

‘It is highly efficient and has had much greater impact than we expected. It might be a bit Scrum-geekish, but getting this aspect to work is really interesting because the whole testing process was a hard nut to crack.’

Far fewer faults

For a long time, releases were as big a challenge as the test problem.

‘We are on the third version of TradingFloor now. “Version 2” is still a put-down among many of our developers and me too, because the fault risk was high every time we released,’ says Fehrend.

To handle the problems, Fehrend and Fischer started implementing a set of guidelines called Continuous Delivery. It is not part of Scrum, but the guidelines are a good fit with the agile method. The object was to put all troubleshooting on the developer’s desk. In a little bigger set-up, this can be done using automated tests and automated gateways. When the developer checks something in it does not move out to production until it has been put through the troubleshooting mill. Fehrend explains:

‘The release is the ultimate feedback: Did it work or didn’t it? If you can move the experience as close to the developer as possible, the faults are fixed at once.
We have asked for as much feedback as possible and that has enhanced quality.’

The combination of Scrum and a general improvement of agile processes has sharply reduced the number of faults in the TradingFloor project.

‘Critical faults have been reduced to virtually zero. Amazingly few things are faulty live. The release problems have been eliminated. And that from having been an area where, in bad periods, we spent 30-40% of our time correcting faults,’ says Fehrend.

Faster releases are an additional bonus of the new delivery model. It used to typically take two to four weeks from the time the code was finished until the new functionality could go live. Nowadays, the team are ready to release about once a week – or more often, if operations ask for it.