Simple Solutions to Complex Testing Problems

When dealing with complex problems, complex solutions aren’t always required.

When I was running a Testing Consultancy around ten years ago, one of my biggest clients was a major utilities provider that wanted to cut testing costs for new system releases by 20% without sacrificing quality.

They believed that this was going to be complicated and time-consuming, so the initial piece of work was a 20-day assignment to come up with an approach to solving the problem, followed by the implementation of my recommendations.

When I started, I asked for all the information that they had on the testing they did: the number of testing phases; the costs per phase; the number of test cases executed per phase; and the number of defects found per phase.

They told me that had the data I requested, but not in a consolidated report that would be easy to read, but they definitely had the raw data so I could create the report for myself.

I decided to spend the first morning creating the report to get a better understanding of their situation.

They had several phases of testing: System Test, Integration Test, User Acceptance Test, Regression Test I and Regression Test II.

For the purpose of this post, the difference between the different types of testing doesn’t really matter, as we’re looking at the costs and effectiveness of each phase.

After about 2 hours of analysing the data, I created a chart, which I shared with the management team, to ensure I had a good understanding.

When I presented the Report Summary, the management team said that the Information I had presented was accurate and in line with their understanding of the costs and effectiveness of each phase.

They did tell me that they were surprised that I had called the meeting on the first day, just to go through the chart.

They felt it could have waited a week or two until I had some idea of how I was going to resolve their complex problem.

Reducing testing effort by 20% was no easy task, especially if you wanted to ensure that there was no real drop in quality.

The problem was though they were looking at things from the wrong angle, they assumed that their Complex Problem needed a Complex Solution.

Whereas I was looking to see if there was a simple solution first.

I told them that I think I’d found a Simple Solution, one that was very easy to implement, fully saves 20%, small impact on overall Quality, and probably at zero cost to implement.

They were stunned when I told them, with looks of disbelief all over their faces.

So I showed a second chart I had prepared.

The second chart was the same as the first chart but with just the Regression Test I highlighted.

I said why don’t you stop doing Regression Test I, it costs 20% of your budget, and it’s only finding 2% of your defects, so it’s not very effective.

Also, your Regression Test II includes 5 million customer accounts, so there is a good chance that the defects missed in Regression Test I would be found here.

If we take this approach we can reduce your costs by 20% and still have probably 99% of the defects found, so not a significant drop in quality.

They discussed the idea for about 20 minutes and then finally agreed with my proposal.

So now we had a Simple Solution, for what had been perceived as a Complex Problem.

Unfortunately, for me, they also ended the assignment too, which means I just go paid for one day of work, rather than the full 20 days.

Seems they had found another simple way to reduce cost without impacting quality!

If you want to learn more about creating highly engaged teams or being a better leader click the link to view our course.