Lately I’ve been doing a lot of thinking about what differentiates traditional testing from the context driven approach. Of course there are many aspects to this.
But a major difference is the mindset of the tester. Many call it a paradigm shift. I agree. When it comes down to it, I realise a major part of my own mindset change is related to my approach to Risk!
Companies employ testers because they want to avoid the risk of something going wrong with the software once it goes live. Traditionally, testing is very much a Risk Averse activity… “Let’s mitigate any potential risks by trying to test everything”
I believe this mindset has given rise to the ‘run the full regression pack at all times’ practice (sometimes at the expense of thorough testing of the actual change!).
This has served us testers ok for the last 15 years but now more than ever, we found ourselves in a climate where companies know quality is important (yet another UK bank recently suffered a complete outage of customer facing systems leaving thousands without access to their own money) but that quality must be achieved ever faster and cheaper!
So the pressure often falls on the testers to test in a shorter time period. But we’re expected to provide the same level of confidence as if we had been allowed to test for as long as our risk-averse natures would have liked to. We kid ourselves that investing in large automation packs will assist. But if we are being asked to test faster and cheaper, investing in a year long automation program might not be the ‘solution of the week’!
So lately I’ve been challenging myself to really evaluate the test coverage and design plans for each project I work with. Surely we can design a more efficient test approach.
Become Risk Aware
In my experience the best way to do this is to understand and articulate the inherent risks within the product /change and ensure we focus on providing information to the project on the status of those risks.
To do that I needed to change my Risk Averse mindset. I used to want to test as much as I could in an effort to mitigate any potential risk anywhere.
I now focus my time on becoming Risk Aware! I spend time understanding the technology, the architecture, the code structure, the team, the business … And then brainstorm with the team to identify where the risks lie.
Focus on Product Risk
We traditionally do this risk analysis for the project risks. I’ve seen test plans full of risks like ‘if Dev don’t deliver on time, our testing will be affected’ or ‘If x component is not in my environment my results won’t be truly representative of production’ etc.
But now I am doing this for the Product Risk. I define a product risk as “something that could go wrong with the system when in production… A risk that I can mitigate by performing a test”
We brainstorm to identify as many product risks as we can think of (some might call this Poka Yoke). We then attempt to prioritise these risks on the basis… “If I could only test one thing, which would it be. What would be the 2nd…” etc
We then communicate our Product Risk list to the project team to see if they can identify any more. We also check they are in agreement with our priority assessment.
Once agreed, instead of designing our tests with the intention of achieving functional coverage, we design our tests to mitigate the risks (i.e. determine whether or not the risk has become an issue (bug))
Of course we are also know and track what system coverage we are achieving whilst testing for the risks.
Risk Based Test Design
I call this approach Risk Based Test Design (RBTD). This differs from Risk Based Testing (RBT) in the amount of upfront effort required. With RBT we tend to write down all functional tests, then use a risk assessment to determine which of these tests to run. Often resulting in time spent writing tests for low priority items which rarely get executed.
With RBTD, we can determine what risks we will test before we design the tests, this prevents effort spent on tests which may never tell us anything important about the risks to production success.
This is a hard mindset shift. It was very hard for some of our risk averse testers to let go of their safety blanket of a ‘full’ regression pack! But they very quickly learnt that by doing thorough impact and risk analysis of a change, they were accurately able to predict where issues could occur.
The teams that have switched to this Risk Aware approach have suffered from no decrease in quality in production. But they have benefitted from being able to test changes faster, delivering a greater rate of change to production. Not only that, but the test report speaks directly to the business…. “You were worried that risk X could occur. We have proven it did not occur during testing”
… With a bi-product that the testers are now recognised as true experts in the system.