The Cambridge Tester Meetup last night was a discussion on testing standards. Not on the specific question of ISO 29119 (for which see Huib Schoots’ excellent resource) but more generally on the possibility of there being a standard at all. It was structured along the lines of Lean Coffee with thoughts and questions being thrown down on post-its and then grouped together for brief discussion.
I’ve recorded the content of the stickies here with just a little post-hoc editing to remove some duplication or occasionally disambiguate. The titles were just handles to attach stickies to once we had a few in an area and I haven’t tried to rationalise them or rearrange the content.
- Testing is a creative process so can’t be standardised.
- Testing doesn’t fit into a standard format, so how can there be a standard for it? (Do we mean “good testing” whatever that is?)
- New tools, technology might not fit into a standard.
- Standardisation destroys great design ideas by encouraging/forcing overly broad application.
- Can a general standard really fit specific project constraints?
- Each tester is different.
- A standard limits new thinking.
- Could a standard be simply “Do the best you can in the time you have”?
- Who do certifications serve anyway? What do they want from them?
- As litigation becomes more prevalent who is protected by a standard? Customers, producers, users?
- With a standard, companies can be “trusted” (QA-approved sticker).
- People outside of test are usually very opinionated. Do standards help or hinder?
- End users care because of the possible added costs.
- A testing standard would provide false reassurance for companies.
- How does an agile team fit in the standard?
- Too much documentation? Standards may cause the need for more documentation to show compliance.
- Standard language for communicating test ideas.
- Divide the testing community – good or bad?
- Respond to feedback and criticism.
How Much? or Alternatives
- Do we need an alternative at all?
- Where are the standards for science, consultancy, product management, development?
- Use as much or as little of a standards as needed?
- Could a standard be subjective?
- Standards for products, or the process of creating products?
- What else do we need or want instead?
- Could a standard cover the minimum at least?
- A standard should be flexible to adapt to project constraints.
- Can a single standard fit different products? (Angry Birds vs nuclear reactor).
- Uniformisation of some testing (bring up the baseline).
- There are already some government standards.
- Infinite space of testing. Can a standard capture that?
- Can some aspects of testing be covered by standards? If so, which?
Can’t we Just Explore?
- Scientists do. Why can’t we? (But what about mandated science?)
- Approaches in methodologies used set out in a common understood format could help consistency.
Fear of Being Assessed?
- Are testers just scared of being evaluated or taking responsibility?
- I’m too shy.
- Could it open up law suites, blame and other consequences?
- Should you insure yourself or your company against any not conforming to the standard?
- Anything unstructured used as an addition to, rather than part of, the primary approach. Stops people hiding?
Show Me the Money?
- What is the motivation of those seeking to create certification? (Rent-seekers?)
- It’s just to make money for ISO companies.
- Adds organisation to a “messy” activity.
Certify Testers Not Testing
- Can you differentiate certifications for testers from certifications for pieces of work? (c.f. Kaner)
- Can you say “product tester by a tester certified XYZ”?
- How would recruitment distinguish between testers and checkers?
- An independent body to audit the testing/tester on real project work? (Who audits the auditor?)
- Qualification vs certification vs standardisation.
Standards in Other Industries
- Learn more about standards in other industries and how they dealt with their first standard.
- Standards in e.g. car safety are on the result of the work not the methodology?
- Universities and schools start teaching testing. Should they teach about the standards?
- Standards to help produce evidence of testing not just test plans, which are usually fiction.
- “Informed” standards (courses, talks etc), “in-house” standards?
- Are objections to certification objections to theoretical risks but in practice it’s possible to have something good enough?
- Would companies without testers need a testing standard?
- Development standards to be closely linked to testing standards.
- Easy to find jobs abroad (if there were standardisation).
- A standard would be good as a product.
- Would a standard really impact our day-to-day job?
- Is the standard simply a reason to justify testing?
- Is the idea of a standard predicated on an outdated idea of testing?
As you can see there was no shortage of ground to cover but, with only a couple of hours, plenty was necessarily shallow or not dug into at all.
To pull out a handful of points that I found particularly interesting: we were not shy about asking questions and we were prepared to aim them at ourselves; we bumped into the distinction between certifying product, tester and testing multiple times; we didn’t really explore what we meant by standards, certification and qualification and what the differences between them might be; while the discussion was entered into with an open mind (which was the remit) there were sometimes implicit assumptions about what a standard must entail (inflexibility; lots of documentation etc) which were mostly negative and where positives were proposed they tended to be viewed more as possibilities.