Several times in recent months I’ve found myself being asked how many test cases there are in automated suites at Linguamatics. On each occasion I have had to admit that I don’t know and, in fact, I’m not particularly motivated to calculate that particular metric.
Depending on how the conversations went, I’ve asked in return what my questioner thinks we can understand about the quality, coverage, relevance, value and so on of testing by knowing the number of cases. (And let’s leave vocabulary aside for now.)
Are you a counter? Try this scenario: you wrote sum, a program that adds two numbers, and have asked me to test it…
Sure! I can test that for you. To do it, I’ll write a test case generator. Here it is:
for i in `seq 1 $1`
s=$(($i + $i))
echo "r=\`./sum $i $i\`; if [ \$r -eq $s ]; then echo \"OK\"; else echo \"NO\"; fi"
A somewhat rudimentary framework? Absolutely, but it’ll do for this exercise. Let’s have it make three test cases:
$ sh checker.sh 3 > run.sh
$ more run.sh
r=`./sum 1 1`; if [ $r -eq 2 ]; then echo "OK"; else echo "NO"; fi
r=`./sum 2 2`; if [ $r -eq 4 ]; then echo "OK"; else echo "NO"; fi
r=`./sum 3 3`; if [ $r -eq 6 ]; then echo "OK"; else echo "NO"; fi
And run them:
$ sh run.sh
Yeah, it passes! #GoDev!
What’s that? You say there’s loads of numbers in the world and three isn’t very big in comparison. Well, pray tell, how many cases do you want? Ten? A hundred? A thousand? A round thousand, yeah!
$ sh checker.sh 1000 > run.sh
$ sh run.sh
Now we’re really sure your program works, right? Right?