By David Greenlees
In a recent testing coaching session with Anne-Marie Charrett we touched on the above three subjects. Obviously an hour and a half is no where near long enough to go into any great level of detail, so Anne-Marie left me with a challenge to write about oracles in software testing. I will be doing that, but will also be including the preceding actions of observation and inference. Why? They are part of a logical flow which I have begun to knowingly use in my testing. I highlight knowingly because I think I always have done this, but have not applied critical thinking to it at the time.
During the coaching session we played an online game which involved many observations, inferences, and oracles. At the time, this simply seemed to be a game which I was playing until Anne-Marie reminded me to slow down my thinking and focus on each action separately. Where I initially fell over was jumping directly to my inference without thinking about my observation. To cut a long story short, once I was critically thinking about each of these actions, I was able to win the game! I may have won it eventually anyway, but it would have taken much longer.
A definition for each from the World Wide Web:
- Observation – Detailed examination of phenomena prior to analysis, diagnosis, or interpretation.
- Inference – The process of arriving at some conclusion that, though it is not logically derivable from the assumed premises, possesses some degree of probability relative to the premises.
- Oracle (in software testing) – Heuristic (useful, fast, inexpensive, and fallible) principles or mechanisms by which we recognize problems.
How I define them:
- Observation – What you see.
- Inference – What your observation tells you.
- Oracle (in software testing) – How you know if your inference is correct.
After the coaching session I decided I would try this approach out again. Now, what was a simple application that I had not used before… Google Calendar!
Fig. 1 – Entry screen.
So what was my initial observation? The calendar view defaults to ‘Week’ view. Then I thought about it a bit more, is that truly an observation or an inference? The observation is that the calendar was presented in ‘Week’ view, and from that I had already inferred that it was a default. Maybe the correct observation is that my calendar had defaulted to ‘Week’ view. I cannot be 100% certain that this would be the case for all users. I could make the assumption, but we all know how dangerous they can be in testing.
So some more observations (the numbers for each will remain consistent throughout the article):
- The time zone was displayed;
- There was a red line on the calendar indicating the time;
- The Google search bar was displayed at the top of the page;
- The Google logo was displayed at the top left of the page; &
- A series of buttons were displayed above the calendar view on the right hand side.
Obviously there are many more, however I’ll move on for the purpose of this article.
So I then decided to work backwards and thinking about what inferences I would have made if I had looked at the application how I used to test:
- The time zone was displayed as my current time zone;
- The red line on the calendar matched the time zone and the correct time;
- I can search the web from the Google search bar;
- I can go to google.com via the Google logo; &
- I can change the calendar view via the buttons above it.
Now it was time to do some testing and see if my inferences would have been correct had I done it the old non-critical thinking way:
- Incorrect; &
Hey, 2 out of 4 isn’t so bad right? In this case maybe not. However what if I was observing some sort of mission critical product where lives were at stake? Then 2 out of 4 could be very bad. This is assuming that I had not gone on to test the product, which I would have of course!
So why were the 2 incorrect?
- This particular search bar was for the calendar application only. When you search for something, it only searched in your calendar for results; &
- The Google logo was not actually a link, it was simply an image of the logo.
I think they were fairly strong inferences to make! Why though? Why was I so confident in making those inferences? That’s right Anne-Marie, I haven’t forgotten about you… oracles!
One reference which I like to use when looking at oracles is the HICCUPS(F) mnemonic from James Bach (I believe the (F) came from Michael Bolton):
- History: The present version of the system is consistent with past versions of itself.
- Image: The system is consistent with an image that the organization wants to project.
- Comparable Products: The system is consistent with comparable systems.
- Claims: The system is consistent with what important people say it’s supposed to be.
- Users’ Expectations: The system is consistent with what users want.
- Product: Each element of the system is consistent with comparable elements in the same system.
- Purpose: The system is consistent with its purposes, both explicit and implicit.
- Statutes: The system is consistent with applicable laws.
That’s the HICCUPPS part. What’s with the (F)? “F” stands for “Familiar problems”:
Familiarity: The system is not consistent with the pattern of any familiar problem.
So back to the question of why I made those inferences that turned out to be incorrect:
- For the search function the Product heuristic would be relevant. For many other Google applications the search bar can be used to search the web, not just that particular application; &
- For the Google logo the Users’ Expectations and Comparable Products heuristics would be relevant. My expectation as a user would be that this image would in fact be a link back to google.com, and when observing the behaviour of a comparable product this is further confirmed.
Now what were my oracles? How did I recognise that these were in fact potential problems?
- Other Google applications (namely News and Gmail) were my oracles for the search function; &
- Bing.com (namely the Bing logo) was my oracle for the Google logo function (or lack thereof).
A simple, yet extremely valuable exercise. These principles and practices can be applied to your everyday testing with ease. It’s important to slow down and apply your critical thinking skills throughout. Making inferences and assumptions can save a bucket load of time, but if they are incorrect for whatever reason you may find you’re in a bucket load of…………..
Having said that, can we actually test all assumptions? Even if we could, would it be worth it? I would argue that you would very quickly end up testing assumptions that have a minimal impact, and the type of quality information you’re obtaining by doing so would not be value for money. A method I’ve found valuable in these situations is to apply a negative likelihood to the assumption.
So let’s take one of my inferences:
- I can search the web from the Google search bar.
This directly relates to an assumption, in fact it is an assumption. Now in reality it would be fairly quick to test this assumption, but for the purpose of my point let’s pretend that it would take up to one day to test. I’ll use an average contractor rate of $800 per day. So to test this assumption we’re looking at approx. $800 (that doesn’t take into account project time duration loss, etc).
If there was a high likelihood that users are going to make the same assumption, and therefore try to search the web from this search bar, spending $800 may be a wise move. However, is it the type of function that users will complain about if they cannot search the web? Are they more likely to just say, “Oh, you can only search the calendar from this search box.” And simply move on effectively making it a non-issue?
Let’s say we don’t test it to save $800 because we believe it to be such a trivial matter. Then, when it’s released the users go mad with frustration and call/email Google with complaints and upgrade suggestions. This could cost a lot more than $800 when considering the time it would take for staff to handle these requests, and potentially update the function to now include a web search.
It’s important to recognise your assumptions, and even more importantly to determine which of those assumptions you will spend time and money on testing. You need to consider the context of your environment and how that relates to the product you’re testing.
So now here is a challenge for you. Over the coming weeks while you’re testing, I’d like you to think about observations, inferences, and oracles. By think, I mean really think. Stop and reflect on each of these tasks one by one (similar to the process I have used above). Once you have done that, I’ve love to hear how you went and what difference you think it made. Email me at xtremedmgATgmailDOTcom
I’ll gather the responses and include them in a post on my Blog (with your permission of course).
Your Blog Here
Join us for the first course in our series. It presents the mission of testing, oracle problem, measurement problem, impossibility of complete testing.
Online Education for Testing Practitioners
Learn more »
Theme The theme for CAST 2014 is “The Art and Science of Testing.” In some contexts and by some people Software Testing is considered to be more of a science and by others more of an art. To some, it’s an outgrowth of computer science, mathematics and physics. To others, it’s an expression of soft [...]
Skilled testers look at testing in a different way every time they need to execute a test. In this video Paul Holland discusses the difference between a skilled tester and a non-skilled tester. Are you a skilled tester?
By: Peter Walen (Twitter/Blog) Test Leadership Camp an open space meeting organized by the LeadershipSIG of AST, held the Thursday after CAST. One of the topics discussed was a fundamental concept, “What is Leadership?” The discussion was wide-ranging and covered a variety of topics. The primary points under discussion involved a variety of topics around [...]
- PNSQC 2014 - PNSQC on CAST 2014 (Aug 11-13)
- Star CAST 2014 - PNSQC on CAST 2014 (Aug 11-13)
- CAST 2014: Theme & CFP on Call for Participation [OPEN]
- Elias Nogueira – Bug ou Feature? Pagamento de Fatura no Itau Internet Bank on “Emerging Topics” Track
- Taking a Break – What taking time off from pretty much everything can do for/to you/me | Sowing Seeds on CAST 2014 (Aug 11-13)
- December 2013 (9)
- November 2013 (37)
- October 2013 (60)
- September 2013 (64)
- August 2013 (95)
- July 2013 (74)
- June 2013 (57)
- May 2013 (52)
- April 2013 (63)
- March 2013 (70)
- February 2013 (77)
- January 2013 (87)
- December 2012 (46)
- November 2012 (51)
- October 2012 (62)
- September 2012 (53)
- August 2012 (42)
- July 2012 (54)
- June 2012 (33)
- May 2012 (44)
- April 2012 (62)
- March 2012 (95)
- February 2012 (60)
- January 2012 (80)
- December 2011 (98)
- November 2011 (83)
- October 2011 (137)
- September 2011 (44)
- August 2011 (61)
- July 2011 (61)
- June 2011 (66)
- May 2011 (60)
- April 2011 (62)
- March 2011 (41)
- February 2011 (36)
- January 2011 (70)
- December 2010 (77)
- November 2010 (56)
- October 2010 (46)
- September 2010 (26)
- August 2010 (23)
- July 2010 (18)
- June 2010 (27)
- May 2010 (40)
- April 2010 (31)
- March 2010 (30)
- February 2010 (3)
- January 2010 (11)
- December 2009 (17)
- November 2009 (19)
- October 2009 (11)
- September 2009 (19)
- August 2009 (16)
- July 2009 (11)
- June 2009 (28)
- May 2009 (5)
- April 2009 (11)
- March 2009 (5)
- January 2009 (1)
- December 2008 (4)
- November 2008 (2)
- October 2008 (1)
- August 2008 (4)
- July 2008 (3)
- May 2008 (1)
- A big thanks to @mheusser @erikld and @simonsaysnomore for organizing #WHOSE2013 about 13 hours ago ReplyRetweetFavorite
- #WHOSE2013 Day 2. A big thanks to @HylandSoftware for providing the meeting space for this event. about 14 hours ago ReplyRetweetFavorite
- #WHOSE2013 starts tomorrow, we are looking forward to what they come up with. http://t.co/ZQCL95jLeg 07:19:16 PM December 04, 2013 ReplyRetweetFavorite
- Need inspiration for your #CAST2014 proposal? Check out past conference talks. http://t.co/7fbHqlHfTr 04:50:47 PM December 04, 2013 ReplyRetweetFavorite
- First CFP submission for #CAST2014 has been received. Are you next? http://t.co/x7lPH5QskV 02:56:48 PM December 04, 2013 ReplyRetweetFavorite