Archives

On Testing and Quality Engineering (Rhythm of Testing)

On February 29, 2016, in Syndicated, by Association for Software Testing
0

The other day I read an article on how Quality Engineering was something beyond testing. It struck me that, in the course of reading that article, it struck me that the author had a totally different understanding of those two terms.

Here then, is my response…



On Testing and Quality Engineering

A common view of testing, perhaps what some consider is the “real” or “correct” view, is that testing validates behavior. Tests “pass” or “fail” based on expectations and the point of testing is to confirm those expectations.

The challenge of introducing the concept of “Quality” with this conception of testing brings in other problems. It seems the question of “Quality” is often tied to a “voice of authority.” For some people that “authority” is the near-legendary Jerry Weinberg: “Quality is value to some person.” For others the “authority” is Joseph Juran: “fitness for use.”

How do we know about the software we are working on? What is it that gives us the touch points to be able to measure this?

There are the classic measures used by advocates of testing as validation or pass/fail: 

·         percentage of code coverage;
·         proportion of function coverage;
·         percentage of automated Vs. manual tests;
·         number of test cases run;
·         number of passing test cases;
·         number of failing test cases;
·         number of bugs found or fixed.

For some organizations, these may shed some light on testing or on the perceived progress of testing. But they speak nothing about the software itself or the quality of the software being tested, in-spite of the claims made by some people.

One response, a common one, is that the question of the “quality of the software” is not a concern of “testing,” that it is a concern for “quality engineering.” Thus, testing is independent of the concerns of overall quality.

My view of this? 

Hogwash.
Rubbish. 

 When people ask me what testing is, my working definition is:

Software testing is a systematic evaluation of the behavior of a piece of software,
based on some model.

By using models that are relevant to the project, epic or story, we can select appropriate methods and techniques in place of relying on organizational comfort-zones. If one model we use is “conformance to documented requirements” we exercise the software one way. If we are interested in aspects of performance or load capacity, we’ll exercise the software in another way.

There is no rule limiting a tester to using a single model. Most software projects will need multiple models to be considered in testing. There are some concepts that are important in this working.

What does this mean?

Good testing takes disciplined, thoughtful work. Following precisely the steps that were given is not testing, it is following a script. Testing takes consideration beyond the simple, straightforward path.

As for the idea of “documented requirements,” they serve as information points, possibly starting points for meaningful testing.

Good testing requires communication. Real communication is not documents being emailed back and forth. Communication is bi-directional. It is not a lecture or a monologue. Good testing requires conversation to help make sure all parties are in alignment.

Good testing looks at the reason behind the project, the change that is intended to be seen. Good testing looks to understand the impact within the system to the system itself and to the people using the software.

Good testing looks at these reasons and purposes for the changes and compared them to the team and company purpose and values.  Are they in alignment with the mission, purpose and core values of the organization? Good testing includes a willingness to report variances in these fundamental considerations beyond requirements and code.

Good testing can exercise the design before a single line of code is written. Good testing can help search out implied or undocumented requirements to catch variances before design is finalized.

Good testing can help product owners, designers and developers in demonstrating the impact of changes on people who will be working with the software. Good testing can help build consensus within the team as to the very behavior of the software.

Good testing can navigate between function level testing to broader aspects of testing, by following multiple roles within the application and evaluating what people using or impacted by the change will experience.

Good testing can help bring the voice of the customer, internal and external, to the conversation when nothing or no one else does.

Good testing does not assure anything. Good testing challenges assurances. It investigates possibilities and asks questions about what is discovered.

Good testing challenges assumptions and presumptions. It looks for ways in which those assumptions and presumptions are not valid or are not appropriate in the project being worked on.

Good testing serves the stakeholders of the project by being in service to them.

What some people describe as “quality engineering” is, in my experience, part of good software testing.

 

On Testing and Quality Engineering (Rhythm of Testing)

On February 29, 2016, in Syndicated, by Association for Software Testing
0

The other day I read an article on how Quality Engineering was something beyond testing. It struck me that, in the course of reading that article, it struck me that the author had a totally different understanding of those two terms.

Here then, is my response…



On Testing and Quality Engineering

A common view of testing, perhaps what some consider is the “real” or “correct” view, is that testing validates behavior. Tests “pass” or “fail” based on expectations and the point of testing is to confirm those expectations.

The challenge of introducing the concept of “Quality” with this conception of testing brings in other problems. It seems the question of “Quality” is often tied to a “voice of authority.” For some people that “authority” is the near-legendary Jerry Weinberg: “Quality is value to some person.” For others the “authority” is Joseph Juran: “fitness for use.”

How do we know about the software we are working on? What is it that gives us the touch points to be able to measure this?

There are the classic measures used by advocates of testing as validation or pass/fail: 

·         percentage of code coverage;
·         proportion of function coverage;
·         percentage of automated Vs. manual tests;
·         number of test cases run;
·         number of passing test cases;
·         number of failing test cases;
·         number of bugs found or fixed.

For some organizations, these may shed some light on testing or on the perceived progress of testing. But they speak nothing about the software itself or the quality of the software being tested, in-spite of the claims made by some people.

One response, a common one, is that the question of the “quality of the software” is not a concern of “testing,” that it is a concern for “quality engineering.” Thus, testing is independent of the concerns of overall quality.

My view of this? 

Hogwash.
Rubbish. 

 When people ask me what testing is, my working definition is:

Software testing is a systematic evaluation of the behavior of a piece of software,
based on some model.

By using models that are relevant to the project, epic or story, we can select appropriate methods and techniques in place of relying on organizational comfort-zones. If one model we use is “conformance to documented requirements” we exercise the software one way. If we are interested in aspects of performance or load capacity, we’ll exercise the software in another way.

There is no rule limiting a tester to using a single model. Most software projects will need multiple models to be considered in testing. There are some concepts that are important in this working.

What does this mean?

Good testing takes disciplined, thoughtful work. Following precisely the steps that were given is not testing, it is following a script. Testing takes consideration beyond the simple, straightforward path.

As for the idea of “documented requirements,” they serve as information points, possibly starting points for meaningful testing.

Good testing requires communication. Real communication is not documents being emailed back and forth. Communication is bi-directional. It is not a lecture or a monologue. Good testing requires conversation to help make sure all parties are in alignment.

Good testing looks at the reason behind the project, the change that is intended to be seen. Good testing looks to understand the impact within the system to the system itself and to the people using the software.

Good testing looks at these reasons and purposes for the changes and compared them to the team and company purpose and values.  Are they in alignment with the mission, purpose and core values of the organization? Good testing includes a willingness to report variances in these fundamental considerations beyond requirements and code.

Good testing can exercise the design before a single line of code is written. Good testing can help search out implied or undocumented requirements to catch variances before design is finalized.

Good testing can help product owners, designers and developers in demonstrating the impact of changes on people who will be working with the software. Good testing can help build consensus within the team as to the very behavior of the software.

Good testing can navigate between function level testing to broader aspects of testing, by following multiple roles within the application and evaluating what people using or impacted by the change will experience.

Good testing can help bring the voice of the customer, internal and external, to the conversation when nothing or no one else does.

Good testing does not assure anything. Good testing challenges assurances. It investigates possibilities and asks questions about what is discovered.

Good testing challenges assumptions and presumptions. It looks for ways in which those assumptions and presumptions are not valid or are not appropriate in the project being worked on.

Good testing serves the stakeholders of the project by being in service to them.

What some people describe as “quality engineering” is, in my experience, part of good software testing.

 

Book review – Threat Modeling (אשרי אדם מפחד תמיד Happy is the man who always fears)

On February 29, 2016, in Syndicated, by Association for Software Testing
0

This post is quite long, for sure it came out longer than I intended. The English version is still below, but you might need to scroll down a bit.

לפני כמה חודשים הגיע הזמן לעדכן את מסמך “מודל האיומים” (Threat modeling) של התוכנה שלנו. וכן, באנגלית זה לגמרי נשמע יותר טוב. בשל צירוף נסיבות אקראי למדי, יצא שאני האדם הכי מתאים בצוות לביצוע המשימה (למעט מנהל צוות הפיתוח, אני היחיד בצוות שהשתתף בתהליך בעבר, ובניגוד אליו אני יכולתי למצוא קצת זמן להקדיש לזה). חדור מוטיבציה ועזוז, ניגשתי למשימה רק כדי לגלות שאין לי מושג ירוק איפה להתחיל – גם עם המסמך אותו אני צריך רק לעדכן מול העיניים. אז התחלתי לחפש מידע – קראתי את הפרק הרביעי בספר הזה שהתגלגל אצלנו במשרד, קראתי הנחיות בדף ויקי פנימי שהיה זמין לי, וישבתי לדבר עם מי שכתב את המסמך המקורי אותו רציתי לעדכן, שאמנם עזב את הצוות, אבל נשאר בחברה בתפקיד אחר. הוא הסביר לי המון על התהליך ועל מה צריך לעשות וכך יכולנו לעדכן את המודל בצורה אפקטיבית. כשהתברר לשנינו שאני גם נהנה מתהליך מידול האיומים, הוא השאיל לי ספר Threat Modeling : Designing for Security שכתב אדם שוסטק (Adam Shostack). 

זה לקח לי קצת זמן, אבל קראתי את הספר מכריכה לכריכה (לא כולל חלק מהנספחים) ובקיצור נמרץ – זה ספר מצויין. חובה לכל מי שרוצה ללמוד קצת על העולם של מידול איומים בפרט ושל אבטחת תוכנה בכלל, והוא גם נראה כמו משהו שיכול להיות שימושי גם למי שיש לו מידה מסויימת של ניסיון. גם אם אבטחת תוכנה אינה מה שאתם עושים, ונדמה לכם שאין למוצר שום קשר לאבטחת תוכנה – אני חושב שכדאי לכם לקרוא את הספר הזה.
אז מה יש בו? הספר מחולק לחמישה חלקים, וכל חלק מכיל כמה פרקים:
  • החלק הראשון , שהיה הכי חשוב לי כאדם חסר ניסיון, הוא “איך מתחילים?” המשפט הראשון בספר (אחרי בערך שלושים וחמישה עמודים של הקדמה שמסבירים מה נרוויח מהספר, למי הוא מיועד, איך הספר בנוי וכיצד להשתמש בו) הוא “כל אחד יכול ללמוד איך למדל איומים, ומעבר לכך – כל אחד צריך.” עם כזה עידוד, איך אפשר שלא להתחיל מייד?
    ובאמת, אנחנו מתחילים מייד. מהר מאוד אנחנו מתחילים לצייר מודל פשוט של התוכנה שלנו ורואים כמה דרכים להוסיף לו מידע מצד אחד, אבל להשאיר את המודל קריא ולא עמוס מאוד. המודל שאנחנו בוחרים בו הוא דיאגרמת זרימת נתונים (Data Flow Diagram) ואחרי שיש לנו דיאגרמה כזו, אנחנו נעזרים בה כדי לשחק במשחק הקלפים שפותח יחד עם הספר וזמין להורדה בחינם (דרך כאן). אפשר, כמובן, לקנות חבילה מסודרת יותר עם קלפים של ממש). אחרי המשחק – יש לנו איומים וצריך להחליט מה לעשות איתם. הספר מזכיר לנו שיש ארבע אפשרויות לניהול איום שמצאנו –
    • אפשר להפעיל מנגנוני פיצוי ובקרה (כלומר,להוסיף הגנות שיעזרו לבטל את האיום הזה, או לצמצם אותו מספיק כדי שנוכל לבחור באחת האפשרויות האחריות). 
    • אפשר למחוק את הקוד הפגיע, או לא לאפשר את התרחיש המסוכן. 
    • אפשר להעביר את האחריות למישהו אחר. למשל – אפשר להשאיר את זיהוי המשתמש למערכת ההפעלה, או שאפשר להודיע ללקוח ש”בזה התוכנה לא מטפלת” (למשל, Mailinator מכריזים על מדיניות הפרטיות שלהם: “אין פרטיות”). כמובן, ההחלטה הזו צריכה להתקבל ברמה העסקית, ויש גבול לעד כמה ניתן להשתמש בה. 
    • אפשר לקבל את הסיכון ולחיות איתו.

    סיימתם עם הצעדים האלה? מצויין. זוהי התורה כולה. שאר הספר הוא רק הרחבה. מכאן אנחנו ממשיכים לדבר על  אסטרטגיות לבצע את תהליך המידול ומתחילים בשאלה שהתחבבה עלי  – מהו מודל האיומים שלך? זו שאלה קצרה שעוזרת להתמקד והתשובה לה יכולה להעביר המון מידע. התשובה לשאלה הזו קצרה גם היא. משהו בסגנון של “תוקף עם מחשב נייד”, או “רשת לא מאובטחת”. כשאנחנו יודעים מה האיום נגדו אנחנו רוצים להתגונן אנחנו יכולים לבצע בחירות שיעזרו לנו להשקיע את המאמץ בהתגוננות מפני גורמים רלוונטיים. כמו שמציין הספר, תשובה נפוצה מאוד לשאלה עשוייה להיות “הא?”, וגם התשובה הזו מספרת לנו לא מעט. בין כל האסטרטגיות השונות בולט במיוחד הדגש שניתן לדיאגרמות זרימת נתונים (יש גם הסבר על איך לכתוב דיאגרמות יעילות), אבל מוצגות גם גישות אחרות, פורמליות יותר ופורמליות פחות.

  • החלק השני מתמקד ב”מציאת איומים” ועוסק בהרחבה במודל STRIDE (שכבר הזכרתי), בעצי התקפה ובמאגרי התקפות\חולשות, כשלכל אחת מהשיטות האלה מוקדש פרק שלם. הפרקים שונים מאוד זה מזה מבחינת איכות החומר הכתוב ומבחינת התועלת שהצלחתי למצוא בהם. כך למשל, בעוד שהפרק על STRIDE מאורגן בצורה מופתית – לכל אחת מהקטגוריות מצורף הסבר קצר על מהות הבעיה, יחד עם טבלה שמציגה מגוון אפשרויות למימוש התקפה – הפרק על עצי התקפה משמעותית חלש יותר ולאחר קריאתו נותרתי בתחושה שאני לא יועד הרבה יותר על השימוש בהם מאשר ידעתי קודם. אולי זה קשור לטענה בסוף הפרק: “מאוד קשה ליצור עצי התקפה”.  במקרה של ספריות ההתקפה, יש פחות צורך לחשוב – שימוש ברשימות כמו Owasp top 10 (רשימה עם עשר המתקפות הכי נפוצות\מזיקות באינטרנט), או בCAPEC לא דורש הרבה מאוד הסברים, והספר מציין, כמו שכדאי לעשות, שלעבוד עם רשימה מפורטת כזו יכול להתברר כלא מעט עבודה.
     לבסוף, כדי להזכיר לנו שהעולם מסובך יותר משנדמה לנו, הפרק האחרון בחלק הזה מתעסק באיומים על פרטיות ובכלים מחשבתיים שיכולים לעזור לנו לשים לב לפגיעה בפרטיות. בעיקר מצא חן בעיני הרעיון של הנוטריקון  LINDDUN
  • החלק השלישי עוסק בניהול האיומים – החל בסוגיות כמו “מתי לבצע פעולות של מידול איומים?” או “כמה זמן להשקיע בזה?”, דרך נקודות שכדאי לשקול בתיעדוף הטיפול ובדרכיםמקובלות לטיפול באיומים נפוצים.
    חמשת הפרקים שבפרק הזה ממוקדים יחסית, ומעניינים במידה כמעט שווה.
    הפרק הזה ארוך למדי, וסקירה מפורטת של כל הפרקים בו תהיה מתישה. לכן, אני רוצה להתעכב על שתי נקודות קטנות.
    הראשונה היא דוגמה לסיבה בגללה אני מוצא את הספר מאוד נגיש ומאוד קריא לציבור הרחב: בפרק השביעי מזכיר המחבר בדיחה מוכרת על שני אנשים (אליס ובוב, כי אנחנו מתעסקים באבטחת תוכנה) שבורחים מדוב, בוב עוצר לנעול את נעלי הריצה שלו, כי הוא לא צריך לרוץ מהר יותר מהדוב, רק מהר יותר מאליס. הספר מסביר שהאנלוגיה הזו לא טובה במקרה שלנו, או, כפי שהוא מנסח זאת: “… לא זו בלבד שיש דובים רבים ביער, אלא שהם מקיימים כנסים בהם הם מדברים על טכניקות שיאפשרו להם לאכול את אליס וגם את בוב לארוחת צהריים”. והנה לכם – תמונה מנטלית שקל לזכור.
    הנקודה השנייה היא הפרק העשירי, בו מצאתי את עצמי מהנהן בראשי שוב ושוב – נושא הפרק הוא “איך לוודא שאיומים מטופלים”, או, בתרגום לעברית, “איך מוודאים שעשיתי עבודה טובה?” זה נשמע דומה לבדיקות תוכנה? זה לא במקרה. בתוך הפרק אפשר למצוא את הציטטה שתמיד נתקלים בה – you can’t test quality in, והכותב מבהיר שבאותו אופן בדיוק אי אפשר לבדוק אבטחה לתוך המוצר. מה שכנראה מצא חן בעיני הכי הרבה היה כשתחושת הבטן שהייתה איתי לאורך קריאת הספר קיבלה אישור כשהכותב טען שבעת מידול איומים, בודקי התוכנה הם “בעלי ברית טבעיים”בין היתר כי הם כבר מתורגלים בלחפש את המקומות בהם דברים עשויים להשתבש.
  • בין הפרקים השונים בחלק הרביעי ניתן למצוא “ספר מתכונים לדרישות”, שאמור לעזור בהתמודדות עם כתיבת דרישות אבטחה (או במצב בו דרישות האבטחה לא כתובות היטב, או בכלל), קצת על איומים בענן ובסביבות web, כמה שאלות לא פשוטות שכדאי להיות מודעים אליהן בזמן שמטפלים בזיהוי משתמשים וניהול זהויות, דיון עמוס למדי על שימושיות, ומעבר זריז על עקרונות בסיס בהצפנה והתקפות נפוצות (ההתקפה החביבה עלי היא קריפטואנליזה בעזרת צינור גומי,  שזו דרך אלגנטית לתאר את השיטה בה תופסים את הג’ינג’י עם המפתח ומפרקים לו את הצורה עד שהוא מספר לנו מה שאנחנו רוצים) ובהקשר הזה אנחנו מקבלים גם תזכורת לגבי פרטיות, הפעם – איך הצפנה יכולה לעזור לנו איתה. 
  • החלק החמישי הוא סיכום, ומתוך שלושת הפרקים שלו – שני פרקים קצת “נמצאים באוויר” כשהם מדברים על “גישות נסיוניות במידול איומים” (כמו למשל, משחק בשם FlipIT), או ברעיונות כלליים כמו עומס קוגניטיבי או  תיאוריית הזרימה. הפרק השלישי ממוקד במטרה קונקרטית מאוד: איך להכניס מידול איומים לארגון שלך? כאן אפשר למצוא עצות פרקטיות על איך ניתן “למכור” את הרעיון להנהלה או למהנדסים מהם יצפו לעשות את העבודה, כולל תשובות להתנגדויות נפוצות. אם טרם השתכנעתם שיש קשר הדוק בין בדיקות תוכנה למידול איומים והתזכורת שיש בפרק הזה לא משכנעת אתכם, אחת ההתנגדויות שמוזכרות כאן היא “אבל אף אחד אף פעם לא יעשה משהו כזה”. 
בסוף הפרק, מצורפים חמישה נספחים. ברור לגמרי שהנספחים האלה הם כלי עבודה – לא כולם יהיו קריאה קלילה, אבל הם כנראה יעזרו עם העבודה עצמה.

  • נספח א’ – תשובות נפוצות לשאלה “מה הוא מודל האיומים שלך?”
  • נספח ב’ – עצי איומים. זו הרחבה של הפרק הרביעי והיא מתישה לקריאה. 
  • נספח ג’ – רשימות תוקפים. כאן מוצג הרעיון הנחמד של שימוש בפרסונות, יחד עם כמה הצעות לרשימות תוקפים. 
  • נספח ד’ – חוקים למשחק הקלפים Elevation of Privelege
  • נספח ה’  – ארבעה מקרים לדוגמה (case studies). המקרים אינם מקרים אמיתיים, אבל הם מעניינים. 

טוב, זה יצא ארוך יותר ממה שהתכוונתי, אבל זה נתן לי תירוץ לרפרף דרך כל הספר שוב, וזה בפני עצמו הצדיק את המאמץ.
אם אתם רוצים, אפשר לקרוא כמה עמודים (=כל הפרק הראשון והשני, עם עמוד מהפרק השלישי)  בחינם בגוגל ספרים. והכי חשוב, במילותיו של הספר – Go Threat model, and make things more secure.

——————————————————-

A while ago, just before I started writing this blog, we got to that point in the year where we should update our threat model (one of the cool things in working in a large company is that there are some mandatory procedures, and for some of them there actually is someone who is responsible of verifying they are done). By what seems to me as a sheer coincidence, I was the most suitable person to lead the threat modeling, as I was the only one who participated in the process last year besides our dev team leader who is awfully occupied (so, me being able to muster some free time trumps his superior knowledge and experience). Full of motivation to dive into this new and shiny field, I started out just to find out that I didn’t have the slightest clue as to where can I begin – even when all I had to do was to update an existing document with the changes we made in the past year. So I starting looking for information and instructions – I read the fourth chapter of this book that was rolling around in the office, read an internal Wiki page that was supposed to help me, somehow, and then I sat down to talk with the guy that has composed the original document I was updating (who, despite having moved to another team, remained at hands reach and was happy to help). His knowledge and expertise helped me tremendously, and I was able to actually start updating our model and even add some improvements to the document (as this document was the product of his self-education, there were some things there that could have been done better, and unlike him – I had help from the first moment I dove into this matter). Anyway, once we both found out that I was enjoying the process, maybe more than I should, he lent me his copy of Adam Shostack’s Threat Modeling:  Designing for Security.
It took me a while to do so, but I read this book front to back (excluding some of the appendices), and in short – that’s a great book. It’s a must-read for anyone who wished to learn a bit about the world of threat modeling, and it is a good source of knowledge for those who wish to learn about software security. On top of that, it seems to my inexperienced eye to be something useful even for those that are familiar with threat modeling. Finally, even if you believe that your product has nothing to do with software security, I think you’ll find reading this book worthwhile. 
So, what’s in it? 
  • The book has five parts, each containing several chapters, the first of which was the most important for me – being completely inexperienced in that field – is labeled “Getting Started”. The first sentence in the book (excluding the thirty odd introductory pages explaining what can the reader expect to gain, who are the intended audience and how to use that book – still it’s the first sentence of the first chapter, so there you have it) is “Everyone can learn to threat model, and what’s more, everyone should“. with such encouragement, how can we not start threat modeling immediately?
    And indeed we do. Very quickly we are drawing a diagram of a software, and discussing a bit how to add information to the diagram without making it unreadable or over detailed. Once we have our diagram (In our case, A Data-Flow Diagram, or a DTD) we go on and start finding threats by playing “Elevation of Privileges” – a card game that was developed along with this book and is freely available to download (you can find it here). Naturally, you can buy a proper deck and save yourself the fuss of printing it. Once we are done playing, the book reminds us that we have four options to deal with a threat we found:
    • Mitigate the threat – Add some control mechanisms and defenses that will make exploiting the weakness you found harder, limit the amount of damage an attack can do via this venue, etc. this is done in order to allow us to choose one of the other options comfortably. 
    • Eliminate the threat – delete the relevant piece of code, block a vulnerable functionality – do something that will make sure that this vulnerability is simply not out there anymore. 
    • Transfer the risk – you can transfer the problem to someone, or something else. For instance, you can decide to leave authentication to the OS, or just notify the user that “The software does not deal with this type of risk (A great example for this is Mailinator, a temporary email service, that have the following privacy policy in their FAQ page – “There is no privacy”). Obviously, such decisions should be business decisions, and you should be careful when deciding to transfer a risk, as you can do that to a limited extent only. 
    • Accept the risk – Sometimes, it’s acceptable to acknowledge a risk and respond with “I’ll take my chances”. This should not be your go-to approach, but if the risk is improbable enough (e.g.: The server drowning if the Netherlands dam breaks and the sea floods everything), or if the impact is low enough in comparison to the cost of fixing it (e.g.: protecting your server physically can cost you a small fortune to maintain something like this) , then just living with the risk might be acceptable. 
    Done with that? Great. That’s all folks, let’s go home. The rest of the book is just some elaboration and expansion of what was done up to this point. The book then continues with a question I learned to like – “what’s your threat model?”. This short question can help learning quite a lot, despite having a short answer as well. Answers can be “a single attacker with a standard laptop”, or “a disgruntled employee”, “cyber-criminals” or “NSA, Mossad, MI5 and every other intelligence agency”. Once you know what is the threat you want to protect yourself against, you can make good choices that will help defending against it. As the book mentions, a common answer to this question is “huh?”, and this answer provides us with a lot of information as well. Specifically, it tells us that we need to start thinking and identifying the threats we are concerned about.  At this point, we get into more details and discuss a bit approaches to threat modeling. The book covers some unstructured & structured approaches to threat modeling (with a bit of focus on Data-Flow-Diagrams, or DFDs, which seem to be the author strong recommendation). While reading this, we encounter some useful terminology and concepts such as assets & stepping stones or trust-boundaries. 
    • The 2nd section deals extensively with the analysis part, or, as the section is named “Finding threats”. It starts by focusing on the STRIDE model (on which I already wrote a bit), and then goes to discuss other methods such as attack trees and attack libraries. Finally, just as a reminder, the section shifts its focus to privacy threats and some tools that may help us noticing those.
      The quality of the different sections vary greatly – while the STRIDE chapter is well organized and very readable, with several examples of how to apply each part of the STRIDE model, the one about attack trees has left me not much better off than where I first began. Perhaps it is simply a proof by example of the claim in the end of the chapter: “It’s very hard to create attack trees”. The chapter on attack libraries is somewhat in the middle – it’s very clear, but a bit boring, since using lists such as OWASP top 10, or CAPEC requires very little explanation. The book reminds us that using attack libraries can be quite a bit of work.
      The chapter about privacy is great – it gave me the feeling that I now have the basic knowledge about some mental tools to address the issue, but also that there is so much more to learn about it by following the hints there (The one i liked best is the LINDDUN acronym). 
    • The third part deals with “Managing and addressing threats”, with all 5 chapters written in about the same quality. The section deals with a lot of subjects, from managing the threat modeling process, to defense strategies to available tools that can be used while threat modeling.
      In this section I want to delve on a short anecdote that demonstrates why I find this book so readable: When dealing with approaches to threats, the author mentions the story about Alice and Bob who were running from a bear, and Bob stopping to put on his running-shoes, justifying that he does not need to run faster than the bear, only faster than Alice. At this point, the author breaks the metaphor by stating that in the software vulnerabilities world, “not only there are multiple bears, but they have conferences in which they discuss techniques for eating both Alice and Bob for lunch”.
      The tenth chapter is the one I found myself nodding the most, as it dealt with the question “How can I make sure my work is complete and that threats are dealt with?”. Does it sound a bit like testing to you? it sure did to me. In this chapter we can find the always repeated truth “you can’t test quality in” (which leads to “and neither can you test security in”). Perhaps the thing that got me most was when my gut feeling matched what I read in the book that stated that testers are “close allies” in threat modeling, as they are already trained to think of “what might go wrong?”
    • The fourth section is dealing with “Threat modeling in technologies and tricky areas”. This section is actually a mixture of various subjects that have very little in common, except being significant enough to have a chapter of their own. Among its chapters we can find “requirements cookbook” which contains possible examples of security requirements for us to write, Web & cloud common threats, some tough questions to consider when dealing with identities and accounts, a content-heavy discussion with tons of references about usability, some basic cryptography concepts and common attacks (the one I liked most was rubber-hose cryptanalysis, which is a fancy term to say “beat the cr** out of someone until they tell you what you want to know”) and we get another reminder about privacy, this time about how can encryption help with privacy. 
    • The fifth and last section is more of a summary. It has three chapters, two of which are somewhat intangible, speaking of ideas such as experimental approaches to threat modeling such a FlipIt, or about stuff like Flow theory and cognitive load. The most concrete chapter in this section is about bringing threat modeling into your organization, with practical advice for “selling” the idea to management or to the engineers who’ll be doing it with you, and how to deal with common objections, and if you haven’t yet become convinced that threat modeling is very much like software testing, and the reminder in this chapter was not enough, one of the  objections mentioned is “No one would ever do that”.
      My impression was that the intangible chapters are meant for experts in threat modeling that are looking for ideas to stimulate their thoughts – For me, most of it went way above my head. 
    By the end of the book there are five appendices, some short, some a bit longer:

    • “Helpful tools” – This appendix contains some answers to “what is your threat model” and some of the common assets a software project might have. 
    • Threat trees – This is an extension of chapter 4 (part of the 2nd section) and is really taxing to read. 
    • Lists of potential attackers. Along with several generic lists, we have also the idea of using attacker personas along with some examples. Also, it’s fun to read. 
    • Rules for the card-game “Elevation of Privileges” that was linked above. 
    • Four “case studies”. They are not real, but are interesting to read. 
    That’s about it, and it definitely came out longer than I expected, but writing this post gave me an opportunity to go over the book again, and this by itself was totally worth it. 
    Also, if you are interested, or think you might be interested, the first sixty-odd pages of the book are free to read on google-books
    And most important, as the book states: Go Threat model, and make things more secure.

    Selecting Platform Configuration Tests (Chris Kenst's Blog)

    On February 29, 2016, in Syndicated, by Association for Software Testing
    0

    I’ve been developing a GUI acceptance test suite to increase the speed of specific types of feedback about our software releases. In addition to my local environment I’ve been using Sauce Labs to extend our platform coverage (mostly by browsers and operating) and to speed up our tests by running more tests in parallel. This […]

    Bug-Free Software? Go For It! (Hiccupps)

    On February 29, 2016, in Syndicated, by Association for Software Testing
    0

    This post is a prettied-up version of the notes for my talk at the second Cambridge Exploratory Workshop on Testing last weekend. The topic for the workshop was When Testing Went Wrong

    Cold fusion is a type of nuclear reaction that, if it were possible, would provide a cheap, clean and safe form of energy. In 1989 two scientists, Fleischmann and Pons, made worldwide headlines when they claimed to have generated cold fusion in a test tube in their lab. Unfortunately, subsequent attempts to replicate their results failed, other scientists started to publicly doubt the experimental methodology, and the evidence presented was eventually debunked.

    Cold fusion is a bit of a scientific joke. Which means that if you are a researcher in that field – now also called Low Energy Nuclear Reactions – you are likely to have a credibility problem before you even start. And, further, that kind of credibility issue will put many off from even starting. Which is a shame because the potential payoff  in this area is high and the cost of it being wrong is relatively low.

    In a fascinating article in Aeon magazine, Huw Price, a Philosophy professor at Cambridge University, writes about how, even if unlikely, cold fusion is not theoretically impossible, and that the apparent orthodox scientific opinion on it is not based on science:

    Cold fusion is tainted, and the taint is contagious … So the subject is stuck in a place that is largely inaccessible to reason – a reputation trap, we might call it.

    This is echoed by Harry Collins in Are We All Scientific Experts Now?:

    There is always enough room to interpret data in more than one way … We need to know motivations as much as we need to know results if we are to understand science.

    Science is not pure. It is not driven only by evidence. Collins observes that, particularly at the cutting edge of research, scientists can easily split into camps. These camps agree on the result, but don’t agree on what it means. When this is the case, when there is room for more than one interpretation, then – since scientists are human – it’s natural for there to be human biases and prejudices at play. And those factors, those frailties, those foibles include things like reputation, preconception and peer pressure.

    You might have seen Bob Marshall blogging and tweeting about whether we really need testers, and using the hashtag #NoTesting? He is provocative:

    So, do we have to test, despite the customer being unkeen to pay for it? Despite it adding little or no value from the customer’s point of view? 

    And he provokes, for example, reactions like this from Albert Gareev

    Recently I’ve been observing some new silly ideas about testing – on how to do as less of it as possible or not do it at all. I don’t bother posting links here – reading those isn’t worth the time.

    To me, there can be value in wondering what Marshall is getting at (which Gareev also does). Engaging with someone with an apparently fundamental opposition to you can be elucidating. A contrary position can make us see our own position in a new light and it’s healthy to do that sometimes.

    There was an interesting (to most testers anyway, I’d hope)  headline out of Princeton earlier this year: Computer scientists launch campaign to guarantee bug-free software. What’s your gut reaction to that? Something like this, perhaps? You can’t get rid of bugs …  and it’s stupid to even think you might be able to!

    But read behind the headline only a little way and you will find that the project is trying to write formal (mathematical logical) specifications for software building blocks, such as a C compiler or an OS, and then chain together such components using consistent specifications.

    Doesn’t a formal spec just shift the specification problem? It still has to be written, right? Perhaps, but a formal language can be reasoned about; proofs can be created for aspects of it; other tools can be brought to bear on it in a way that they cannot with user stories or other (natural language) devices for specification.

    For sure, it’s a non-trivial problem. And perhaps it won’t work. And perhaps it will even prove to have been misguided. And, absolutely, it won’t catch the class of bugs that are do to with the specification being for something other than what the users actually want. But should that mean that we shouldn’t pursue this line? A line that has (relative to all the research being done) low cost, potentially high benefit.

    James Bach might put this kind of effort into the Analytical School. For example:

    The Analytical School way is to limit themselves to laboratory contexts where the numbers apply or trying to change projects to fit the assumptions of the numbers […]  I have a fondness for the Analytical School, but I’m not an academic, so I have to live in a world where I must solve the problems that come to me, rather than the ones I choose.

    He and Cem Kaner, founders of the Context-Driven School of testing, have publicly disagreed here. Kaner says:

    I think it’s a Bad Idea to alienate, ignore, or marginalize people who do hard work on interesting problems.

    Bach speaks to this:

    One of the things that concerns Cem is the polarization of the craft. He doesn’t like it, anymore. I suppose he wants more listening to people who have different views about whether there are best practices or not. To me, that’s unwise.

    And Kaner responds:

     I’ve learned a lot from people who would never associate themselves with context-driven testing.

    And, in fact he actively engages folk outside of the context-driven community, such as with Rex Black, who many would regard as a Factory Schooler. 

    When thought leaders like Bach and Kaner, both of whom contribute so much to the community and craft of testing, say these kinds of things it’s wise to listen. They clearly fall into two different camps on this topic, but they would both, I’m sure, encourage us to think critically about what we are hearing from them, and to take our own view, for ourselves.

    So, to the question that CEWT #2 is posing: when does testing go wrong?  Maybe in ways like this:

    • When we look inwards too much: if we stay in our own bubble we risk lack of exposure to useful information, to things that can help us make connections.
    • When we don’t apply critical thinking: we should strive to understand our sources and the degree of confidence we have in them, and in which areas we think that confidence is justified.
    • When we don’t consider human factors: we should ask ourselves why something is being claimed. 
    • When we create reputation traps: we should be wary of closing off topics for others. Sure, we may legitimately have nothing to learn; but others might.

    Like the scientists mentioned up top, testers are humans, and we have, do, and will continue to make these kinds of mistakes. Testing will always go wrong because it is done by people.

    But that’s also the good news: people have the capacity to observe this happening and attempt to take action to avoid or mitigate it. I want to give myself a chance of spotting approaches that are appropriate to whatever context I find myself in and I think (and perhaps this is my bias) that a sensible way to go about this is to be open to information from anywhere.

    This doesn’t mean that I have to accept everything or even that I shouldn’t be sceptical of everything. Nor that I have to give equal time, effort or respect to everything. It doesn’t mean that I can’t take someone else’s word for something, but I challenge myself to have considered whether that’s sensible this time, for this thing.

    So, if you want to tell me that you’re going to find a way guarantee bug-free software, I say go for it. But when you do, explain what you did and show me the results you got and don’t be surprised if I question them and your motivation.

    Here’s my slides:

    Bug-Free Software? Go For It! (Hiccupps)

    On February 29, 2016, in Syndicated, by Association for Software Testing
    0

    This post is a prettied-up version of the notes for my talk at the second Cambridge Exploratory Workshop on Testing last weekend. The topic for the workshop was When Testing Went Wrong

    Cold fusion is a type of nuclear reaction that, if it were possible, would provide a cheap, clean and safe form of energy. In 1989 two scientists, Fleischmann and Pons, made worldwide headlines when they claimed to have generated cold fusion in a test tube in their lab. Unfortunately, subsequent attempts to replicate their results failed, other scientists started to publicly doubt the experimental methodology, and the evidence presented was eventually debunked.

    Cold fusion is a bit of a scientific joke. Which means that if you are a researcher in that field – now also called Low Energy Nuclear Reactions – you are likely to have a credibility problem before you even start. And, further, that kind of credibility issue will put many off from even starting. Which is a shame because the potential payoff  in this area is high and the cost of it being wrong is relatively low.

    In a fascinating article in Aeon magazine, Huw Price, a Philosophy professor at Cambridge University, writes about how, even if unlikely, cold fusion is not theoretically impossible, and that the apparent orthodox scientific opinion on it is not based on science:

    Cold fusion is tainted, and the taint is contagious … So the subject is stuck in a place that is largely inaccessible to reason – a reputation trap, we might call it.

    This is echoed by Harry Collins in Are We All Scientific Experts Now?:

    There is always enough room to interpret data in more than one way … We need to know motivations as much as we need to know results if we are to understand science.

    Science is not pure. It is not driven only by evidence. Collins observes that, particularly at the cutting edge of research, scientists can easily split into camps. These camps agree on the result, but don’t agree on what it means. When this is the case, when there is room for more than one interpretation, then – since scientists are human – it’s natural for there to be human biases and prejudices at play. And those factors, those frailties, those foibles include things like reputation, preconception and peer pressure.

    You might have seen Bob Marshall blogging and tweeting about whether we really need testers, and using the hashtag #NoTesting? He is provocative:

    So, do we have to test, despite the customer being unkeen to pay for it? Despite it adding little or no value from the customer’s point of view? 

    And he provokes, for example, reactions like this from Albert Gareev

    Recently I’ve been observing some new silly ideas about testing – on how to do as less of it as possible or not do it at all. I don’t bother posting links here – reading those isn’t worth the time.

    To me, there can be value in wondering what Marshall is getting at (which Gareev also does). Engaging with someone with an apparently fundamental opposition to you can be elucidating. A contrary position can make us see our own position in a new light and it’s healthy to do that sometimes.

    There was an interesting (to most testers anyway, I’d hope)  headline out of Princeton earlier this year: Computer scientists launch campaign to guarantee bug-free software. What’s your gut reaction to that? Something like this, perhaps? You can’t get rid of bugs …  and it’s stupid to even think you might be able to!

    But read behind the headline only a little way and you will find that the project is trying to write formal (mathematical logical) specifications for software building blocks, such as a C compiler or an OS, and then chain together such components using consistent specifications.

    Doesn’t a formal spec just shift the specification problem? It still has to be written, right? Perhaps, but a formal language can be reasoned about; proofs can be created for aspects of it; other tools can be brought to bear on it in a way that they cannot with user stories or other (natural language) devices for specification.

    For sure, it’s a non-trivial problem. And perhaps it won’t work. And perhaps it will even prove to have been misguided. And, absolutely, it won’t catch the class of bugs that are do to with the specification being for something other than what the users actually want. But should that mean that we shouldn’t pursue this line? A line that has (relative to all the research being done) low cost, potentially high benefit.

    James Bach might put this kind of effort into the Analytical School. For example:

    The Analytical School way is to limit themselves to laboratory contexts where the numbers apply or trying to change projects to fit the assumptions of the numbers […]  I have a fondness for the Analytical School, but I’m not an academic, so I have to live in a world where I must solve the problems that come to me, rather than the ones I choose.

    He and Cem Kaner, founders of the Context-Driven School of testing, have publicly disagreed here. Kaner says:

    I think it’s a Bad Idea to alienate, ignore, or marginalize people who do hard work on interesting problems.

    Bach speaks to this:

    One of the things that concerns Cem is the polarization of the craft. He doesn’t like it, anymore. I suppose he wants more listening to people who have different views about whether there are best practices or not. To me, that’s unwise.

    And Kaner responds:

     I’ve learned a lot from people who would never associate themselves with context-driven testing.

    And, in fact he actively engages folk outside of the context-driven community, such as with Rex Black, who many would regard as a Factory Schooler. 

    When thought leaders like Bach and Kaner, both of whom contribute so much to the community and craft of testing, say these kinds of things it’s wise to listen. They clearly fall into two different camps on this topic, but they would both, I’m sure, encourage us to think critically about what we are hearing from them, and to take our own view, for ourselves.

    So, to the question that CEWT #2 is posing: when does testing go wrong?  Maybe in ways like this:

    • When we look inwards too much: if we stay in our own bubble we risk lack of exposure to useful information, to things that can help us make connections.
    • When we don’t apply critical thinking: we should strive to understand our sources and the degree of confidence we have in them, and in which areas we think that confidence is justified.
    • When we don’t consider human factors: we should ask ourselves why something is being claimed. 
    • When we create reputation traps: we should be wary of closing off topics for others. Sure, we may legitimately have nothing to learn; but others might.

    Like the scientists mentioned up top, testers are humans, and we have, do, and will continue to make these kinds of mistakes. Testing will always go wrong because it is done by people.

    But that’s also the good news: people have the capacity to observe this happening and attempt to take action to avoid or mitigate it. I want to give myself a chance of spotting approaches that are appropriate to whatever context I find myself in and I think (and perhaps this is my bias) that a sensible way to go about this is to be open to information from anywhere.

    This doesn’t mean that I have to accept everything or even that I shouldn’t be sceptical of everything. Nor that I have to give equal time, effort or respect to everything. It doesn’t mean that I can’t take someone else’s word for something, but I challenge myself to have considered whether that’s sensible this time, for this thing.

    So, if you want to tell me that you’re going to find a way guarantee bug-free software, I say go for it. But when you do, explain what you did and show me the results you got and don’t be surprised if I question them and your motivation.

    Here’s my slides:

    Some kick ass blog posts from last week #8 (Mr.Slavchev())

    On February 29, 2016, in Syndicated, by Association for Software Testing
    0

    Here’s the new portion of kick-ass blog posts from the last week: Great post by Michael Fritzius on continuous integration and how to prioritize our tests into portions, how to run them separately, in parallel and many other cool stuff: Automate Your Automation Interesting set of advises from Simon Knight on how to leave the comfort […]

    The post Some kick ass blog posts from last week #8 appeared first on Mr.Slavchev().

    How to automate check for youtube video subtitle (zagorski software tester)

    On February 27, 2016, in Syndicated, by Association for Software Testing
    0

    TL;DR In this post I will explain how I implemented check that youtube video contains…

    Joking With Jerry Part 1 (Hiccupps)

    On February 27, 2016, in Syndicated, by Association for Software Testing
    0

    This is the first part of the transcript of a conversation about the relationship between joking and testing. The jokers were Jerry Weinberg (JW), Michael Bolton (MB), James Lyndsay (JL), Damian Synadinos (DS), and me (JT).

    See also Intro, Part 2 and Part 3.

    –00–

    JL: I wrote one joke for this. It turned up in my head and it may not be original, but it’s mine. And the process of arrival is an interesting thing for me because you don’t necessarily design a joke, you ready yourself for the arrival of the joke. And that certainly parallels some of the things I do when I’m testing … where you are prepared and you’ve got your data and you’re waiting for something to give you a hint, give you a smell, and then you might tune it a bit.

    So the joke, post-tuning, is this: My name is James and I’m a workaholic … I haven’t had a job in 22 years.

    JW: Do you get a special medal for 22 years? A little coin?

    JL: I know that James Thomas is interested in how a particular idea – a little phrase – is tuned in a Milton Jones or a Steven Wright kind of way for a joke. Like you might do with a test. It has to be harmonised. Somebody comes along with what the next part of the joke might be. Similarly in software testing: you’ll say “here’s the problem I have” and someone else will say “I know how to use that problem you found; here’s my exploit.”

    DS: I’d like to build on that idea. I found it fascinating working with James Thomas. It was very rewarding. Some of my favourite parts were watching him dissect his thought process as he went through joke analysis. Which, by the way, there’s a nice joke about …

    <pause>

    Erm, James, how does that go? It’s fruitless process and the frog dies … I think I botched the ending…

    This is the quote Damian was looking for:“Explaining a joke is like dissecting a frog. You understand it better but the frog dies in the process.” – E.B. White

    JW: That’s like testing also. You run a series of tests, you set up everything and then you botch the ending and don’t get anything useful out of it.

    DS: Julius Sumner Miller used to have a TV show in the 60s or 70s. He used to do experiments and loved it when his experiments would fail. He would make a big point out of how it was an unexpected result and you could learn from that. Perhaps there’s something similar in jokes. You think that you might have a punch line that works, that produces humour and mirth. But it does not, and figuring out why that is so…

    But back to the point about building on jokes I find it interesting to see how sometimes the same joke structure can be changed across history. Perhaps it’s a joke about Michael Jackson being the punch line but if you look up the etymology of the joke it started years ago about King Henry. The structure stays the same but the data is replaced.

    JW: Very nice. I’m going to steal the joke that James Lyndsay made. I’m also a workaholic but I haven’t have a job in 50 years! But that’s like testing too. The best testers steal their techniques from others all the time. Probably the way the major amount of learning takes place in the testing community and anybody who thinks it’s bad to steal jokes is probably not a good tester. And when you steal you do what Damian says; you change the content a little and suddenly the joke is funny again.

    Related to that is the technique of repeated jokes. I talked about that before with James Thomas.

    A snippet from that discussion: 

    JW: One technique of comedians is to tell a joke that gets a good response, then keep referring to it later in variant jokes that produce increased laughter (a sort of in-joke for the audience). I’m not sure how that works when a tester keeps reporting the same fault pattern in a work.

    JT: This is a really interesting thought. Comedians call those things callbacks and I spent some time wondering what analogies to software development might be. In humour, the power increases as the connection is remade. In testing, it can be the case that enough of a body of evidence (or body of dismay) can be built up to provoke a fix. But I’ve also been on the receiving end of disbelief and rejection of the issue from developers who see no value in reiterating the known issue.

    JW: When a comedian tells a joke and it goes over well, they will repeat the punch line a number of times and hopefully find different contexts and it gets funnier each time.

    But there’s another phenomenon that occurs. You repeat the same joke over and over again, and it stops being so funny for some people. My wife, for example … we’ve been married 55 years and the reason we’ve stayed together is that she doesn’t remember my jokes.

    If you keep finding the same error that some developer or team is making there’s no problem of them accepting what you’re saying but they get really frustrated. You get frustrated; keeping finding the same thing, keep reporting it, they don’t seem to do anything about it. That’s a source of big conflict and it’s a way to make jokes not be very funny.

    JT: I was thinking about the same thing and I wonder whether there’s a subtlety. I wonder if it’s not necessarily bad to come across the same underlying fault in multiple contexts that’re still interesting; it’s essentially the same issue but a different way into it. Someone who spends all their time searching in a place where they already know there’s a problem and reporting slight variants of it – much like someone who beats a joke to death – that is a problem. Repetition doesn’t have to mean bad.

    DS: Here’s an idea that might be related. There’s a lot of old chestnuts, old jokes that’ve been told countless times and many people will say “I’ve heard that.”

    I’m a big fan of what they call anti-humour which is taking a very typical joke structure and set-up and turning it on its ear. Somebody starts to tell a joke and the listener is primed and ready for an unexpected ending [and] I find it hilarious for the ending to be entirely expected. Which in itself becomes funny.

    I’ll give you an example: What’s red and smells like blue paint?
    Red paint.

    That idea of taking a script, a joke script, that has been run and executed many times and beaten to death. It might be interesting to take that as the premise and change something about it so that it’s unexpected again in an expected way.

    JW: I’m going to tell a joke… A comedian takes a friend of his to the comedian’s convention. The comedians know all the jokes, so when they get up they just say a number and everyone starts laughing.

    The friend says “what are they doing?” and the comedian says “they’re all comedians and they know all the jokes so we catalogue them all and save a lot of time so we don’t have to repeat the whole joke”.

    Now I know two endings for this joke… One of them is, this guy’s just told number 739 and the room went wild over that and the comedian says “we hadn’t heard that one before.”  The other ending is that another guy gets up and says his number but nobody was laughing and the friend says “you see, it’s not just the joke, it’s the way he tells it”.

    So what does that have to do with testing?

    The most important and least well-done part of testing is telling the joke. We find a joke, we find an unexpected ending and many, many testers are not good at reporting what they find. They tell it in a way that’s not funny. It’s blaming, they call people stupid for making these mistakes, they’re not clear what the joke was. If you’re going to be a good tester, you’ve got to practice your presentation.

    JL: People grasp narratives better than they grasp lists of bullets. If you’ve got that circularity that Jerry was talking about of jokes coming back, particularly.

    I’ve come across situations where you say “… of course that problem what we had with the null pointer exception it’s here too” and that becomes the punchline, the thing that spins it time and time again.

    Weirdly enough in Minecraft every three or four sets of release notes they talk about removing Herobrine.

    But the character has never existed. Running jokes become a kind of narrative and also become in-jokes which then makes it an inclusion/exclusion thing. That can help to get it across (if you’re in). But as soon as you come across someone who doesn’t know who Herobrine is, or even what Minecraft is then suddenly the joke falls flat.

    JW: I’ve been writing fiction recently. People respond better to narrative than to lists. One of the techniques in writing a mystery is not to have the solution pulled out of nowhere. You have to be able to tell people how you got to what you found – like testing. You have to have the clues in the mystery somewhere but you don’t want the readers to know “oh, this is the key clue” when it comes up. So what you do is bury the clue in a list. If you have a list of more than three items people maybe remember the first one and the last one but all the middle ones kind of disappear.

    What does this have to do with testing? When you’re reporting issues, people don’t hear the middle ones if you give them a long list. One of the things testers – like comedians – learn to do is to pace out presentation of your jokes. You can’t just give a bullet list of jokes or they don’t appreciate all the ones in the middle.

    You have to do the most important ones first and hold the others in reserve, find out if people are actually paying attention. If not then you’ve got to back up and take care of the top few.

    JT: I read an article by a comedian recently and he said “if you don’t pause, there’s no applause” and if you’re telling your testing story you might want to make sure give your audience chance to take it in.

    Jerry, you also said “start with the most important first” but I think typically comedians are going to want to put their best gag at the end. I’ve heard a lot of comedians say something like “I’ll start with my second-best joke” and finish with the best.

    JW: Let me clarify that a little bit: when I say start with the most important one, I mean start with the one that’s most important to start with

    … because you want to start with one that you’re sure they’ll respond to. They’ll hear it, they won’t deny it, they’ll realise that there’s something they need to do. If they don’t respond to the one they should respond to then you know it’s hopeless to go on with the rest.

    So you’re constantly testing your audience. This is the same technique in presenting your test results. You’ve got to test the presentation. To me, the way that you do that is to give them something that you’re absolutely sure they’ll respond to.

    We use the same trick in development. We have a tool that we’re sure will work in a certain case and we try it and it fails, then we can forget about the rest. It teaches us something. That’s the great thing about testing to me, you always look at things for the unexpected, the possibility, for the other punch line. That’s the mindset that I think distinguishes real testers from ordinary.

    You’re walking along the street on a Spring day and you’re testing. You’re testing the trees, you’re testing the birds, if they’re singing, if they’re nesting in the same places, if something has changed on your route. It’s like a mental disease, except that I find it very proper.

    JT: It’s about observing and reacting, both testing and joking.

    JW: Yeah.

    DS: Testers react to the things they observe and the things in the environment. I have a background in improvisation, short-form improv comedy. It’s essentially creating something from nothing. You start on a blank stage and create theatre. Some times it’s funny, some times it’s serious but it’s very exploratory and reacting to others and to the environment.

    I’m also a raconteur of sorts. I have some stories that over and over again people at parties and gatherings will tell me “Damian, tell me that air-conditioning story” and I say “well you’ve heard it a dozen times” and they say “I don’t care, I want to hear it 13”. They love the story. So I go through and I tell the same story that I’ve told many times.

    Now I realised that when I tell the story I don’t explore and don’t react. I tell a very structured way. Over the tellings of it many times to many people I pause at certain points, I hit certain words, I punch certain lines, I use certain words. It’s all very planned out but it seems very improvisational.

    Then I saw a movie called Comedian. It’s a documentary that follows Jerry Seinfeld. This was after his TV show and he was incredibly famous and the cameras followed him as he went back on the road on the comedy circuit and began to develop brand new material.

    He would show up at a comedy club unannounced and say “hey can I go on and do ten minutes?” and of course the owner of the club said “You’re Jerry Seinfeld of course you can come on” and the cameras would follow him in and he’d do ten minutes and bomb. It was horrible and it was almost surreal to watch to Jerry Seinfeld, the funniest guy, do horribly.

    But then the cameras would show him back in his office and he would change the joke. He would put a pause in here, he would change a word.  One of the jokes was about a disease and he would try scoliosis and he’d try chicken pox and he tried lupus and he realised that lupus was the funniest disease to have in this particular spot in the joke. And as the documentary progressed you saw this same joke evolve over time and the audience would start to chuckle and then start to giggle.

    By the end he’s doing a two-hour HBO special where he’s telling the same joke that started as something completely unfunny in a nightclub and now it’s a beautiful thing where he pauses and says “ah ah ah”. That “ah ah ah” was planned. It’s not Jerry forgetting the next word. It’s absolutely tailored to getting the maximum response.

    And I realised that’s what I’m doing with my stories. What people thought was off the top of my head was actually a very rehearsed story. So this is the idea that I’m proposing. I want to see if anyone has any ideas how that concept might be related to testing.

    MB: My experience with that, aside from working in a comedy club for three years, was one night I saw George Carlin at the Comedy Store in LA. He was preparing his HBO special. He essentially wrote for something in the order of eight or nine months then he took it to the Comedy Store for about six weeks if I recall correctly, and worked it out. He took the material that he’d written and reads it off a clipboard which is, for standup comedy, almost unheard of. Nobody in standup ever did this, in my observation, other than Carlin.

    And he explained it to us. He said this is my process for doing this HBO special. I do a year’s worth of material and I throw it out at the end of each year. I can’t do the same material in a special year after year. I spend these weeks rehearsing this, repeating it, memorizing it. And then the whole time I’m working on the timing, I’m seeing what works, what doesn’t, I’m editing.

    To see him speak in public, he’s very witty and very off-the-cuff and articulate in all kinds of ways but the act was very, very polished and very well-rehearsed. So well-rehearsed that you couldn’t tell how well-rehearsed it was. Very similar to what Damian said, but even more extreme.

    That’s interesting because it’s the opposite of the way most comics do things. They try little things and see if it works. He was working in a scripted way rather than an exploratory. But he was exploring the script and how to perform the script.

    DS: I recently went to the Comedy Store in LA to see Kevin Nealon and Dana Carvey from Saturday Night Live. They hosted a few younger comedians in what they call New Material Night. Comedians come on for the sole purpose of trying brand new jokes to see if they float or sink.

    Carvey brought with him a piece of paper and gave it to an audience member and said “pass this around and everybody just pick a topic off there and yell it up”. And someone would yell “airplane” and he’d tell a joke and if it got a big laugh he’d note it down and if not he’d say “scratch that one off the list”. That was very amazing to see a comedian do that style of comedy trying the material and testing it in that way.

    JW: How is this related to testing and testers? Number one, testing something is a learned skill. Some people have more talent going in than others but if you don’t work at it the way these comedians do … as Michael says it looks so spontaneous because it’s so well-rehearsed.

    I had the experience, many years ago, after I published The Psychology of Computer Programming, I got invited to Carnegie Mellon where they were experimenting on timing people to debug a program by reading it.

    As a psychologist, the first thing I wanted to do was try the experiment out as a subject. So they gave me the thing and I opened it and pointed at the bug. The guy with the stopwatch said “you’re supposed to tell me what you’re doing” and I said “I’ll tell you what I’m doing – there’s the bug.” It was an off-by-one error.

    It took me some time to get him to stop the stopwatch. And he said “you can’t do that. We’ve had hundreds of people and nobody’s done it in less than six minutes and you did it in six seconds, spontaneously.” And I asked whether I got it right and he said “Yeah, but …” and he couldn’t finish the sentence.

    It looked to him like I was just being spontaneous. I asked who he tested before and he said “Oh they were experienced programmers. Some of them had written 10, 20 or 30 programs.” I did a little estimate and said “in my career so far I’ve looked at over 10000 programs…”

    And when I look at a program it’s like a comedian telling a joke.  This is not the first joke I ever told. It’s not even the first time I ever told that joke. There’s many studies that show that after 10000 times playing chess or doing musical performance something different happens. This is the same with testing. Repetition enhances and that’s the first point.

    The second point may be even more important and I don’t think I’ve ever heard anyone discussing it. One of the biggest things you do as a tester is that you are modelling for your audience. If you make an error report and you bungle it and you don’t convince them or you made a wrong tip and you get challenged, how do you respond?

    If you respond to being tested in a way that you don’t want them to respond,  then you’re teaching them. What you want to teach them is: listen to what I’m saying, think about it before you respond, and then do something about it.

    If you don’t act in that way then your audience, the people you’re testing for, won’t act in that way.

    JT: There’s different audiences though aren’t there? In a small group, or 1-1, you get a different reaction from the person you’re joking with than when you’re a comedian on stage with 1000 people watching. Is it the same? Are you being tested in both cases? Differently?

    JW: One of the mistakes you can make when testing your presentation is to look at the person who is responding the best. That is, most favourably. If you really want to do this right you should look at the person who is not responding and find out why they’re not responding, why they don’t understand your presentation. It’s a very common mistake – one person is smiling and the others are frowning. You’ll look at the smiling person.

    MB: There’s an equally bad mistake you can make, which is to dedicate all of your attention to the sole frowning person.

    JW: Absolutely!

    See also IntroPart 2 and Part 3.

    Image: Ebay, Youtube

    Joking With Jerry Part 1 (Hiccupps)

    On February 27, 2016, in Syndicated, by Association for Software Testing
    0

    This is the first part of the transcript of a conversation about the relationship between joking and testing. The jokers were Jerry Weinberg (JW), Michael Bolton (MB), James Lyndsay (JL), Damian Synadinos (DS), and me (JT).

    See also Intro, Part 2 and Part 3.

    –00–

    JL: I wrote one joke for this. It turned up in my head and it may not be original, but it’s mine. And the process of arrival is an interesting thing for me because you don’t necessarily design a joke, you ready yourself for the arrival of the joke. And that certainly parallels some of the things I do when I’m testing … where you are prepared and you’ve got your data and you’re waiting for something to give you a hint, give you a smell, and then you might tune it a bit.

    So the joke, post-tuning, is this: My name is James and I’m a workaholic … I haven’t had a job in 22 years.

    JW: Do you get a special medal for 22 years? A little coin?

    JL: I know that James Thomas is interested in how a particular idea – a little phrase – is tuned in a Milton Jones or a Steven Wright kind of way for a joke. Like you might do with a test. It has to be harmonised. Somebody comes along with what the next part of the joke might be. Similarly in software testing: you’ll say “here’s the problem I have” and someone else will say “I know how to use that problem you found; here’s my exploit.”

    DS: I’d like to build on that idea. I found it fascinating working with James Thomas. It was very rewarding. Some of my favourite parts were watching him dissect his thought process as he went through joke analysis. Which, by the way, there’s a nice joke about …

    <pause>

    Erm, James, how does that go? It’s fruitless process and the frog dies … I think I botched the ending…

    This is the quote Damian was looking for:“Explaining a joke is like dissecting a frog. You understand it better but the frog dies in the process.” – E.B. White

    JW: That’s like testing also. You run a series of tests, you set up everything and then you botch the ending and don’t get anything useful out of it.

    DS: Julius Sumner Miller used to have a TV show in the 60s or 70s. He used to do experiments and loved it when his experiments would fail. He would make a big point out of how it was an unexpected result and you could learn from that. Perhaps there’s something similar in jokes. You think that you might have a punch line that works, that produces humour and mirth. But it does not, and figuring out why that is so…

    But back to the point about building on jokes I find it interesting to see how sometimes the same joke structure can be changed across history. Perhaps it’s a joke about Michael Jackson being the punch line but if you look up the etymology of the joke it started years ago about King Henry. The structure stays the same but the data is replaced.

    JW: Very nice. I’m going to steal the joke that James Lyndsay made. I’m also a workaholic but I haven’t have a job in 50 years! But that’s like testing too. The best testers steal their techniques from others all the time. Probably the way the major amount of learning takes place in the testing community and anybody who thinks it’s bad to steal jokes is probably not a good tester. And when you steal you do what Damian says; you change the content a little and suddenly the joke is funny again.

    Related to that is the technique of repeated jokes. I talked about that before with James Thomas.

    A snippet from that discussion: 

    JW: One technique of comedians is to tell a joke that gets a good response, then keep referring to it later in variant jokes that produce increased laughter (a sort of in-joke for the audience). I’m not sure how that works when a tester keeps reporting the same fault pattern in a work.

    JT: This is a really interesting thought. Comedians call those things callbacks and I spent some time wondering what analogies to software development might be. In humour, the power increases as the connection is remade. In testing, it can be the case that enough of a body of evidence (or body of dismay) can be built up to provoke a fix. But I’ve also been on the receiving end of disbelief and rejection of the issue from developers who see no value in reiterating the known issue.

    JW: When a comedian tells a joke and it goes over well, they will repeat the punch line a number of times and hopefully find different contexts and it gets funnier each time.

    But there’s another phenomenon that occurs. You repeat the same joke over and over again, and it stops being so funny for some people. My wife, for example … we’ve been married 55 years and the reason we’ve stayed together is that she doesn’t remember my jokes.

    If you keep finding the same error that some developer or team is making there’s no problem of them accepting what you’re saying but they get really frustrated. You get frustrated; keeping finding the same thing, keep reporting it, they don’t seem to do anything about it. That’s a source of big conflict and it’s a way to make jokes not be very funny.

    JT: I was thinking about the same thing and I wonder whether there’s a subtlety. I wonder if it’s not necessarily bad to come across the same underlying fault in multiple contexts that’re still interesting; it’s essentially the same issue but a different way into it. Someone who spends all their time searching in a place where they already know there’s a problem and reporting slight variants of it – much like someone who beats a joke to death – that is a problem. Repetition doesn’t have to mean bad.

    DS: Here’s an idea that might be related. There’s a lot of old chestnuts, old jokes that’ve been told countless times and many people will say “I’ve heard that.”

    I’m a big fan of what they call anti-humour which is taking a very typical joke structure and set-up and turning it on its ear. Somebody starts to tell a joke and the listener is primed and ready for an unexpected ending [and] I find it hilarious for the ending to be entirely expected. Which in itself becomes funny.

    I’ll give you an example: What’s red and smells like blue paint?
    Red paint.

    That idea of taking a script, a joke script, that has been run and executed many times and beaten to death. It might be interesting to take that as the premise and change something about it so that it’s unexpected again in an expected way.

    JW: I’m going to tell a joke… A comedian takes a friend of his to the comedian’s convention. The comedians know all the jokes, so when they get up they just say a number and everyone starts laughing.

    The friend says “what are they doing?” and the comedian says “they’re all comedians and they know all the jokes so we catalogue them all and save a lot of time so we don’t have to repeat the whole joke”.

    Now I know two endings for this joke… One of them is, this guy’s just told number 739 and the room went wild over that and the comedian says “we hadn’t heard that one before.”  The other ending is that another guy gets up and says his number but nobody was laughing and the friend says “you see, it’s not just the joke, it’s the way he tells it”.

    So what does that have to do with testing?

    The most important and least well-done part of testing is telling the joke. We find a joke, we find an unexpected ending and many, many testers are not good at reporting what they find. They tell it in a way that’s not funny. It’s blaming, they call people stupid for making these mistakes, they’re not clear what the joke was. If you’re going to be a good tester, you’ve got to practice your presentation.

    JL: People grasp narratives better than they grasp lists of bullets. If you’ve got that circularity that Jerry was talking about of jokes coming back, particularly.

    I’ve come across situations where you say “… of course that problem what we had with the null pointer exception it’s here too” and that becomes the punchline, the thing that spins it time and time again.

    Weirdly enough in Minecraft every three or four sets of release notes they talk about removing Herobrine.

    But the character has never existed. Running jokes become a kind of narrative and also become in-jokes which then makes it an inclusion/exclusion thing. That can help to get it across (if you’re in). But as soon as you come across someone who doesn’t know who Herobrine is, or even what Minecraft is then suddenly the joke falls flat.

    JW: I’ve been writing fiction recently. People respond better to narrative than to lists. One of the techniques in writing a mystery is not to have the solution pulled out of nowhere. You have to be able to tell people how you got to what you found – like testing. You have to have the clues in the mystery somewhere but you don’t want the readers to know “oh, this is the key clue” when it comes up. So what you do is bury the clue in a list. If you have a list of more than three items people maybe remember the first one and the last one but all the middle ones kind of disappear.

    What does this have to do with testing? When you’re reporting issues, people don’t hear the middle ones if you give them a long list. One of the things testers – like comedians – learn to do is to pace out presentation of your jokes. You can’t just give a bullet list of jokes or they don’t appreciate all the ones in the middle.

    You have to do the most important ones first and hold the others in reserve, find out if people are actually paying attention. If not then you’ve got to back up and take care of the top few.

    JT: I read an article by a comedian recently and he said “if you don’t pause, there’s no applause” and if you’re telling your testing story you might want to make sure give your audience chance to take it in.

    Jerry, you also said “start with the most important first” but I think typically comedians are going to want to put their best gag at the end. I’ve heard a lot of comedians say something like “I’ll start with my second-best joke” and finish with the best.

    JW: Let me clarify that a little bit: when I say start with the most important one, I mean start with the one that’s most important to start with

    … because you want to start with one that you’re sure they’ll respond to. They’ll hear it, they won’t deny it, they’ll realise that there’s something they need to do. If they don’t respond to the one they should respond to then you know it’s hopeless to go on with the rest.

    So you’re constantly testing your audience. This is the same technique in presenting your test results. You’ve got to test the presentation. To me, the way that you do that is to give them something that you’re absolutely sure they’ll respond to.

    We use the same trick in development. We have a tool that we’re sure will work in a certain case and we try it and it fails, then we can forget about the rest. It teaches us something. That’s the great thing about testing to me, you always look at things for the unexpected, the possibility, for the other punch line. That’s the mindset that I think distinguishes real testers from ordinary.

    You’re walking along the street on a Spring day and you’re testing. You’re testing the trees, you’re testing the birds, if they’re singing, if they’re nesting in the same places, if something has changed on your route. It’s like a mental disease, except that I find it very proper.

    JT: It’s about observing and reacting, both testing and joking.

    JW: Yeah.

    DS: Testers react to the things they observe and the things in the environment. I have a background in improvisation, short-form improv comedy. It’s essentially creating something from nothing. You start on a blank stage and create theatre. Some times it’s funny, some times it’s serious but it’s very exploratory and reacting to others and to the environment.

    I’m also a raconteur of sorts. I have some stories that over and over again people at parties and gatherings will tell me “Damian, tell me that air-conditioning story” and I say “well you’ve heard it a dozen times” and they say “I don’t care, I want to hear it 13”. They love the story. So I go through and I tell the same story that I’ve told many times.

    Now I realised that when I tell the story I don’t explore and don’t react. I tell a very structured way. Over the tellings of it many times to many people I pause at certain points, I hit certain words, I punch certain lines, I use certain words. It’s all very planned out but it seems very improvisational.

    Then I saw a movie called Comedian. It’s a documentary that follows Jerry Seinfeld. This was after his TV show and he was incredibly famous and the cameras followed him as he went back on the road on the comedy circuit and began to develop brand new material.

    He would show up at a comedy club unannounced and say “hey can I go on and do ten minutes?” and of course the owner of the club said “You’re Jerry Seinfeld of course you can come on” and the cameras would follow him in and he’d do ten minutes and bomb. It was horrible and it was almost surreal to watch to Jerry Seinfeld, the funniest guy, do horribly.

    But then the cameras would show him back in his office and he would change the joke. He would put a pause in here, he would change a word.  One of the jokes was about a disease and he would try scoliosis and he’d try chicken pox and he tried lupus and he realised that lupus was the funniest disease to have in this particular spot in the joke. And as the documentary progressed you saw this same joke evolve over time and the audience would start to chuckle and then start to giggle.

    By the end he’s doing a two-hour HBO special where he’s telling the same joke that started as something completely unfunny in a nightclub and now it’s a beautiful thing where he pauses and says “ah ah ah”. That “ah ah ah” was planned. It’s not Jerry forgetting the next word. It’s absolutely tailored to getting the maximum response.

    And I realised that’s what I’m doing with my stories. What people thought was off the top of my head was actually a very rehearsed story. So this is the idea that I’m proposing. I want to see if anyone has any ideas how that concept might be related to testing.

    MB: My experience with that, aside from working in a comedy club for three years, was one night I saw George Carlin at the Comedy Store in LA. He was preparing his HBO special. He essentially wrote for something in the order of eight or nine months then he took it to the Comedy Store for about six weeks if I recall correctly, and worked it out. He took the material that he’d written and reads it off a clipboard which is, for standup comedy, almost unheard of. Nobody in standup ever did this, in my observation, other than Carlin.

    And he explained it to us. He said this is my process for doing this HBO special. I do a year’s worth of material and I throw it out at the end of each year. I can’t do the same material in a special year after year. I spend these weeks rehearsing this, repeating it, memorizing it. And then the whole time I’m working on the timing, I’m seeing what works, what doesn’t, I’m editing.

    To see him speak in public, he’s very witty and very off-the-cuff and articulate in all kinds of ways but the act was very, very polished and very well-rehearsed. So well-rehearsed that you couldn’t tell how well-rehearsed it was. Very similar to what Damian said, but even more extreme.

    That’s interesting because it’s the opposite of the way most comics do things. They try little things and see if it works. He was working in a scripted way rather than an exploratory. But he was exploring the script and how to perform the script.

    DS: I recently went to the Comedy Store in LA to see Kevin Nealon and Dana Carvey from Saturday Night Live. They hosted a few younger comedians in what they call New Material Night. Comedians come on for the sole purpose of trying brand new jokes to see if they float or sink.

    Carvey brought with him a piece of paper and gave it to an audience member and said “pass this around and everybody just pick a topic off there and yell it up”. And someone would yell “airplane” and he’d tell a joke and if it got a big laugh he’d note it down and if not he’d say “scratch that one off the list”. That was very amazing to see a comedian do that style of comedy trying the material and testing it in that way.

    JW: How is this related to testing and testers? Number one, testing something is a learned skill. Some people have more talent going in than others but if you don’t work at it the way these comedians do … as Michael says it looks so spontaneous because it’s so well-rehearsed.

    I had the experience, many years ago, after I published The Psychology of Computer Programming, I got invited to Carnegie Mellon where they were experimenting on timing people to debug a program by reading it.

    As a psychologist, the first thing I wanted to do was try the experiment out as a subject. So they gave me the thing and I opened it and pointed at the bug. The guy with the stopwatch said “you’re supposed to tell me what you’re doing” and I said “I’ll tell you what I’m doing – there’s the bug.” It was an off-by-one error.

    It took me some time to get him to stop the stopwatch. And he said “you can’t do that. We’ve had hundreds of people and nobody’s done it in less than six minutes and you did it in six seconds, spontaneously.” And I asked whether I got it right and he said “Yeah, but …” and he couldn’t finish the sentence.

    It looked to him like I was just being spontaneous. I asked who he tested before and he said “Oh they were experienced programmers. Some of them had written 10, 20 or 30 programs.” I did a little estimate and said “in my career so far I’ve looked at over 10000 programs…”

    And when I look at a program it’s like a comedian telling a joke.  This is not the first joke I ever told. It’s not even the first time I ever told that joke. There’s many studies that show that after 10000 times playing chess or doing musical performance something different happens. This is the same with testing. Repetition enhances and that’s the first point.

    The second point may be even more important and I don’t think I’ve ever heard anyone discussing it. One of the biggest things you do as a tester is that you are modelling for your audience. If you make an error report and you bungle it and you don’t convince them or you made a wrong tip and you get challenged, how do you respond?

    If you respond to being tested in a way that you don’t want them to respond,  then you’re teaching them. What you want to teach them is: listen to what I’m saying, think about it before you respond, and then do something about it.

    If you don’t act in that way then your audience, the people you’re testing for, won’t act in that way.

    JT: There’s different audiences though aren’t there? In a small group, or 1-1, you get a different reaction from the person you’re joking with than when you’re a comedian on stage with 1000 people watching. Is it the same? Are you being tested in both cases? Differently?

    JW: One of the mistakes you can make when testing your presentation is to look at the person who is responding the best. That is, most favourably. If you really want to do this right you should look at the person who is not responding and find out why they’re not responding, why they don’t understand your presentation. It’s a very common mistake – one person is smiling and the others are frowning. You’ll look at the smiling person.

    MB: There’s an equally bad mistake you can make, which is to dedicate all of your attention to the sole frowning person.

    JW: Absolutely!

    See also IntroPart 2 and Part 3.

    Image: Ebay, Youtube
    Page 1 of 612345...Last »

    Looking for something?

    Use the form below to search the site:


    Still not finding what you're looking for? Drop a comment on a post or contact us so we can take care of it!