I just attended the fourth Workshop on Huesristic and Exploratory Testing (WHET4) the weekend of July 7-8 just before CAST focused on the question: What is Boundary Testing?

It was quite intriuging as it quickly became apparent boundary testing is not as well understood as we might think. It has not been covered in much depth as it is considered so simple. Most people tie the concept of boudary testing with equivalence class. But equivalence classes are equally simple on the surface, but have subtle complexity. Now when you talk about the boundary of an equivalence classes, things become even murkier.

It was quite intriuging as it quickly became apparent boundary testing is not as well understood as we might think. It has not been covered in much depth as it is considered so simple. Most people tie the concept of boudary testing with equivalence class. But equivalence classes are equally simple on the surface, but have subtle complexity. Now when you talk about the boundary of an equivalence classes, things become even murkier.

**A History of some things I already understood:**

The simple, most classical example is easy: We have a legal range of integer values, e.g. 0-100. Here the boundaries are pre-defined. We have at least 2 equivalence classes, legal values (0-100) and other integer values that are not legal. Most people sub-divide illegal values into 2 classes, the lower illegal values (in this case all negative integers) and the upper illegal values (in this case integers > 100).

So far the equivalence classes are based on legal values or their relationship (lower or higher) to the legal values. Studies by Hamlet and others have shown that off-by-1 errors are unfortunately not uncommon. Off-by-1 errors frequenly occur due to using < when <= or vice-versa are meant and similar mistakes. So some people, based on likelihood of errors look at the boundary values where an off-by-1 error could occur as the own special equivalence classes. The 100 limit could be encoded as <=100, but might be mistakenly coded as <100. Thus if 100 were treated as an illegal value it have been mis-classified and be an error. Similarly, the 100 limit could be coded as <101, but mistakenly coded as <=100. Thus 101 could mistakenly be treated as a legal value. This is why, for this example, the limit and 1 greater the limit are so frequently quoted as the boundary values of this example. Similarly for 0 and -1 as the lower legal value and 1 less than the lower value. (There are other reasons 0 and negative number could be interesting making them even better candidates for finding a bug).

A linearizable variable’s values can be mapped to a number line. From the above, everyone knows how to apply classical boundary testing to linearizable input variables where the analysis is so simple.

**Now how to go beyond linearizable input variables?**

That was a major topic of this WHET.

A simple extension: go beyond just input variables. Obviously output variables, but also other influencers such as time and outcomes such as internal state change. Also, of course, consider more than one variable at a time in the analysis.

But what about linearizable? Some characterstics about a number line:

1) it is ordered (has precendence)

2) it is continuous (descretely as integers sometimes, but w/o missing integers)

1) it is ordered (has precendence)

2) it is continuous (descretely as integers sometimes, but w/o missing integers)

Myself and many WHET attendees didn’t feel these were essential required characteristics. A major disagreement arose about the partitioning into equivalence classes. I’m in the partitioning the domain into disjoint sets of equivalence classes camp, but equally many felt you could have boundaries between overlapping sets.

Another area, of no surprise to me, was the stress (or performance) notion of boundary testing. Scott Barber, from the performance perspective, discussed testing for the knee of a curve. Here we know there are boundaries, but we don’t know what they are until we test. This is different from hunting for unkonwn boundaries. Another characteristic, not unique to stress testing, is what I will cause physcial boundaries. The example I focus on is 100% cpu utilization. For a single cpu (clocked at the same speed and many other qualifications), you can’t go above 100%. It isn’t that 101% is an illegal value, it is an impossible value.

The most interesting new concept I learned was Cem Kaner’s way of expressing boundaries as mis-classification or chance of confusion.

I have often tried to instruct my students in this. Below is one of my classic non-linear boundary examples that also rely on a theory of error.

I have often tried to instruct my students in this. Below is one of my classic non-linear boundary examples that also rely on a theory of error.

Program feature to set ON or OFF some feature and is specified as

SET ON | OFF

What tests would you try?

All students give

SET ON

SET OFF

Now what would you give for an illegal value? A typical new student answer is:

SET X

to which I push back: Yes this would be an invalid value, but how likely is it to find any other bug, or as Kaner says, can you have a more powerful testcase? (Case A is more powerful than Case B, if Case A finds all the defects of Case B and more).

Better are cases like:

1. SET

2. SET ONN

3. SET OF

Case 1 is used by those choosing to see if there is an undocumented default.

Cases 2 and 3 I consider boundary cases, that is they could be misclassified.

Case 3 theory of error is that programmer may have compared the first 2 characters as that is sufficient to distinguish the two legal values (or even more lazily only compares the second character).

Case 2 theory of error is similar to 3, except it also incorporates the idea of only comparing up to the length of the expected value.

The beauty of cases 2 and 3 is besides being boundary cases, they are very concievable typographical errors (making failures more justifiable). In fact, for text strings, typographical errors are typically boundary errors. If you consider spell correcting programs, they have algorithms for determining the "nearest" correct spelling. I consider mis-spellings that a spell correcting program would sugget or change to the correct spelling, as being boundaries.

All students give

SET ON

SET OFF

Now what would you give for an illegal value? A typical new student answer is:

SET X

to which I push back: Yes this would be an invalid value, but how likely is it to find any other bug, or as Kaner says, can you have a more powerful testcase? (Case A is more powerful than Case B, if Case A finds all the defects of Case B and more).

Better are cases like:

1. SET

2. SET ONN

3. SET OF

Case 1 is used by those choosing to see if there is an undocumented default.

Cases 2 and 3 I consider boundary cases, that is they could be misclassified.

Case 3 theory of error is that programmer may have compared the first 2 characters as that is sufficient to distinguish the two legal values (or even more lazily only compares the second character).

Case 2 theory of error is similar to 3, except it also incorporates the idea of only comparing up to the length of the expected value.

The beauty of cases 2 and 3 is besides being boundary cases, they are very concievable typographical errors (making failures more justifiable). In fact, for text strings, typographical errors are typically boundary errors. If you consider spell correcting programs, they have algorithms for determining the "nearest" correct spelling. I consider mis-spellings that a spell correcting program would sugget or change to the correct spelling, as being boundaries.

WHET4 attendees:

Rob Sabourin, Karen Johnson, David Gilbert, Michael Bolton, Cem Kaner, Ross Collard, Doug Hoffman, Keith Stobie, Mike Kelley, Tim Coulter, Henrik Andersson, Scott Barber, Dawn Haynes, James Bach, Jon Bach, Paul Holland

Rob Sabourin, Karen Johnson, David Gilbert, Michael Bolton, Cem Kaner, Ross Collard, Doug Hoffman, Keith Stobie, Mike Kelley, Tim Coulter, Henrik Andersson, Scott Barber, Dawn Haynes, James Bach, Jon Bach, Paul Holland

Advertisements

Very useful information. Thanks for sharing this. I like your way of presentation.

Pingback: Thoughts on Rimby’s How Agile Teams Can Use Test Points | Musing about Software Testing