Bj Rollison posted an interesting blog entry about Combinatorial Testing: Testing with Negative Values. When you do multiple invalid values you must do a separate analysis from that with all valid values.
I worry a lot about how invalid values, which usually suppress any useful usage of the rest of the values (“error masking”), are handled in combinatorial testing. The AETG authors in their original paper covered this quite well. Academics get faulty conclusions (that BC performs as well as AETG) by acknowledging fault masking, but not accounting for it (semantic information).
D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton
The AETG System: An Approach to Testing Based on Combinatorial Design
IEEE Transactions On Software Engineering, July 1997 (Vol. 23, No. 7)
“To avoid having one invalid value mask another the AETG System uses only one invalid value per test case. It creates the test for an invalid value by taking a valid test for the relation and substituting the invalid value in place of the field’s valid value in that test.”
M. Grindal, J. Offutt, S. F. Andler
Combination Testing Strategies – A Survey
SOFTWARE TESTING, VERIFICATION AND RELIABILITY
Softw. Test. Verif. Reliab. 2005; 15:167–199
Published online 10 March 2005 in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/stvr.319
“The each-used, pair-wise, t-wise, and N-wise coverage criteria are purely combinatorial and do not use any semantic information”
“To avoid one error value masking another Cohen et al. suggest that only one error value of any parameter should be included in each test case. This observation was also made and explained in an experiment by Grindal et al. .”
“They showed that BC performs as well, in terms of detecting faults, as AETG and OAs, despite fewer test cases for a number of test objects seeded with 131 faults.”
I posted the following comment to Bj’s blog about this:
I think your analysis about invalid values fails to cover one of the most insidious points about them, which is error masking.
You mention for Testing single invalid input values that
“The above model file produces a baseline set of combinatorial tests that includes all valid combinations as well as combinations that include 1 invalid value . . .”
The “as well as” is critical. You do not get this benefit in your Testing multiple invalid input values analysis. You state early on:
“Testing different combinations of valid values for multiple input parameters that affect a common output condition or state sometimes exposes unexpected issues”
But when not all the values are valid, you don’t have a common output condition. As you describe you have Error and NoError output conditions.
Say one test case instance is
Font: Tahoma, Style: Bold, Effects: Strike, Colors: Purple Size: emptyString,
Suppose it returns an error (or as one commenter indicate errors for every incorrect field). That is great for the testing of errors, but you’ve now screwed up a fundamental assumption of combinatorial testing. Suppose this is the only test case with the pair <Tahoma, Bold> or the pair <Bold, Strike> and that the pair does not work correctly. Because you gave one or more negative inputs, you might never execute the code that involves problem with the pair <Tahoma, Bold>. Combinatorial testing presumes all the values are involved in the output, but when invalid values are used that isn’t the case. Most programs stop processing of inputs (the valid pairs) after detecting one or more (or all) of the invalid inputs.
I believe in multiple invalid value testing as I’ve seen unique errors due to it. However, you must do invalid value testing in a separate combinatorial analysis from the all valid value testing combinatorial analysis. Unlike the analysis you show with emphasis on valid values (weights of 50 for Ariel and 10 for black, etc.), you should emphasize weights on the invalid values for a multiple invalid value combinatorial analysis. In fact, you don’t really care about generating any cases that have only valid values and you could create a constraint to that effect. Valid value testing must be done in a separate analysis. For the single input invalid case, PICT optimizes this for you by doing the “all valid combinations” and a second analysis for “combinations that include 1 invalid value”. When you do multiple invalid values you must manually do the two separate analyses.