Pair-wise covering array versus “combinations”


A tester recently asked:

Subject: Pairwise testing over repeated set of parameters

 

I have a test which includes adding headers to an http message.  There are 10 headers I need to test and I’d like to test all possible combinations when sending a message.  If I generate all possible pairs, that gives me 45 possible tests (not including setting the same header twice).  I’d like to reduce this number of tests by setting 4 headers at once.  This would, of course, allow me to test 6 pairs at a time.  I tried creating a pict file similar to below, but it generated far to many combinations, and what I noticed is that it seems to treat a combination of 

 

H1        H2        H3        H4

A          B          C          D

 

As a different test case than

 

H1        H2        H3        H4

B          C          D          A

 

MODEL

 

H1: A, B, C, D, E, F, G, H, I , J

H2: A, B, C, D, E, F, G, H, I , J

H3: A, B, C, D, E, F, G, H, I , J

H4: A, B, C, D, E, F, G, H, I , J

 

H1 <> H2;

H1 <> H3;

H1 <> H4;

H2 <> H3;

H2 <> H4;

H3 <> H4;        

To which I replied:

I suspect the confusion concerns the term “combinations” which you expect in a strict mathematical sense.

 

First you state:  “test all possible combinations”  and give 2 parameters (H1 and H2) each with 10 values.  This gives 100 (10×10) pair-wise combinations (as PICT generates) – in this case the Cartesian product, but you state the answer as 45 which is the number of combinations from a single set (choosing without replacement 2 items from A-J).

 

PICT treats the values of each parameter as a separate domain.   It is most easily illustrated with:

H1: A, B, C, D, E, F, G, H, I , J

H2: a, b, c, d, e, f, g, h, i, j

 

You expect PICT to effectively ignore the case (treat values in different headers as the same).  But they are all considered as unique.  Think of each value has having its header prefixed to it.  H1:a is not the same as H2:a

 

This is also why you don’t get the result you desire with 4 headers, because:

H1        H2        H3        H4

H1:A    H2:B    H3:C     H4:D

 

Is a different test case than

 

H1        H2        H3        H4

H1:B     H2:C    H3:D    H4:A

 

You want combinations as unordered selections from a set.  Strictly mathematically speaking, the types of combinations PICT is geared towards are covering arrays (see www.pairwise.org)

 

You don’t need PICT to generate simple combinations from a set. 

Using the Excel COMBIN function we see COMBIN(10,2) = 45.  For your example, you expect 210 as returned from Excel’s COMBIN(10,4)

 

A simple Excel query will create what you want.     Define four columns labeled H1, H2, H3, H4 with the 10 values.  Select each column range of cells and “label” it.

Then save your Excel file.

Instructions below for Office 2007:

            Select H1 thru J, right click, choose “Name a range” (auto set to H1) and click OK.  Repeat H2-H4.

            Click on “Data” tab and select “From Other Sources”.  Choose “From Microsoft query”.  Choose “Excel files”

            Choose the file you just created.  Add the four columns you selected.   Ignore the error about unable to join.

            [The MS Query editor pops up]  Click SQL and enter something like:   (mostly just add the WHERE clause).  “File” Menu “Return Data”.

 

SELECT H1_.H1, H2_.H2, H3_.H3, H4_.H4

FROM `Combo.xlsx`.H1_ H1_, `Combo.xlsx`.H2_ H2_, `Combo.xlsx`.H3_ H3_, `Combo.xlsx`.H4_ H4_

WHERE H1_.H1 < H2_.H2 AND H2_.H2 < H3_.H3 AND H3_.H3 < H4_.H4

 

This produces the 210 combinations you seek. 

 

Note your analysis of your problem presumes that the ordering of the headers is NOT significant.  Thus whether header B appears in first or second place is immaterial.

A disadvantage of the simple technique I used for generation is that it guarantees an ordering to the parameters.  Thus if have A in second thru fourth position of the header were important, this set of 210 cases would never find it because A only occurs in the first header ever.   But that is what you asked for. 

 

My paranoid tester self would at least shuffle the four values (in each case) amongst the headers randomly (still only 210 cases) such that every value appears in every header position at least once.

Advertisements

About testmuse

Software Test Architect in distributed computing.
This entry was posted in software testing. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s