“task-list” vs “strategic” testing strategies


We must do check list testing, but too few teams evaluate and evolve their check lists.  Do they still make sense?  Do they still provide the best value?   Some things, e.g. virus checking, we must do because although the risk may be miniscule, the failure consequence (and negative publicity) are astronomical.  

 “Strategic” testing is a deep strategy insight into where to look for the bugs, and the most efficient tools for doing the digging.

The principal problem with Strategic Testing as described, is that most teams lack the data to make decisions upon.  We gather the wrong info.   When we tried to compare Model Based testing with traditional hand crafted testing, we could say that the Model method found X bugs in Y weeks, but we were hard pressed to find comparable data for hand crafted testing.   And it gets more complicated as you look at priority of the bugs found, etc.  

Some of our instincts about testing turn out to misguided.   

There were several enjoyable talks for me this year at PSNQC.  The proceedings are at        http://www.pnsqc.org/proceedings/pnsqc2004.pdf

Is all pairs testing really better than random?      
    See PNSQC talk
 Pairwise Testing: A Best Practice That Isn’t

Are other partitioning methods, e.g. based on code coverage, really better than random?    See

V. N. Nair, et al, “A Statistical Assessment of some Software Testing Strategies and Application of Experimental Design Techinques”, Statistica Sinica 8(1998), 165-184

Simeon C. Ntafos, “On Comparisons of Random, Partition, and Proportional Partition Testing” IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 27, NO. 10, OCTOBER 2001

So the statistics are HARD.   But most teams could do MUCH better in the analysis realm.

See PNSQC_2004 "Becoming a Better Tester"

 

Keith.

Advertisements

About testmuse

Software Test Architect in distributed computing.
This entry was posted in software testing. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s