“small” versus “large” tests

am frequently asked and give advice on simple versus complex tests.  As the article below quotes:

as expressed by Beizer [1995], the choice is not just between “large” and “small” tests, but between “several simple, obvious tests” and “fewer, but grander, tests”.

I have generally leaned toward the fewer, grander as most of MS seems to be in the several obvious camp.   However, I’ve realized this can cause complications for code coverage based Test Prioritization.   Below is the first article I’ve seen investigating this.  I have two major qualms with it:

            1) They use function coverage for minimization resulting in significant defect detecting losses.   Block coverage minimization isn’t so bad.

            2) Their method of creating larger test cases mostly just shows setup/cleanup cost differences.

They use seeded faults.  This is it typical for academic papers and rare to avoid.  

On test suite composition and cost-effective regression testing


Regression testing is an expensive testing process used to revalidate software as it evolves. Various methodologies for improving regression testing processes have been explored, but the cost-effectiveness of these methodologies has been shown to vary with characteristics of regression test suites. One such characteristic involves the way in which test inputs are composed into test cases within a test suite. This article reports the results of controlled experiments examining the effects of two factors in test suite composition—test suite granularity and test input grouping—on the costs and benefits of several regression-testing-related methodologies: retest-all, regression test selection, test suite reduction, and test case prioritization. These experiments consider the application of several specific techniques, from each of these methodologies, across ten releases each of two substantial software systems, using seven levels of test suite granularity and two types of test input grouping. The effects of granularity, technique, and grouping on the cost and fault-detection effectiveness of regression testing under the given methodologies are analyzed. This analysis shows that test suite granularity significantly affects several cost-benefit factors for the methodologies considered, while test input grouping has limited effects. Further, the results expose essential tradeoffs affecting the relationship between test suite design and regression testing cost-effectiveness, with several implications for practice.



ACM Transactions on Software Engineering and Methodology (TOSEM) archive
Volume 13 ,  Issue 3  (July 2004) table of contents

Pages: 277 – 331  
Year of Publication: 2004


Gregg Rothermel  University of Nebraska—Lincoln, Lincoln, NE Sebastian Elbaum  University of Nebraska—Lincoln, Lincoln, NE Alexey G. Malishevsky  Oregon State University, Corvallis, OR Praveen Kallakuri  University of Nebraska—Lincoln, Lincoln, NE Xuemei Qiu  Oregon State University, Corvallis, OR


About testmuse

Software Test Architect in distributed computing.
This entry was posted in software testing. Bookmark the permalink.

One Response to “small” versus “large” tests

  1. Pingback: Thoughts on Rimby’s How Agile Teams Can Use Test Points | Musing about Software Testing

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s