am frequently asked and give advice on simple versus complex tests. As the article below quotes:
as expressed by Beizer , the choice is not just between “large” and “small” tests, but between “several simple, obvious tests” and “fewer, but grander, tests”.
I have generally leaned toward the fewer, grander as most of MS seems to be in the several obvious camp. However, I’ve realized this can cause complications for code coverage based Test Prioritization. Below is the first article I’ve seen investigating this. I have two major qualms with it:
1) They use function coverage for minimization resulting in significant defect detecting losses. Block coverage minimization isn’t so bad.
2) Their method of creating larger test cases mostly just shows setup/cleanup cost differences.
They use seeded faults. This is it typical for academic papers and rare to avoid.
Regression testing is an expensive testing process used to revalidate software as it evolves. Various methodologies for improving regression testing processes have been explored, but the cost-effectiveness of these methodologies has been shown to vary with characteristics of regression test suites. One such characteristic involves the way in which test inputs are composed into test cases within a test suite. This article reports the results of controlled experiments examining the effects of two factors in test suite composition—test suite granularity and test input grouping—on the costs and benefits of several regression-testing-related methodologies: retest-all, regression test selection, test suite reduction, and test case prioritization. These experiments consider the application of several specific techniques, from each of these methodologies, across ten releases each of two substantial software systems, using seven levels of test suite granularity and two types of test input grouping. The effects of granularity, technique, and grouping on the cost and fault-detection effectiveness of regression testing under the given methodologies are analyzed. This analysis shows that test suite granularity significantly affects several cost-benefit factors for the methodologies considered, while test input grouping has limited effects. Further, the results expose essential tradeoffs affecting the relationship between test suite design and regression testing cost-effectiveness, with several implications for practice.