I’ve not done the research to support this, but my gut feel has always been that scenario testing (especially when done with models) is an initial big win over simple feature testing. I drew some graphs for a colleague to attempt to illustrate the point.
Scenario testing is only cost effective when you don’t find too many bugs. There is a cost to diagnosing and reproducing bugs which is higher than fine grained unit or component tests. However, more bugs can be found faster with fewer tests. So if there aren’t too many bugs, the savings in test writing will far outweigh the slightly increased cost of reporting bugs.
Because it takes a lot of little tests to cover a product, it takes time to create and automate all those functional or component tests.
Scenario tests cover more of the product faster at the beginning providing better
-
High quality tests.
-
Focus on improving test coverage. Improving test coverage doesn’t mean blindly adding test cases but adding tests with clear goals in mind and being aware of what is being tested and how we’re testing.
A problem with Scenario tests, in my mind, it they are not cost effective for reaching the highest levels of code coverage (whether that should even be a goal, is another separate, big, debate).
But a combination strategy can leverage each techniques strengths for optimal testing.
