At ETSI MBT UC I’ve seen more real data presented on MBT success in Industry than ever before. When the slides are publicly posted (soon) I will summarize many of them. In my keynote I gave our standard data of 40% savings in productivity with same requirements coverage (a “quality” measure) which comes from
Grieskamp, W., Kicillof, N., Stobie, K. and Braberman, V. (2011), Model-based quality assurance of protocol documentation: tools and methodology. Software Testing, Verification and Reliability, 21: 55–71. doi: 10.1002/stvr.427
The feasibility and scalability of MBT is evident in the fact that the project has delivered ‘model to metal’ test suites for over 75 protocols, and this number is growing. At the end of the project, around half of the 250 protocols in scope will have been modeled, reflecting an investment of over 50 person-years in MBT application alone. In addition a substantial investment has been made in tool development, based on a continuous feedback loop from the test-suite development process into the Spec Explorer development team. According to a preliminary statistical analysis, the application of MBT resulted in a 42% productivity gain when compared with traditional test suites in a site where similar numbers of requirements have been verified with each approach.
These techniques were used in real products for direct comparison. Analysis of the real costs involved in time and investments across all the consortium members showed that not only could direct test costs be reduced by 15% using model-based testing, but that test coverage could be improved by 10%. This translates into an overall improvement of some 20 to 25% in test costs.
Of course vendors and companies selling MBT services claim much higher returns such as 5x or 7x.
I need to see the data.