Model-Based Testing (MBT) Productivity


At ETSI MBT UC I’ve seen more real data presented on MBT success in Industry than ever before. When the slides are publicly posted (soon) I will summarize many of them. In my keynote I gave our standard data of 40% savings in productivity with same requirements coverage (a “quality” measure) which comes from

Grieskamp, W., Kicillof, N., Stobie, K. and Braberman, V. (2011), Model-based quality assurance of protocol documentation: tools and methodology. Software Testing, Verification and Reliability, 21: 55–71. doi: 10.1002/stvr.427

quoted below:

The feasibility and scalability of MBT is evident in the fact that the project has delivered ‘model to metal’ test suites for over 75 protocols, and this number is growing. At the end of the project, around half of the 250 protocols in scope will have been modeled, reflecting an investment of over 50 person-years in MBT application alone. In addition a substantial investment has been made in tool development, based on a continuous feedback loop from the test-suite development process into the Spec Explorer development team. According to a preliminary statistical analysis, the application of MBT resulted in a 42% productivity gain when compared with traditional test suites in a site where similar numbers of requirements have been verified with each approach.

I also became aware of the D-MINT project – a large European study specifically targeted at sponsoring and studying MBT projects in Industry. Their results across over 10 companies and projects:

These techniques were used in real products for direct comparison. Analysis of the real costs involved in time and investments across all the consortium members showed that not only could direct test costs be reduced by 15% using model-based testing, but that test coverage could be improved by 10%. This translates into an overall improvement of some 20 to 25% in test costs.

Of course vendors and companies selling MBT services claim much higher returns such as 5x or 7x.
I need to see the data.

Advertisements

About testmuse

Software Test Architect in distributed computing.
This entry was posted in Uncategorized. Bookmark the permalink.

2 Responses to Model-Based Testing (MBT) Productivity

  1. Holger says:

    Hi,
    thanks for this interesting post. Please feel free to read my article concerning model based test specifications: http://blog.protocolbench.org/2013/07/next-generation-of-epassport-testing
    In my blog I’m publishing ideas concerning protocols and the way to model and test them (currently focussing on ePassports).
    Cheers,
    Holger

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s