Reality Check: % of test cases automated

One of the things I like most about being a Test Architect at Microsoft is having senior tester peers to compare notes with.  Recently a tester solicited input about % of test cases to automate and here were two responses that I completely agree with.

 

 

From: William Rollison
Subject: RE: Reality Check: % of test cases automated

 

Without context the percentage of automated tests really doesn’t make much sense. For example, if we were talking about the BVT/BAT test suite than a target goal of 100% automation makes pretty good sense.  Having a high percentage of automated regression tests in the regression test suite makes a lot of sense as well.

 

But, someone who just throws out numbers and wildly says that nn% of tests should be automated has absolutely no clue about the value or purpose of test automation, doesn’t realize the cost of test automation development, and certainly hasn’t thought about a realistic automation strategy that will provide a reasonable ROI for the group or the company.

 

I just simply can’t believe that we can’t make smarter business decisions of what to automate and instead simply try to measure progress with these ridiculous measures such as % of automation or other wild ass guesses.

 

I don’t know whether to laugh at the fact that we seem to have some who are so clueless about testing, or to cry because we in test can’t articulate the challenges of testing more clearly.

 

I recently read Software Testing Techniques: Finding the Defects that Matter and completely concur with Loveland, et. el., assessment about building credibility. They state "One of the things that testers must do to build a valuable and respectful relationship with development is to establish technical credibility." 

 

I think one reason why some may not understand testing is because testing today still lacks credibility with management. That perception will persist until we really understand there is a difference between "bug-finders" and professional testers.

 

Now, I am shutting my eyes tightly, tapping my heals together, repeating to myself…"There’s no place like home, there’s no place like home…" and hoping that maybe tomorrow will bring the reality we seek.

 

– Bj –

Test Architect

I.M. Testy blog

 

From: Noel Nyman
Subject: RE: Reality Check: % of test cases automated

 

>> If you feel your team is doing a good job of automation coverage…

Taking automation percentages out of context gives you little more than arbitrary numbers that probably have no meaning at all. You can achieve high automation coverage numbers for products that have no GUI, though seldom 100%. You can rarely get the same numbers for GUI oriented products, and if you try to do that, you’ll spend more time maintaining the automation than finding any bugs.

 

Your numbered list

1.                   % of your total test cases that are automated

2.                   % of your automatable test cases that are automated

3.                   How many test cases do you have in your total test bed?

also assumes that all “test cases” have about the same value, size and coverage. I’ve seldom seen that situation except with thousands of one line “test cases” that made the value of counting them useless.

 

You can also define “coverage” to make your existing automation appear to have good numbers…line, path, feature, function, etc…especially if you feel that you have adequate automation now.

 

Keith can give you the best realistic guidance because he can easily review your GUI/non-GUI code ratios, establish the risky areas that require proportionally more testing, select several coverage metrics that best help achieve your testing goals, and identify which areas you can automate efficaciously. If your senior managers won’t take Keith’s word for it, they won’t believe any of the rest of us either!

Do not seek to follow in the footsteps of the wise. Seek what they sought.
Basho