“Don’t confuse manual testing with manual testing”
This made sense in context today at a meeting (as I will explain). The issue is people using one term to mean something else. In particular to think that the method of testing, “manual” or “automated”, really tells me anything about what, why, when or how they are testing. I will expand upon Lessons Learned in Software Testing, Lesson 108 “Don’t equate manual testing to automated testing.” The Lessons Learned book makes an implicit assumption about what manual testing is. As I learned from Specialisterne, not everyone treats manual testing as improvisation. Specialsterne perform “specialized work where the requirements for being systematic, having a strong attention to details and being persistent are important”. The authors also narrowly state “automated test does the same thing each time it runs”. They’ve fallen into a trap about what automated testing can be. I’ve done model based testing and stress testing which can randomly do things. Most automated fuzz testing is all about doing different things each time it runs. Harry Robinson is promoting the notion of Exploratory Test Automation which doesn’t have to do the same thing each time it runs or you can use a tool like Pex for Automated Exploratory Testing.
“Doing the same thing each time it runs” is an aspect of retesting of defect fixes, not a quality of manual versus automated. In my PNSQC 2009 paper, Too much automation or not enough? When to automate testing, I discussed various levels of scripted tests. Very detailed scripted tests attempt to do the same thing each time they are run whether via automation or by human.
So after hearing that a feature team did most of their testing manually using vendors, our Test Director was a little surprised why our exploratory manually testing vendors found issues. I asked the feature team test lead to clarify the type of manual testing they asked their vendors to do. It was scripted regression testing. At which point I said “Don’t confuse manual testing with manual testing” which more precisely should be interpreted as “Don’t confuse exploratory testing looking for bugs with detailed scripted regression testing” (regardless of it being manual or automated).