I attended Alan Page’s QA SIG talk on Adventures in Modern Testing and heard 2 interesting quotes:
“All the testing getting in the way of quality” and
“Help team write fewer tests“.
In talking with Alan after the talk, I discovered I misunderstood which “team” should write fewer tests. Alan was thinking the quality / test team should write fewer or no tests – as those are picked up by developers on feature teams.
Brian Gaudrea at the meeting also mentioned having the test / quality team align better with continuous improvement initiatives that save companies money, to transition from a cost center to a cost reducer.
My own thought is:
How can we help teams write fewer automated checks or tests?
(1) My model based testing historical bias is : don’t write tests (or automated checks as they are now sometimes called), but generate them!
Academia has been working on automated test generation and automated test suite augmentation for decades; see for example: An Orchestrated Survey on Automated Software Test Case Generation.
We need to encourage all of development to discover ways to generate more effective tests, while writing fewer hand-scripted automated checks. Besides model based testing, there are static analysis tools, symbolic executor tools, and many other tools that auto-generate tests.
We need to understand how to generalize a test issue so tests aren’t written individually. A simple example, from John Lambert, was his realization that many devs weren’t verifying simple things like null object values as possible inputs.
Did he write a bunch of tests with nulls? NO
He wrote a simple scanner that determined when a method could accept an object, then generated and ran the null object test. He found lots of issues.
More recently, I learned about the difficulty of counting characters, glyphs, or Unicode units (UTF-*) for Emoji’s.
Should you write a bunch of Emoji tests? NO
You should find a way to add Emoji’s to your current string character tests.
(2) I think the future is humans writing fewer automated checks. Automated check writing will instead be done via artificial intelligence.
With the AI tool sets and testing data sets we have today, it will not be long before most automated checks are created by AI, not by humans.
Alan Page also mentioned the GTAC 2007 talk by Ed Keyes – Sufficiently Advanced Monitoring is Indistinguishable from Testing. Perhaps that can be combined with an automated adversary, similar to generative adversarial networks, or GANs. The AI generates input based on previous logs and monitoring results, and the monitoring tries to determine if the system is being fooled (increases the error rate). This is a more advanced form of simple fuzz testing (generated inputs looking merely for simple failures).
I think all of the above can help with Alan’s motto:
“Accelerate the achievement of shippable quality”