Write fewer automated checks/tests


I attended Alan Page’s QA SIG talk on Adventures in Modern Testing and heard 2 interesting quotes:

 “All the testing getting in the way of quality”  and
Help team write fewer tests“.

In talking with Alan after the talk, I discovered I misunderstood which “team” should write fewer tests. Alan was thinking the quality / test team should write fewer or no tests – as those are picked up by developers on feature teams.
Brian Gaudrea at the meeting also mentioned having the test / quality team align better with continuous improvement initiatives that save companies money, to transition from a cost center to a cost reducer.

My own thought is:

How can we help teams write fewer automated checks or tests?

(1) My model based testing historical bias is : don’t write tests (or automated checks as they are now sometimes called), but generate them!
Academia has been working on automated test generation and automated test suite augmentation for decades; see for example: An Orchestrated Survey on Automated Software Test Case Generation.
We need to encourage all of development to discover ways to generate more effective tests, while writing fewer hand-scripted automated checks.     Besides model based testing, there are static analysis tools, symbolic executor tools, and many other tools that auto-generate tests.
We need to understand how to generalize a test issue so tests aren’t written individually.  A simple example, from John Lambert, was his realization that many devs weren’t verifying simple things like null object values as possible inputs.
Did he write a bunch of tests with nulls?  NO
He wrote a simple scanner that determined when a method could accept an object, then generated and ran the null object test.  He found lots of issues.
More recently, I learned about the difficulty of counting characters, glyphs, or Unicode units (UTF-*) for Emoji’s.
Should you write a bunch of Emoji tests?   NO
You should find a way to add Emoji’s to your current string character tests.

(2) I think the future is humans writing fewer automated checks.  Automated check writing will instead be done via artificial intelligence.
With the AI tool sets and testing data sets we have today, it will not be long before most automated checks are created by AI, not by humans.
Alan Page also mentioned the GTAC 2007 talk by Ed Keyes – Sufficiently Advanced Monitoring is Indistinguishable from Testing.   Perhaps that can be combined with an automated adversary, similar to generative adversarial networks, or GANs.   The AI generates input based on previous logs and monitoring results, and the monitoring tries to determine if the system is being fooled (increases the error rate).   This is a more advanced form of simple fuzz testing (generated inputs looking merely for simple failures).

I think all of the above can help with Alan’s motto:
“Accelerate the achievement of shippable quality”

Advertisements

About testmuse

Software Test Architect in distributed computing.
This entry was posted in software testing. Bookmark the permalink.

4 Responses to Write fewer automated checks/tests

  1. Matt Griscom says:

    AI/ML isn’t there yet, and won’t be for five years IMO.

    • testmuse says:

      And yet companies like https://www.appvance.com/ claim it is already here “Machine Learning & Expert Systems Approach to QA Automation

      Appvance AI quickly learns how real users use an application, learns how to create proper requests, automatically generates correlation data, connects to data sources, and then automatically generates 100’s, 1000’s, or 10,000+ scripts at your command, literally in a few seconds. Blazingly fast, no-hands, automatic test generation.”

      • Matt Griscom says:

        My understanding is that, while this could be useful, it’s really ineffective at answering the question “Does the system do what we need it to do?” quickly.

  2. Nash Stewart says:

    I’m excited for AI being able to generate more effective automated tests. A lot of tests just need humans to design them, however. At some organizations I’ve worked for the product and configurations are so complex that automated test design can take a lot of planning.

    Sure an AI can generate emoji tests or null tests in the future, but I’m not convinced that organizations will trust AI to make tests that are very complex. I love AI though. It’s exciting to think of the possibilities.

    On the first point, “All the testing getting in the way of quality”. This is so true. I think it’s better to have a little bit more complex tests that include more functionality than to have more tests. It cuts down on test suite execution time and allows for more robust test suites that are easier to manage and maintain.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s