I’ve spent the last 3 days at International Symposium on Software Testing and Analysis (ISSTA).
Always interesting to catch up on the researchers of software testing. The details of static analysis can be deep and beyond me, but most of it is quite approachable. For example, A Metric for Software Readability abstract is only at the junior college level of readability J .
My own groups work was summarized in the keynote (The Real Value of Testing) as the only bright light in the failure of specification based development including models. As I suspected, electronic voting for elections is still a hoax (details: Are Your Votes Really Counted? Testing the Security of Real-world Electronic Voting Systems).
The big new term I learned that I’m behind the times on is “Concolic Testing”. From Concolic Testing tutorial (2007):
Concolic testing automates test input generation by combining the concrete and symbolic (concolic) execution of the code under test. Traditional test input generation techniques use either (1) concrete execution or (2) symbolic execution that builds constraints and is followed by a generation of concrete test inputs from these constraints. In contrast, concolic testing tightly couples both concrete and symbolic executions: they run simultaneously, and each gets feedback from the other.
Concolic appears to be the future for automated test input generation with numerous papers on this topic showing variations. Concolic is a catchier name than Directed Assisted Random Testing (DART) or Dynamic Symbolic Execution (DSE).
An intern in our group, Xiao Qu, presented her paper Configuration-Aware Regression Testing: An Empirical Study of Sampling and Prioritization which use combinatorial interaction testing techniques to model and generate configuration samples for use in regression testing. She has been helping to introduce combinatorial testing for parameters into Spec Explorer. Configuration testing is my favorite use of PICT. There result:
By running the same set of test cases using additional configurations, we were able to uncover faults previously undetected.
Several object oriented software measures for Java can be measured with varying outputs for the same measure from different tools. Moral of Comparing Software Metrics Tools is you can’t trust a single tool as it may mis-guide you if you don’t fully and completely understand deep, detailed little assumptions.
Researchers hunger for real industrial data. AFID: An Automated Fault Identification Tool was written to automatically gather test cases and bugs from evolving software for empirical study by researchers. Many researchers compared based on the 7 programs in the standard “Siemens Benchmark Suite”. Small programs listed in numerous papers at this and other conferences.
At the follow on State-space Exploration for Automated Testing (SSEAT) workshop
I spoke with Myra Cohen who works on combinatorics and would like information about typical application of pair-wise testing in industry. What type of domain to you use pair-wise testing in? (e.g. API, GUI, Configurations), How many factors do you consider in a single pair-wise problem? How many values for the factor with the most values? How many constraints? My own experience using it for configuration testing of Indigo configurations was about 20 factors with no more than 10 values per factor.
I found one article in the poster session of great interest. I’m a big believer in test set minimization for regression testing. Most methods up to now are heuristic (e.g. greedy algorithm). MINTS: A General Framework and Tool for Supporting Test-suite Minimization (and paper) attempts to find the optimal solution (using constraint solving of binary Integer linear programming) including multiple weighted and prioritized criteria like code coverage, time to execute, defect finding history. Their data indicates
MINTS was able to generate optimal test suites in only a few seconds, a time comparable to that of heuristic (i.e., suboptimal) approaches.
For small test suites (the Siemens 7) it works as fast as the heuristic techniques.