6CCS3SMT/7CCSMASE
Tutorial 10 (Week 11) Solution
1. Weak mutation “kills” a mutant when a difference between the state of the original and mutated program is detected, whether or not this difference would eventually show up in program output. Strong mutation succeeds in “killing” a mutation only when program output is different. Therefore, a test case that kills this mutant under weak mutation but not under strong mutation is one that causes some differences in internal state but eventually produces the same output as the original program.
In the example program, mutation of the first of three if statements has made it identical in effect to the third if statement. A test case that should return True from the third if statement, but instead returns True from the first if statement, will therefore kill the mutant under weak mutation but not under strong mutation. If s1 is “cat” and s2 is “caat” we will obtain this outcome, because we cannot observe from the output that the program correctly judged that “cat” is within edit distance 1 of “caat” by incorrectly concluding that “cat” is transformed to “caat” by substituting one letter for another.
2.
• Invalid: Mutants that are not syntactically correct. For example, we can apply the operator sdl (statement deletion) to remove the “if” statement in line 18, resulting in a syntactically incorrect program.
• Valid but not useful: Mutants whose behavior differs in a large fraction of possible executions, so that they are killed almost immediately even with very weak test suites. For example, the mutant obtained by changing the operator == to != in line 7 will result in immediate return of the procedure, which will be detected by almost all test cases.
• Valid and equivalent: Mutants that cannot be distinguished from the original program by any execution. For example, we can replace “+1” with “++” in line 18, thus advancing the pointer. This, however, will not affect the execution at all.
• Valid and non-equivalent: Useful mutants, which are valid and not equivalent to the original program, have behaviors that differ from the original program in only a small fraction of possible executions. For example, the mutant obtained by removing the +1 in one of the if statements in lines 27-31 will fail on a test case that checks this type of similarity.
3. Two possible approaches: finger- prints and on-line comparison.
– The fingerprint approach stores a value computed from the output, rather than the output itself.
– An on-line comparison oracle does not store the output of the trusted alternate version, but rather runs the trusted version in tandem with the version under test, incrementally comparing output as it is produced. All but a small portion of the output can be discarded as soon as it has been used in the comparison.
4. The “test first” approach associated with extreme programming aligns test oracles with program specification by using test cases (which include pass/fail criteria) as a form of specification. In this sense it meets our stated goal of unifying specification and test oracle.
i. cost has been moved from a gap between specification and oracle to a gap between the specification we would prefer to state and the specification we can express through test cases.
5. Advantages of off-line checks may include:
a. A computationally expensive analysis may be performed with
limited perturbation of the running program.
b. The same event log may be used for several different checks.
c. Because logs may be retained between runs, it is possible to
designate test case that involves more than one execution, e.g.,
comparing logs produced under different circumstances.
d. Some problems of online self-checks(e.g., retention of “before”
values) become trivial when analyzing a log file.
– Advantages of on-line self-checks over off-line log analysis:
o Direct access to program state is sometimes much simpler and more efficient than recording an adequate log record
o Production of a log record is a side effect of program execution. Log recording may itself cause program failure.
o If logging is activated for testing and disabled in the production environment, we have introduced a potentially significant difference.