Microsoft Word – COMP3053.SQA-CW03-Specification.docx
COMP3053.SQA Coursework 03
TDD to create ART Harness; MT (30%)
Learning Aims:
This coursework is designed to give you experience writing high-quality object oriented code using
Test-Driven Development (TDD); to give you a chance to implement some Adaptive Random Testing
algorithms; and to get you thinking about testing when faced with the Oracle Problem – using
Metamorphic Testing (MT).
Background:
You have been working at the prestigious software development company TopQualitySoftware Inc.
Since your arrival there, you have been promoted to “technical lead,” and have been receiving
increasing amounts of responsibility.
TopQualitySoftware’s Chief Technology Officer (CTO), Ms Big, has spoken with you a number of
times about the importance of software quality and appropriate documentation. As part of this, she
has decided to give you a new project:
“I want you, using TDD, to build me a test harness. The test harness will work from the
command line. It will take a string of arguments, using flags if necessary, and then
perform random testing or adaptive random testing on a specified program. For now, all
the programs the harness will be used on will have reasonably simple, numeric,
parameters.”
Your friend, Mr Nice, was very excited to hear about this new project. Because he is a QA expert,
and an advocate of not only Random Testing, but also Adaptive Random Testing, he had a lot of
very helpful advice for you:
“This is a great idea. I think you should research a little about pseudo-random number
generators (PRNGs), especially how to use them in Java, and the importance of the seed.
It is probably a good idea to research how to scale the output of Java’s PRNG to different
ranges, without causing a negative impact on the quality of the pseudo-random number
sequence. Think carefully about what you’ll record in any data file your harness will
generate.”
You will soon be familiar with the DART, RRT, and MART versions of ART, and know the various
control parameters each would require. Mr Nice, is also very enthusiastic about this project. He
tells you that it is really important that you get this project right. Perhaps this will be your ticket to
becoming the vice-president! As an expert in ART he has some more very helpful advice for you:
“In addition to all the things you need to do for an ordinary random testing harness, you
should choose a version of ART that will work well. Personally, I like the Restriction-based
ART implementation (RART, or RRT), but you can decide which one to implement
yourself. Remember that each ART implementation has different parameters that will
need to be passed as arguments on the command line! You’ll probably need to include
some additional information in the data file, too.”
You are very eager to make sure that this project is a success, and you have been studying
Metamorphic Testing (MT) to see if there is any way to test the harness once completed.
何姐
何姐
Basic/Outline Harness Interaction:
After some careful consideration, you come up with an outline plan for how a simple random
testing (not ART) test harness could work. From the command line, it would take a sequence of
arguments introduced by flags. You have decided that the flags (and their arguments) needed are
as follows:
-p
-s
-n
-a
-r
Assuming the harness is called TestHarness, and that both the program under test and the
oracle are in the same folder (and called PUT and Oracle, respectively), then a typical call to the
harness might be:
%TestHarness -p PUT -o Oracle -s 7 -n 10000 -a 2 -r 0.0 1.0 -1.0 0.0
Which would mean that TestHarness would generate 10000 test cases using the PRNG with the
seed 7; each test case would be a 2-tuple with the first number in the range [0.0, 1.0), and the
second number in the range [-1.0, 0.0); these test cases would be applied to the programs PUT and
Oracle, with the outputs from both compared to see if they are the same or not … if the output is
not the same, that would mean that the harness had found evidence of a failure/fault/bug.
After some further consideration, you have decided that, in addition to those already specified in
the ordinary test harness, the following additional flags (and their arguments) would be needed if
you were to implement ALL four of the ART algorithms that you know:
-m [RT|DART|MART|RRT]
-k
-P
-R
Of course, you only need to implement one version of ART!
Again assuming the harness is called TestHarness, and that both the program under test and
the oracle are in the same folder (and called PUT and Oracle, respectively), then a call to the
harness to run random testing (RT) might be:
%TestHarness –m RT –p PUT –o Oracle –s 7 –n 10000 –a 2 –r 0.0 1.0 -1.0 0.0
Which would mean that TestHarness would generate 10000 test cases using the PRNG with the
seed 7; each test case would be a 2-tuple with the first number in the range [0.0, 1.0), and the
second number in the range [-1.0, 0.0); these test cases would be applied to the programs PUT and
Oracle, with the outputs from both compared to see if they are the same or not … if the output is
not the same, that would mean that the harness had found a failure/fault/bug.
A similar call to the harness, but this time to use the DART implementation of ART, with a candidate
set size of 10, might be:
%TestHarness –m DART –k 10 –p PUT –o Oracle –s 7 –n 10000 –a 2 –r 0.0 1.0
-1.0 0.0
You have also decided that when the test harness is run, the outputs will be put into a newly
created file called “TestResults.dat”. You need to think very carefully about what should go into this
file, and what should happen if the file already exists.
Finally, you should be careful to build the harness in such a way that it reacts appropriately to
incorrect input. Error messages or exceptions may be a good idea.
Metamorphic Testing:
In addition to the TDD-supported unit testing and regression testing that you should perform, you
want to use your knowledge of Metamorphic Testing (MT) to further test the harness. You should
identify five (5) metamorphic relations (MRs) for the harness.
Deliverables/Submissions:
§ Java Code:
You should use Java to produce both the unit tests (for TDD) and the production (main)
code. All the Java code (including any additional libraries) should be put into a zip file (not a
package).
§ (Brief) User Manual
You should prepare a very brief User Manual, no more than about 250 words. This should
explain clearly to any users how to set up, prepare, and use the test harness.
§ Code Report
You should write a report about the code used in the test harness. This is a document that
would go to a maintenance team, so it should include details like algorithms used, external
dependencies, known issues, etc. This should not be more than about 500 words.
§ QA Report detailing testing
You should write a report about the QA process followed to produce the final test harness.
This should include all the steps taken to ensure the quality of the harness, including a
listing of (at least) five MRs used for Metamorphic Testing. This should not be more than
about 500 words.
Submission:
All four deliverables should be put into a .zip file and submitted through Moodle, before the
deadline.
Assessment:
Submissions will be manually marked and graded, with the weighting of the parts as follows.
– 60% for the Java code
– 10% for User Manual
– 15% for Code Report
– 15% for QA Report
Deadline:
The deadline will be Thursday, 4 pm, December 2nd, 2021.