COMP3053.SQA Coursework 03 TDD to create ART Harness; MT (30%)
Learning Aims:
This coursework is designed to give you experience writing high-quality object oriented code using Test-Driven Development (TDD); to give you a chance to implement some Adaptive Random Testing algorithms; and to get you thinking about testing when faced with the Oracle Problem — using Metamorphic Testing (MT).
Background:
You have been working at the prestigious software development company TopQualitySoftware Inc. Since your arrival there, you have been promoted to “technical lead,” and have been receiving increasing amounts of responsibility.
TopQualitySoftware’s Chief Technology Officer (CTO), Ms Big, has spoken with you a number of times about the importance of software quality and appropriate documentation. As part of this, she has decided to give you a new project:
“I want you, using TDD, to build me a test harness. The test harness will work from the command line. It will take a string of arguments, using flags if necessary, and then perform random testing or adaptive random testing on a specified program. For now, all the programs the harness will be used on will have reasonably simple, numeric, parameters.”
Your friend, Mr Nice, was very excited to hear about this new project. Because he is a QA expert, and an advocate of not only Random Testing, but also Adaptive Random Testing, he had a lot of very helpful advice for you:
“This is a great idea. I think you should research a little about pseudo-random number generators (PRNGs), especially how to use them in Java, and the importance of the seed. It is probably a good idea to research how to scale the output of Java’s PRNG to different ranges, without causing a negative impact on the quality of the pseudo-random number sequence. Think carefully about what you’ll record in any data file your harness will generate.”
You are familiar with the DART, RRT, and MART versions of ART, and know the various control parameters each would require. Mr Nice, is also very enthusiastic about this project. He tells you that it is really important that you get this project right. Perhaps this will be your ticket to becoming the vice-president! As an expert in ART he has some more very helpful advice for you:
“In addition to all the things you need to do for an ordinary random testing harness, you should choose a version of ART that will work well. Personally, I like the Restriction-based ART implementation (RART, or RRT), but you can decide which one to implement yourself. Remember that each ART implementation has different parameters that will need to be passed as arguments on the command line! You’ll probably need to include some additional information in the data file, too.”
You are very eager to make sure that this project is a success, and you have been studying Metamorphic Testing (MT) to see if there is any way to test the harness once completed.
Basic/Outline Harness Interaction:
After some careful consideration, you come up with an outline plan for how a simple random testing (not ART) test harness could work. From the command line, it would take a sequence of arguments introduced by flags. You have decided that the flags (and their arguments) needed are as follows:
-p
-s
-n
-a
-r
Assuming the harness is called TestHarness, and that both the program under test and the oracle are in the same folder (and called PUT and Oracle, respectively), then a typical call to the harness might be:
%TestHarness -p PUT -o Oracle -s 7 -n 10000 -a 2 -r 0.0 1.0 -1.0 0.0
Which would mean that TestHarness would generate 10000 test cases using the PRNG with the seed 7; each test case would be a 2-tuple with the first number in the range [0.0, 1.0), and the second number in the range [-1.0, 0.0); these test cases would be applied to the programs PUT and Oracle, with the outputs from both compared to see if they are the same or not … if the output is not the same, that would mean that the harness had found a failure/fault/bug.
After some further consideration, you have decided that, in addition to those already specified in the ordinary test harness, the following additional flags (and their arguments) would be needed if you were to implement ALL four of the ART algorithms that you know:
-m [RT|DART|MART|RRT]
-k
-P
-R
Of course, you only need to implement one version of ART!
Again assuming the harness is called TestHarness, and that both the program under test and the oracle are in the same folder (and called PUT and Oracle, respectively), then a call to the harness to run random testing (RT) might be:
%TestHarness –m RT –p PUT –o Oracle –s 7 –n 10000 –a 2 –r 0.0 1.0 -1.0 0.0
Which would mean that TestHarness would generate 10000 test cases using the PRNG with the seed 7; each test case would be a 2-tuple with the first number in the range [0.0, 1.0), and the second number in the range [-1.0, 0.0); these test cases would be applied to the programs PUT and Oracle, with the outputs from both compared to see if they are the same or not … if the output is not the same, that would mean that the harness had found a failure/fault/bug.
A similar call to the harness, but this time to use the DART implementation of ART, with a candidate set size of 10, might be:
%TestHarness –m DART –k 10 –p PUT –o Oracle –s 7 –n 10000 –a 2 –r 0.0 1.0 -1.0 0.0
You have also decided that when the test harness is run, the outputs will be put into a newly created file called “TestResults.dat”. You need to think very carefully about what should go into this file, and what should happen if the file already exists.
Finally, you should be careful to build the harness in such a way that it reacts appropriately to incorrect input. Error messages or exceptions may be a good idea.
Metamorphic Testing:
In addition to the TDD-supported unit testing and regression testing that you should perform, you want to use your knowledge of Metamorphic Testing (MT) to further test the harness. You should identify five (5) metamorphic relations (MRs) for the harness.
Deliverables/Submissions: § Java Code:
You should use Java to produce both the unit tests (for TDD) and the production (main) code. All the Java code (including any additional libraries) should be put into a zip file (not a package).
§ (Brief) User Manual
You should prepare a very brief User Manual, no more than about 250 words. This should explain clearly to any users how to set up, prepare, and use the test harness.
§ Code Report
You should write a report about the code used in the test harness. This is a document that would go to a maintenance team, so it should include details like algorithms used, external dependencies, known issues, etc. This should not be more than about 500 words.
§ QA Report detailing testing
You should write a report about the QA process followed to produce the final test harness. This should include all the steps taken to ensure the quality of the harness, including a listing of (at least) five MRs used for Metamorphic Testing. This should not be more than about 500 words.
Submission:
All four deliverables should be put into a .zip file and submitted through Moodle, before the deadline.
Assessment:
Submissions will be manually marked and graded, with the weighting of the parts as follows.
– 60% for the Java code
– 10% for User Manual
– 15% for Code Report
– 15% for QA Report
Deadline:
The deadline will be Friday, 4 pm, December 14th, 2018.