COMP2511
Test Design
“Testing shows the presence, not the absence of bugs.”
—Edsger W. Dijkstra
Software Testing: Test-Driven Development (TDD)
v Every iteration in the software development process must be preceded with a plan to properly
verify (test) that the developed software meets the requirements (i.e., post conditions).
v A software developer must not create a software artifact and later think of how to test it!
v Testing is an essential integral part of developing a software solution. It must not be considered
as an afterthought !
v Incremental development must be tested against test suites, during every iteration. Every code
modification and/or refactoring must be followed by a proper testing, using the predefined test
suites.
v Testing must be setup, based on the requirement specifications, before you start implementing
your solution.
COMP2511: Test Design 2
Requirements/
Specifications
Interfaces
(method signatures,
pre/post conditions)
Classes
(method
implementations)
Test Sets
(Unit testing,
Integration testing, etc.)
Software Testing: Input Space Coverage
v Testing must not be conducted haphazardly by trial-and-error.
v Testing must be conducted systematically, with a well thought out testing plan.
v The aim should be to consider a possible input space and cover it as much as possible.
v Often this is achieved by dividing the input space into “equivalence groups” and selecting a
representative input from each equivalence group. Here, the assumption is: from the same
equivalence group, a program is expected to behave similarly on each input.
v For example, for the method boolean isSorted (list);
possible input cases to consider,
o Input list: 34, 12, 15, 21, 5, 21. [random all positive]
o Input list: -13, -12, -77, -60, -55. [random all negative]
o Input list: 10, -11, 17, 31, 50, 42. [random mix positive/negative]
o Input list: 22, 22, 22, 22, 22, 22. [all same]
o Input list: 3, 7, 34, 41, 53, 99. [increasing order]
o Input list: 99, 45, 0, -10, -34, -89. [decreasing order]
o Etc.
o Etc.
COMP2511: Test Design 3
Software Testing: Input Space Coverage
v Consider borderline cases, often called boundary testing.
v For example, for the method String getGrade( marks );
possible input cases to consider,
o 0
o 50
o 65
o 75
o 85
o 100
v For multiple input values, consider possible input combinations, prioritise them and consider as
many as possible, given the available time and resources. Again, divide possible combinations
into homogenous subsets and select representative combinations.
COMP2511: Test Design 4
Software Testing: Code Coverage
v Code coverage is a useful metric that can help you assess the quality of your test suite.
v Code coverage measures the degree to which a software is verified by a test suite, by
determining the number of lines of code that is successfully validated by the test suite.
v The common metrics in most coverage reports include:
• Function coverage: how many of the functions defined have been called.
• Statement coverage: how many of the statements in the program have been executed.
• Branches coverage: how many of the branches of the control structures (if statements for
instance) have been executed.
• Condition coverage: how many of the boolean sub-expressions have been tested for a true
and a false value.
• Line coverage: how many of lines of source code have been tested.
v For more, see https://www.atlassian.com/continuous-delivery/software-testing/code-coverage
COMP2511: Test Design 5
Randomness in Software Testing and Simulation
Randomness is also useful!
v Software Testing:
• random data is often seen as unbiased data
• gives average performance (e.g. in sorting algorithms)
• stress test components by bombarding them with random data
v Software Simulation:
• generating random behaviours/movements.
For example, may want players/enemies to move in a random pattern.
Possible approach: randomly generate a number between 0 to 3,
• 0 means front movement, 1 means left movement,
2 means back movement, 3 means right movement.
• the layout of a dungeon may be randomly generated
• may want to introduce unpredictability
COMP2511: Test Design 6
Random Numbers
v How can a computer pick a number at random?
Ø it cannot !
v Software can only produce pseudo random numbers.
Ø a pseudo random number is one that is predictable!
• (although it may appear unpredictable)
v Implementation may deviate from expected theoretical behaviour.
COMP2511: Test Design 7
Generating Random Numbers in Java
Using random class,
v Need to import the class java.util.Random
v Option-1: Creates a new random number generator.
vRandom rand = new Random();
v Option-2: Creates a new random number generator using a single long seed.
Ø Important: Every time you run a program with the same seed, you get exactly the same sequence of
‘random’ numbers.
Random rand = new Random(long seed);
v To vary the output, we can give the random seeder a starting point that varies with time.
For example, a starting point (seed) is the current time.
v Go to the API for more information at
https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/Random.html
COMP2511: Test Design 8
https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/Random.html
Basic Test Template
COMP2511: Test Design 9
1) Set up Precondition (i.e. @BeforeEach, etc.)
2) Act (call the method)
3) Verify Post condition (i.e. @AfterEach, Asserts, etc.)
• Normally, each test should run independently, order of execution should not be important.
• However, if required, you can order execution of the tests using @TestMethodOrder
Avoid Repetition in Test Suites: Parameterized Tests
• JUnit offers Parameterized Tests.
• Parameterized test executes the same test over and over again using different input values and
tests output against the corresponding expected results.
• A data source can be used to retrieve data for input values and expected results.
• The @Before annotation can be used if you want to execute some statement such as
preconditions before each test case.
• The @After annotation can be used if you want to execute some statements after each Test Case
for e.g. resetting variables, deleting temporary files, variables, etc.
• For more information, see JUnit 5 tutorial .
COMP2511: Test Design 10
https://www.vogella.com/tutorials/JUnit/article.html
Parameterized Test: An Example
COMP2511: Test Design 11
For more information, see JUnit 5 tutorial at
https://www.vogella.com/tutorials/JUnit/article.html.
https://www.vogella.com/tutorials/JUnit/article.html
Avoid Repetition in Test Suites: Dynamic Tests
COMP2511: Test Design 12
For more information, see JUnit 5 tutorial at
https://www.vogella.com/tutorials/JUnit/article.html.
• JUnit also offer
Dynamic Tests.
• For more information,
see JUnit 5 tutorial .
https://www.vogella.com/tutorials/JUnit/article.html
https://www.vogella.com/tutorials/JUnit/article.html
Software Testing: Summary
• Always follow Test Driven Development.
• Software Testing is hard!
• Not possible to completely test a nontrivial software system, given the limited available
resources.
• We assume that a selected representative test cases capture system behavior of test cases not
considered.
COMP2511: Test Design 13
End
COMP2511: Test Design 14