SOFT2201/COMP9201: Software Construction and Design 1
Testing
Dr. Xi Wu
School of Computer Science
The University of Sydney
Page 1
Copyright warning
COMMONWEALTH OF AUSTRALIA Copyright Regulations 1969 WARNING
This material has been reproduced and communicated to you by or on behalf of the University of Sydney
pursuant to Part VB of the Copyright Act 1968 (the Act).
The material in this communication may be subject to copyright under the Act. Any further copying or communication of this material by you may be the subject of copyright protection under
the Act.
Do not remove this notice.
The University of Sydney Page 2
Agenda
– Software Testing – Unit Testing
The University of Sydney Page 3
Software Engineering Body of Knowledge
– Software Requirements
– Software Design / Modelling
– Software Construction
– Software Testing
– Software Maintenance
– Software Configuration Management
– Software Engineering Process
– Software Engineering Tools and Methods – Software Quality
Software Engineering Body of Knowledge (SWEBOK) https://www.computer.org/web/swebok/
The University of Sydney
Page 4
Why Software Testing?
The University of Sydney Page 5
• • •
Societies, businesses and governments depend on SW
• Power, Telecommunication, Education, Government, Transport, Finance, Health
• Work automation, communication, control of complex systems Large software economies in developed countries
• IT application development expenditure in the US more than $250bn/year1
• Total value added GDP in the US2: $1.07 trillion
Emerging challenges
• Security, robustness, human user-interface, and new computational platforms
Software is Everywhere!
1 Chaos Report, Standish group Report, 2014 2 softwareimpact.bsa.org
The University of Sydney
Page 6
Software Failure – Ariane 5 Disaster5
What happened?
• European large rocket – 10 years development, ~$7 billion
• Unmanaged software exception resulted from a data conversion
from 64-bit floating point to a 16-bit signed integer
• Backup processor failed straight after using the same software
• Exploded 37 seconds after lift-off
Why did it happen?
• Design error, incorrect analysis of changing requirements, inadequate validation and verification, testing and reviews, ineffective development processes and management
5 http://iansommerville.com/software-engineering-book/files/2014/07/Bashar-Ariane5.pdf The University of Sydney
Page 7
Why Software Testing?
– Software development and maintenance costs • Financial burden of failure
– Total costs of imperfect software testing for the US in 2002 was AUD86 billion*
– One third of the cost could be eliminated by ‘easily’ improved software testing
– Need to develop functional, robust and reliable software – Human/social factor
• Dependence on software in many aspects of their lives – Small software errors can lead to disasters
* NIST study 2002 The University of Sydney
Page 8
What is Software Testing?
The University of Sydney Page 9
Software testing
– Software process to
l demonstrate that software meets its requirements (validation
testing)
l Find incorrect or undesired behaviour caused by defects (defect testing)
l e.g. crashes, incorrect results, data corruption
– Part of the software Verification and Validation (V&V) process
The University of Sydney Page 10
Types of testing
– Unit testing
l Verify functionality of software components independent of the
whole system
– Integration testing
l Verify interactions between software components
– System Testing
l Verify functionality and behaviour of the entire software system l Includes security, performance, reliability, and external interfaces
– Acceptance testing
l Verify desired acceptance criteria are met from the users point of
view
The University of Sydney Page 11
Software Verification and Validation
– Software testing is part of software V&V
– The goal of V&V is to establish confidence that the software is “fit
for purpose”
– Software Validation
–
l Are we building the right product?
l Ensures that the software meets customer expectations Software Verification
l Are we building the product correctly
l Ensure the software meets the stated functional and non-functional
requirements
The University of Sydney Page 12
Black box or White box
– Black box testing
l The internals of the software system is unknown
l Only inputs to the system are controlled, and outputs from the system are measured
l Specification-based testing
l May be the only choice to test libraries without access to internal
– White box testing
l The internals of the software system are known l The internal structure is tested directly
l Unit, integration, system testing
The University of Sydney Page 13
Types of testing
Functional testing
l Unit
l Integration
l System
l Regression
l Interface
l User Acceptance l Configuration
l Sanity
Non-functional testing
l Performance l Stress
l Reliability
l Usability
l Load
l Security
The University of Sydney
Page 14
Who should design and run tests?
The University of Sydney Page 15
Test engineer
– Independent testers
l Independent testers do not have the same biases as the developer l Different assumptions
l Domain specific knowledge of testing
– Developer
l Understands the system being developed
l Domain specific knowledge of the system
l Cheaper
l Can finish writing the system faster without tests since they won’t make mistakes
The University of Sydney Page 16
Unit Testing
The University of Sydney Page 17
Unit testing
– The process of verifying functionality of software components independently
l Unit can mean methods, functions, or object classes l Verify that each unit behaves as expected
l Carried out by developers and software testers
l First level of testing
The University of Sydney
Page 18
Why unit testing
– Maintain and change code at a smaller scale
– Discover defects early and fix it when its cheaper – Simplify integration testing
– Simplify debugging
– Code reusability
The University of Sydney
Page 19
How to do the unit test
– Identify the unit that you want to test
– Design test case
– Prepare test data (input and expected output) – Run test case using test data
– Compare result to expected output
– Prepare test reports
The University of Sydney
Page 20
Designing test cases
– Effective test cases show:
l The unit does what it is supposed to do
l Reveal defects, if they exist (does not do what it is not supposed to do)
– Design two types of test case
l Test normal operation of the unit
l Test abnormal operation (common problems)
The University of Sydney Page 21
Designing test cases – techniques
– Partition testing (equivalence partitioning)
l Identify groups of tests that have common characteristics
l From each group, choose specific tests
l Use program specifications, documentation, and experience
– Guideline-based testing
l Use testing guidelines based on previous experience of the kinds
of errors made
l Depends on the existence of previous experience (developer/product)
The University of Sydney Page 22
Equivalence partitioning
– Groups of test input that have common characteristics l Positive numbers
l Negative numbers
l Boundaries
– Program is expected to behave in a comparable way for all members of a group
l Control flow should be similar for all members – Choose test cases from each partition
The University of Sydney
Page 23
Test case selection
– Understanding developers thinking
l Easy to focus on typical values of input
l Common case, and what was asked for l Easy to overlook a typical value of input
l Users, other developers, new features, all have different expectations
– Choose test cases that are
l On boundaries of partitions
l In ‘midpoint’ of partitions
l NB: Boundaries may be unclear (-1, 0, 1, 0.5)
The University of Sydney
Page 24
Test cases – identifying partitions
– Consider this specification:
l The program accepts 4 to 8 inputs that are five digit integers
greater than 10,000
– Identify the input partitions and possible test inputs
?
The University of Sydney
Page 25
Test cases – identifying partitions
– Consider this specification:
l The program accepts 4 to 8 inputs that are five digit integers
greater than 10,000
– Identify the input partitions and possible test inputs
– How many values l <4, 4-8, >8
– How many digits
l < 5, 5, > 5, non-digits
– How big
l > 10000
l etc.
The University of Sydney
Page 26
Test case selection guidelines
– Knowledge of types of test case effective for finding errors – If testing sequences, arrays, lists:
l Single value
l Different sequences of different sizes
l Test partition boundaries (first, middle, last) l Consider order of values
The University of Sydney
Page 27
Test case selection guidelines
– Choose inputs that force the system to generate all expected error messages
– Design inputs that cause buffer overflows
– Repeat input
– Force invalid outputs to be generated
– Force computations results that are too large or too small
– Domain specific knowledge!
The University of Sydney
Page 28
Acquiring domain specific knowledge
– Be an expert on the system, or type of system – or,
– Make many mistakes
– Identify mistakes
– Write tests to identify mistakes
– Fix mistakes
– Be an expert on the system, or type of system
– Regression testing!
The University of Sydney
Page 29
Regression testing
– If a defect is identified in software it can be fixed l How did it get there?
l How do you stop it happening again?
The University of Sydney Page 30
Regression testing
– Regression: a defect that has been fixed before, happens again l Human error
l Version control problems
l Specific case is fixed, but the general case remains
l Convergent evolution
The University of Sydney Page 31
Regression testing
– As defects in software are fixed, tests are written that demonstrate that the software is fixed (at least in regard to that particular defect)
l Tests can be re-run with each change in the software system l Regression testing
l Frequently automated
The University of Sydney
Page 32
When to test
The University of Sydney Page 33
When to test
– Continuously
– When the software system changes l Code changes
l Design changes
l Infrastructure changes
l At regular intervals in case the above missed a change
The University of Sydney
Page 34
How to test
The University of Sydney Page 35
How to test
– Write testable code
public static void main(String[] args) { // All the code
// All
}
The University of Sydney
Page 36
How to test
– Write testable code
public static void main(String[] args) { Application app = new Application();
}
Public class Application { Application() {
// All the code }
}
The University of Sydney
Page 37
How to test
– Write testable code
public static void main(String[] args) { Application app = new Application();
app.doEverything(); }
public class Application { Application() {
// Construct the application }
public void doEverything() { // All the code
} }
The University of Sydney
Page 38
How to test
– Write testable code
public class Application { Application() {
// Construct the application }
public void doEverything() { // Most of the code doSomeOfTheThings();
}
public void doSomeOfTheThings() {
// Some of the code }
}
The University of Sydney
Page 39
How to test
– Write testable code
public class Application { Application() {
// Construct the application }
public void doEverything() {
// Some code
Thing = doSomeOfTheThings(thing); // More code
}
public BigThing doSomeOfTheThings(LittleThing littleThing) {
// Some of the code that deals with LittleThings }
}
The University of Sydney
Page 40
How to test
– Write testable code
public class Application { // …
public void doEverything(LittleThingFactory littleThingFactory) { LittleThing firstThing= littleThingFactory.makeThing(); LitteThing secondThing = doStuff(firstThing); doStuff(secondThing);
doStuffWithTwoThings(firstThing, secondThing); doSomeOfTheThings(thing);
// …
}
protected BigThing doSomeOfTheThings(LittleThing littleThing) {
// Some of the code that deals with LittleThings }
// …
}
The University of Sydney
Page 41
Unit Testing in Java
The University of Sydney Page 42
Unit testing terminology
– Unit test
l A piece of code written by a developer that executes a specific functionality in the code under test and asserts a certain behaviour or state as correct
l Small unit of code (method/class)
l External dependencies are removed
l (Mocking) – Test fixture
l Testing context
l Shared test data
l Methods for setting up test data
The University of Sydney
Page 43
Unit testing frameworks for Java
– JUnit
– TestNG
– Jtest
– Many others
– Custom, developer-written, tests
The University of Sydney
Page 44
JUnit
import static org.junit.Assert.assertEquals; import org.junit.Test;
import mypackage.Calculator;
class CalculatorTest { @Test
void addition() {
Calculator calculator = new Calculator(); assertEquals(2, calculator.add(1, 1));
} }
The University of Sydney
Page 45
JUnit
import static org.junit.Assert.assertEquals; import org.junit.Test;
import mypackage.Calculator;
class CalculatorTest { @Test
void addition() {
Calculator calculator = new Calculator(); assertEquals(2, calculator.add(1, 1));
} }
The University of Sydney
Page 46
JUnit
import static org.junit.Assert.assertEquals; import org.junit.Test;
import mypackage.Calculator;
class CalculatorTest { @Test
void addition() {
Calculator calculator = new Calculator(); assertEquals(2, calculator.add(1, 1));
} }
The University of Sydney
Page 47
JUnit
import static org.junit.Assert.assertEquals; import org.junit.Test;
import mypackage.Calculator;
class CalculatorTest { @Test
void addition() {
Calculator calculator = new Calculator(); assertEquals(2, calculator.add(1, 1));
} }
The University of Sydney
Page 48
JUnit
import static org.junit.Assert.assertEquals; import org.junit.Test;
import mypackage.Calculator;
class CalculatorTest { @Test
void addition() {
Calculator calculator = new Calculator(); int expected = 2;
int actual = calculator.add(1, 1); assertEquals(expected, actual);
} }
The University of Sydney
Page 49
JUnit
import static org.junit.Assert.assertEquals; import org.junit.Test;
import mypackage.Calculator;
class CalculatorTest { @Test
void addition() {
Calculator calculator = new Calculator(); int expected = 2;
int actual = calculator.add(1, 1); assertEquals(expected, actual);
} }
The University of Sydney
Page 50
JUnit
import static org.junit.Assert.assertEquals; import org.junit.Test;
import mypackage.Calculator;
class CalculatorTest { @Test
void addition() {
Calculator calculator = new Calculator(); int expected = 2;
int actual = calculator.add(1, 1); assertEquals(expected, actual);
} }
The University of Sydney
Page 51
JUnit constructs
– JUnit test
l A method only used for testing
– Test suite
l A set of test classes to be executed together
– Test annotations
l Define test methods (e.g., @Test, @Before) l JUnit uses the annotations to build the tests
– Assertion methods
l Check expected result is the actual result l e.g., assertEquals, assertTrue, assertSame
The University of Sydney
Page 52
JUnit annotations
– @Test
l Identifies a test method
– @Before
l Execute before each test
– @After
l Execute after each test
– @BeforeClass
l Execute once, before all tests in this class
– @AfterClass
l Execute once, after all tests in this class
The University of Sydney
Page 53
JUnit assertions
– assertEquals(expected, actual)
import static org.junit.Assert.assertEquals; import org.junit.Test;
import mypackage.Calculator;
class CalculatorTest { @Test
void addition() {
Calculator calculator = new Calculator(); int expected = 2;
int actual = calculator.add(1, 1); assertEquals(expected, actual);
} }
The University of Sydney
Page 54
JUnit assertions
– assertEquals(message, expected, actual)
import static org.junit.Assert.assertEquals; import org.junit.Test;
import mypackage.Calculator;
class CalculatorTest { @Test
void addition() {
Calculator calculator = new Calculator();
int expected = 2;
int actual = calculator.add(1, 1);
assertEquals(“Expected value != actual”, expected, actual);
} }
The University of Sydney
Page 55
JUnit assertions
– assertTrue
import static org.junit.Assert.assertEquals; import org.junit.Test;
import mypackage.Calculator;
class CalculatorTest { @Test
void addition() {
Calculator calculator = new Calculator(); assertTrue(2 == calculator.add(1, 1));
} }
The University of Sydney
Page 56
JUnit assertions
– assertTrue
import static org.junit.Assert.assertEquals; import org.junit.Test;
import mypackage.Calculator;
class CalculatorTest { @Test
void addition() {
Calculator calculator = new Calculator(); assertTrue(“Can’t do 1 + 1 :(“, 2 == calculator.add(1, 1));
} }
The University of Sydney
Page 57
JUnit assertions
import …
class CalculatorTest { Calculator calculator @Before
void setup() {
calculator = new Calculator(); }
@Test
void additionBothPositive() { assertEquals(2, calculator.add(1, 1)); assertEquals(5, calculator.add(4, 1));
assertEquals(5, calculator.add(2, 3)); }
…
The University of Sydney
Page 58
Tasks for Week 7
• Submit weekly exercise on canvas before 23.59pm Sunday
• Submit assignment 2 on canvas and Ed platform before its due.
– All assignments are individual assignments
– Please note that: work must be done individually without consulting someone else’s solutions in accordance with the University’s “Academic Dishonesty and Plagiarism” policies
• Attend Helpdesk session if you have any questions/difficulties on implementation perspective
The University of Sydney Page 59
What are we going to learn next week?
• Code Review
The University of Sydney Page 60