Software Construction & Design 1
The University of Sydney Page 1
Software Design and
Construction 2
SOFT3202 / COMP9202
Advanced Testing
Techniques (1)
School of Information Technologies
Dr. Basem Suleiman
The University of Sydney Page 2
Copyright Warning
COMMONWEALTH OF AUSTRALIA
Copyright Regulations 1969
WARNING
This material has been reproduced and communicated to
you by or on behalf of the University of Sydney
pursuant to Part VB of the Copyright Act 1968 (the
Act ).
The material in this communication may be subject
to copyright under the Act. Any further copying or
communication of this material by you may be the
subject of copyright protection under
the Act.
Do not remove this notice.
The University of Sydney Page 3
Agenda
– Testing Types
– Integration Testing, Regression Testing
– Advanced Testing Techniques
– Test doubles (Dummies, Fakes, Stubs, Spies, Mocks)
– Contract Test
– Testing Frameworks
– Mockito
The University of Sydney Page 4
Advanced Testing Types
Integration testing, regression testing
The University of Sydney Page 5
Software Components/Sub-systems
The University of Sydney Page 6
Integration Testing
– The process of verifying interactions/communications among software
components behave according to its specifications
– Independently developed (and tested) units may not behave correctly when
they interact with each other
– Activate corresponding components and run high-level tests
– Incremental integration testing vs. “Big Bang” testing
.
The University of Sydney Page 9
Interaction Errors
– Parameter interfaces
– Methods in objects have a parameter interface
– Procedural interfaces
– Objects and reusable components
– Message passing interfaces
– One component encapsulates a service from another component by passing a
message to it
– Shared memory interfaces
– Interfaces in which block of memory is shared between components (e.g.,
embedded systems)
.
The University of Sydney Page 10
Your Testing Exposed Bugs
– What would you do when your testing reveal bugs/errors?
– You fixed the discovered bugs, what should happen next?
– You extended one class with additional functionality (new feature), what
should happen next?
.
The University of Sydney Page 11
Regression Testing
– Verifies that the software behaviour has not changed by incremental
changes to the software
– Bug fixes, code extension, code enhancements
– Modern software development processes are iterative/incremental
– Changes may be introduced which may affect the validity of previous tests
– Regression testing is to verify
– Pre-tested functionality (and non-functional properties) still working as expected
– No new bugs are introduced
.
The University of Sydney Page 12
Regression Testing – Techniques
. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.460.5875&rep=rep1&type=pdf
Type Description
Retest All Re-run all the test cases in a test suit
Test Selection Re-run certain test cases based on the changes in the code
Test case
prioritization
Re-run test cases in order of its priority; high, medium, low. Priority
determined by how criticality and impact of test cases on the product
Hybrid Re-run selected test cases based on it’s priority
The University of Sydney Page 14
Test-Driven Development
The University of Sydney Page 15
Test-driven Development
– A software development approach for developing the code incrementally
along with a set of tests for that increment
– Write tests before code
– All tests must pass before starting the next increment
– Introduced in the XP agile development method
The University of Sydney Page 16
TDD Cycle
– “A rapid cycle of testing, coding, and refactoring”
– Kent Beck
– “Every few minutes, TDD provides proven code
that has been tested, designed, and coded”
– Red, Green, Refactor cycle
The University of Sydney Page 17
TDD Cycle – Think
– Think of a behavior you want your code to have (small
increment; few lines of code)
– Think of a test (few lines of code) that will fail unless the
behavior is present
– Pair programming helps
– Driver and navigator
..
The University of Sydney Page 18
TDD Cycle – Red (Run the Test)
– Write the tests – only enough code for the current
increment of behavior
– Typically less than 5 lines of code
– Code for the class behavior and its public interface
(encapsulation)
– Tests use method and class names do not exist yet
– Run your entire suite of tests and enjoy the test failure
– Results in Red progress bar (testing tools)
..
https://www.jamesshore.com/Agile-Book/test_driven_development.html
The University of Sydney Page 19
TDD Cycle – Green (Write Code)
– Write the code; just enough to get the test to pass
– Less than 5 lines
– It’s okay to hard code, you’ll refactor
– Run your tests again, and enjoy the tests passing
– Results in Green progress bar (testing tools)
..
The University of Sydney Page 20
TDD Cycle – Refactor
– Review the code and look for possible improvements
– Ask your navigator if they made any notes
– Series of very small refactorings
– 1-2 minutes each, no longer than 5 minutes
– Run the tests after each refactoring
– Should always be green (pass!)
– Test failed and no obvious answer, get back to good code
– Refactor many times, improve design
– Refactoring isn’t about changing behavior
..
The University of Sydney Page 21
TDD Cycle – Repeat
– Repeat to add new behavior, start the cycle over again
– Tiny bit of well-tested, well-designed code will be incrementally
created
– Typically, run through several cycles very quickly, then spend
more time on refactoring
– Do not skip any step, especially refactoring, to speed up!
..
The University of Sydney Page 22
TDD – Red, Green, Refactor
– Use Red, Green, Refactor cycle to impalement TDD
– Think of a code behavior, then choose a small increment and
then a test
– Write the test for the current increment code and run the entire
suit of tests – should fail (Red bar)
– Write just enough code to get the test pass and run the tests
again – should pass (Green bar)
– Review the code and look for improvements – small set of
refactoring and run the tests after each (Refactor)
..
https://www.jamesshore.com/Agile-Book/test_driven_development.html
The University of Sydney Page 23
TDD – Example
– Java class to parse HTTP query string (name-value pair)
– E.g., http://example.com/page/to/page?title=Central+Park&where=US
– Think
– “class to separate name/value pairs into a HashMap” or “class to put one
name/value pair into a HashpMap”? Why?
– Class QueryString won’t return a HashMap, but a method (valueFor(name)) to
access the name-value pairs. Shall you proceed with writing the test?
– Count() method instead to return total number of name-value pairs (more
suitable for one increment)
–
..
https://www.jamesshore.com/Agile-Book/test_driven_development.html
The University of Sydney Page 24
TDD Example – Red Bar
..
public void testOneNameValuePair() {
QueryString qs = new QueryString(“name=value”);
assertEquals(1, qs.count());
}
public class QueryString {
public QueryString(String queryString) {}
public int count() { return 0; }
}
The University of Sydney Page 25
TDD Example – Green Bar & Refractor
..
public int count() { return 1; }
Refactor
Change the QueryString name to HttpQuery() – noted for next cycle
Another test to try
The University of Sydney Page 26
TDD Example – Repeat
..
public void testNoNameValuePairs() {
QueryString qs = new QueryString(“”);
assertEquals(0, qs.count());
}
Thinking
• Remove the hard-coded line but not time yet to deal with multiple query string
• Testing an empty string would require coding the count() properly
Emerging thoughts (noted for later cycles)
• Test the case of a null argument to the QueryString constructor
• Deal with the tests duplication tests that needed refactoring
The University of Sydney Page 27
TDD Example – Green & Refactor
..
public class QueryString {
private String _query
public QueryString(string queryString) {
_query = queryString;
}
public int count() {
if (“”.equals(_query)) return 0;
else return 1;
}
}
Refactor (notes):
• Rename QuerySting
• testNull()
• Refactor duplicate tests
The University of Sydney Page 28
TDD Example – testNull()
..
public void testNull() {
try {
QueryString qs = new QueryString(null);
fail(“Should throw exception”);
}
catch (NullPointerException e) {
// expected
}
}
• Test the case when the query string is null
• Red Bar – think of the behavior when the value is null
• Through an exception (Null is illegal) – simple design
public QueryString(String queryString) {
if (queryString == null) throw new
NullPointerException();
_query = queryString;
}
The University of Sydney Page 29
TDD Example – valueFor()
..
public void testOneNameValuePair() {
QueryString qs = new
QueryString(“name=value”);
assertEquals(1, qs.count());
assertEquals(“value”,
qs.valueFor(“name”));
}
• Implement valueFor() method to return the associated value give a name/value pair
• Emerging thoughts: test for a name doesn’t exist (noted)
public String valueFor(String name) {
String[] nameAndValue =
_query.split(“=”);
return nameAndValue[1];
}
The University of Sydney Page 30
TDD Example – Repeat
..
• Code passed the tests, but it was incomplete
• Multiple name/value pairs …
• Repeat …
The University of Sydney Page 31
TDD – Benefits
– Help developers to understand the requirements and write better
code
– Simplify debugging
– Easier to find and fix mistakes in small code chunks
– Reduce cost of regression testing
– Improved design and code quality
– Research shows TDD substantially reduces the incidence of defects
– Reuse tests as the software grow, and use it as documentation
The University of Sydney Page 32
Refactoring
– “Refactoring is the process of changing the design of your code without
changing its behavior” – Kent Beck
– Change the how not the what
– Refactoring is reversible!
– Analyze the design of existing code and improve it
– Code improvements can be identified with code smells
The University of Sydney Page 33
How to Refactor
– Refactor constantly in a series of small transformations
– Learn from in-depth catalog of refactoring
– Refactor intuitively through learning the mindset behind refactoring
– Learn how to refactor manually
– Development frameworks/tools can help automating some refactoring
– Use continuous integration practices and automation tools
– Version control system, build and test automation, IDEs
The University of Sydney Page 34
Test Double
The University of Sydney Page 35
Movie – “Stunt Double”
https://i.ytimg.com/vi/xm7kzxXHF38/maxresdefault.jpg
http://cdn.kickvick.com/wp-content/uploads/2014/07/celebrity-stunt-doubles.jpg
https://amp.businessinsider.com/images/525328ce6bb3f78e7afdcbb2-750-563.jpg
https://i.ytimg.com/vi/xm7kzxXHF38/maxresdefault.jpg
http://cdn.kickvick.com/wp-content/uploads/2014/07/celebrity-stunt-doubles.jpg
https://amp.businessinsider.com/images/525328ce6bb3f78e7afdcbb2-750-563.jpg
The University of Sydney Page 36
Test Double
– “A test double is an object that can stand in for a real object in a test, similar
to how a stunt double stands in for an actor in a movie” – Google Testing Blog
– Includes stubs, mocks and fakes
– Commonly referred to as “mocks”, but they have different uses!
– Why test double?
– Dependency on components that cannot be used
– Reduce complexity, test indecently
https://testing.googleblog.com/2013/07/testing-on-toilet-know-your-test-doubles.html
https://testing.googleblog.com/2013/07/testing-on-toilet-know-your-test-doubles.html
The University of Sydney Page 38
Test Double – Types
Type Description
Dummy Pass object(s) that never actually used (to fill parameter list)
Stub Test-specific object(s) that provide indirect inputs into SUT
Spy Capture indirect output calls made by the SUT to another component
for later verification
Fake Objects to provide simpler implementation of a heavy component
Mock Object(s) that verify indirect output of the tested code
The University of Sydney Page 39
Dummy Object
– Dummy, dummy parameter/value
– Pass object with no implementation (dummy) and never actually used
– E.g., Fill in parameter lists
– SUT’s methods to be called often take objects stored in instance variables
– Those objects, or some of its attributes, will never be used in the testing
– Preparing the SUT into right state (conform to the signature of some methods
need to be called)
The University of Sydney Page 40
Dummy Object – Example
The University of Sydney Page 41
(Test) Stub
– A test-specific object that provides indirect inputs during tests
– E.g., Object requires data from a database to answer a method call
– Used to verify logic independently when it depends on inputs from other
components
– Verify indirect inputs of the SUT
– It does not deal with indirect outputs of the system
The University of Sydney Page 42
(Test) Stub – Example
The University of Sydney Page 46
(Test) Spy
– Capture output calls made by the SUT to another component for later
verification
– Verify indirect outputs of the SUT
– Get enough visibility of the outputs generated by the SUT (observation
point)
– E.g., email service that records no. of messages sent
The University of Sydney Page 47
(Test) Spy
– Capture output calls made by the SUT to another component for later
verification
– Verify indirect outputs of the SUT
– Get enough visibility of the outputs generated by the SUT (observation
point)
– E.g., email service that records no. of messages sent
The University of Sydney Page 48
Fake (Object)
– Objects to provide simplified implementation of a heavy (real) component
– E.g., in-memory implementation of repository using simple collection to store
data
– SUT depends on other components that are unavailable or make testing
complex or slow
– Run tests faster
– Should not be used when want to control inputs to SUT or outputs of SUT
The University of Sydney Page 49
Fake (Object) – Example
The University of Sydney Page 51
Mock (Object)
– Object(s) that verify indirect output of the SUT
– E.g., function that calls email sending service, not to really send emails
but to verify that email sending service was called
– Calling real implementation during testing is tedious, or the side effect is not
the testing goal
– Unlike all doubles, mocks verify correctness against expectations
The University of Sydney Page 52
Mock (Object) – Example
The University of Sydney Page 53
Mock (Object)
The University of Sydney Page 54
Test Doubles
– Understand the differences carefully and use the one that serve the
verification type and purpose and how it should be run
– Don’t be fooled by the mocking frameworks terminology – focus on the
verification purpose
– Read Fowler’s Mocks aren’t Stubs
– Check xUnit Test Patterns for more advanced details
https://martinfowler.com/articles/mocksArentStubs.html
http://xunitpatterns.com/
The University of Sydney Page 55
Contract Test
The University of Sydney Page 56
Test Double – External Services
– Test double to interact with external/remote service
– How accurate/reliable is a test double?
https://martinfowler.com/bliki/ContractTest.html
The University of Sydney Page 58
Contract Test
– The process of running periodic tests against real components to check the
validity of test doubles results
– How?
– Run your own test against the double
– Periodically run separate contract tests (real tests to call the real service)
– Compare the results
– Check the test double in case of results inconsistency/failures
– Also, consider service contract changes
https://martinfowler.com/bliki/ContractTest.html
The University of Sydney Page 59
Integrated (Broad) Tests
“Broad tests done with many modules active” – integrated testing
Read more for further discussion – https://martinfowler.com/bliki/IntegrationTest.html
The University of Sydney Page 60
Collaborative (Narrow) Tests
Read more for further discussion – https://martinfowler.com/bliki/IntegrationTest.html
“Narrow tests of interaction with
individual test doubles” – Collaboration
Tests
“supported by Contract Tests to ensure
the faithfulness of the double” –
Contract Tests
The University of Sydney Page 61
Integration Testing
Frameworks
Mockito
The University of Sydney Page 62
Mocking Frameworks
– Mockito
– JMock
– EasyMock
– Mountebank
– Others …
http://www.mbtest.org/
http://jmock.org/
http://easymock.org/
http://www.mbtest.org/
http://jmock.org/
http://easymock.org/
The University of Sydney Page 63
Mockito
– An open source testing (test spy) framework for Java
– It has a type called ‘spy’ which is partial mock1
– Verify interactions after executing tests (what you want)
– Not expect-run-verify (look for irrelevant interactions)
– Interaction among objects/components not unit testing
– Allows to specify order of verification (not all interactions)
https://github.com/mockito/mockito/wiki/FAQ
https://github.com/mockito/mockito/wiki/FAQ
The University of Sydney Page 64
Mockito – Constructs
Mockito Features Description
mock(), @Mock or
Mokito.mock()
Different ways to create a mock
Answer or MockSettings Interfaces to specify how a mock should behave (optional)
when() Specify the mock to return a value when a method is called
Spy() or @Spy Caution: creates a (partial mock) for a given object
@InjectMocks automatically inject mcoks/spies annotated with @Mock() or
@Spy()
verify() Check methods were called with given arguments
http://static.javadoc.io/org.mockito/mockito-core/2.24.0/org/mockito/Mockito.html
Note: call MockitoAnnotations.initMocks(testClass) (usually in a @Before method) to get the annotations to work.
Alternatively, use MockitoJUnit4Runner as a JUnit runner
http://static.javadoc.io/org.mockito/mockito-core/2.24.0/org/mockito/stubbing/Answer.html
http://static.javadoc.io/org.mockito/mockito-core/2.24.0/org/mockito/MockSettings.html
http://static.javadoc.io/org.mockito/mockito-core/2.24.0/org/mockito/Mockito.html
The University of Sydney Page 65
Mockito Example
The University of Sydney Page 66
Mockito – Method Call
– Use Mockito.when() and thenRturn() to specify a behavior when a method is
called
– Example of methods supported in Mockito
Method Purpose
thenReturn(valueToBeReturned) Return a given value
thenThrow(Throwable tobeThrown) Throws given exception
Then(Answer answer) User created code to answer
The University of Sydney Page 67
Mockito – ‘When’ Example
http://static.javadoc.io/org.mockito/mockito-core/2.24.0/org/mockito/Mockito.html#when-T-
The University of Sydney Page 68
Mockito – Verifying Behavior
– Mockito.verify (T mockTobeVerified, verificationMode mode)
– Verifies certain behavior happened at least once (default) – e.g., a method is called once
– Different verification modes are available
Verification Mode Description
Times(int wantedNoCalls) Called exactly n times, default = 1
atMost(in maxNoOfCalls) Called at most n times
atLeast(int minNoOfCalls) Called at least n times
never() Never called
Timeout (int milliseconds) Interacted in a specified time range
The University of Sydney Page 69
Mockito – Verifying Behavior Example
– Default mode is times (1) which can be omitted
– Argument passed are compared suing equals() method
The University of Sydney Page 70
Mockito – Verifying Order of Calls
– InOrder (mocks) allows verifying mocks in order
– verify(mock): verifies interactions happened once in order
– verify(mock, VerificationMode mode): verifies interactions in order
http://static.javadoc.io/org.mockito/mockito-core/2.24.0/org/mockito/InOrder.html
http://static.javadoc.io/org.mockito/mockito-core/2.24.0/org/mockito/InOrder.html
The University of Sydney Page 71
Writing Good Tests
The University of Sydney Page 73
Writing Good Tests
– Reliable
– Free of bugs, defects or errors
– Fast
– Should not be counterproductive, will be run very frequently
– Keep it compact and readable
– Lots of refactoring
– Follow recommended coding practices (e.g., naming conventions, documentation)
– Cover wide range to show positive cases and errorenous code paths
https://github.com/mockito/mockito/wiki/How-to-write-good-tests
https://github.com/mockito/mockito/wiki/How-to-write-good-tests
The University of Sydney Page 74
Writing Good Tests
– Do not mock everything
– It’s anti-pattern
– Understand mocking framework’s capabilities
– Mock syntax vs. actual purpose of mocking
– Read Fowler’s Mocks aren’t Stubs
– Do not mock type you do not own
– Third-party library or API – owner change the signature and behavior of the API
– Contract test ?
– Do not mock value objects
– Instantiating an object is too painful – not a valid reason
– Can be a sign that the code needs some serious refactoring or use builders for the value
objects (some tools such as Lombok can help)
https://github.com/mockito/mockito/wiki/How-to-write-good-tests
https://martinfowler.com/articles/mocksArentStubs.html
https://github.com/mockito/mockito/wiki/How-to-write-good-tests
The University of Sydney Page 75
W4 Lecture: Advanced Testing
Techniques 2
W4 Tutorial + quiz
Next Lecture/Tutorial
The University of Sydney Page 76
References
– Ian Sommerville. 2016. Software Engineering (10th ed.) Global Edition. Pearson
– James Shore and Shane Warden. 2007. The Art of Agile Development
– Martin Fowler, various testing articles. https://martinfowler.com/
– Gerard Meszaros, xUnit Test Patterns: Refactoring Test Code. Addison-Wesley
– Martin Fowler, Mocks Arent Stubs,
[https://martinfowler.com/articles/mocksArentStubs.html]
– Gaurav Duggal, Bharti Suri, Understanding Regression Testing Techniques.
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.460.5875&rep=rep1&t
ype=pdf
– Bernd Bruegge and Allen, H. Dutoit. 2009. Object-Oriented Software Engineering
Using Uml, Patterns, and Java (3rd ed.). Pearson
– Michal Lipski, Pragmatists: Test doubles: Fakes, Mocks and Stubs.
https://blog.pragmatists.com/test-doubles-fakes-mocks-and-stubs-1a7491dfa3da