Test-driven development
CSE 216 : Programming Abstractions
Dr. Ritwik Banerjee
Computer Science, Stony Brook University
Testing is not debugging
• Testing is the process of running a program on a set of test cases and comparing the actual against the expected results.
• It is meant to find defects in code and experimentally verify that the program does what it is supposed to do. It can be manual or automated.
• There are different types of testing: unit, integration, system/acceptance, stress, load, etc.
• Debugging is the process of finding and removing a specific bug from the program. It is always a manual, one-off process, since all bugs are different.
• Defensive programming is the philosophy of writing programs in a way such that debugging and testing is easier.
© 2019 Ritwik Banerjee 1
Testing is not debugging
while (before submission deadline) { code;
test;
if (test is not successful) {
debug; }
}
• Testing tells us if, when, and (possibly) where something is wrong.
• But it does not tell us how to fix it.
• This is where debugging comes into the picture.
© 2019 Ritwik Banerjee 2
Debugging is a manual process of understanding an error and correcting it.
Locate the problem:
the line number in a certain method within a certain class.
Understand why that line is wrong.
Then, either fix the algorithm/process, or change the process altogether.
Testing is not debugging
Of course, this is not always easy!
Some bugs may not be obvious (e.g., a loop that goes on infinitely, but only if a certain condition is true).
So, always design your code so that the different scenarios in which the code will be run are easy to test.
But never design your code with the idea that you will rely on debugging to write the correct code!
© 2019 Ritwik Banerjee 3
Some common bugs
• unconstructed objects
• uninitialized variables
• NullPointerException in Java
• forgetting to re-initialize of update a loop-
variable
• improper iteration
• IndexOutOfBoundsException in Java
• forgetting to implement a base-case in recursion
• Some bugs can kill, others are like the common cold that will only bother you for a couple of days.
• Syntax errors are the easiest to catch.
• These are compile-time errors, and you can immediately reach the
exact location of the error.
• Runtime errors are relatively harder to spot.
• You may not realize it until, at runtime, your code crashes.
• First, you figure out which object caused it.
bugs
More on
• You then have to backtrack through the code and figure out why that situation arose in the first place.
• Logical errors are the hardest.
• The code compiles. Runs. And does something different only
sometimes under certain circumstances.
• May the universe be kind to you if you are in this situation!
How to live normally
(and still be a programmer) ?
Always remember that Murphy’s Law reigns supreme!
• Where there’s smoke, there’s fire ⇒ where there’s code, there’s a bug
Solution
• Design and document your code.
• Avoid the “will deal with this later” phenomenon.
• In extreme cases, when you must delay a bug-fix, at least locate the bug, and then use a “todo” or “fixme” comment temporarily so that you fix it as soon as you look at your code the next time.
© 2019 Ritwik Banerjee 6
How to debug?
There are different approaches to debugging.
1.
The one who prints everything.
• Makes code very messy.
• May be completely useless in many modern coding scenarios (e.g., multithreaded programs, GUI systems, web and/or mobile applications).
The one who uses a debugger.
2.
Testing
01
Recall that debugging happens when you realize that there is a bug.
02
Often, the discovery of a bug is the result of testing a software.
03
Testing is only as good at the conditions tested for!
• untested conditions can still cause a program to function in invalid/improper ways.
© 2019 Ritwik Banerjee 8
Building a software product
Select
the work-force with great care.
Don’t succumb
to external pressure at the cost of the quality of your software code.
Don’t fall behind
schedule and enter the worker-hour trap of thinking about software development!
© 2019 Ritwik Banerjee 9
Worker-hours
• Labor is often measured in worker-hours, worker-months, worker-years, etc. A famous example from the yesteryears of the gaming industry, “Doom”:
• Originally announced as Doom 4 in 2008.
• Finally released in 2016 simply as “Doom”.
• Took 8 years and more than 100 worker-years of labor to develop.
• When asked in 2012, the company spokesperson and CEO John Carmack said that it will be “done when it’s done” … and the game was very positively received upon its extremely delayed release.
But why not double the size of the team and get it done in half the time?
© 2019 Ritwik Banerjee
10
The mythical story of the
worker-month
If a software takes one expert programmer a year to develop
… and due to market pressures, the company wants the product in a month,
“1#$%$&'($)”×”12,’-.h0 = 12#$%$&'($)0″×”1,’-.h?”
Throwing additional developers into a project that is late is likely to delay a project even further. Instead,
• Removefeaturesthatwerepromisedbutnotyetcompleted.
• Don’tmultiplyworkerbees,butmakesurethereisatleastonequeenbee
… the worker who has detailed knowledge of all the modules.
© 2019 Ritwik Banerjee
11
Design. Then implement. When a modular design has been completed, ask these questions:
• Can all the modules be developed in parallel? Unlikely, because usually, there are dependencies.
• Can classes within a module be developed in parallel? Unlikely, because usually, there are dependencies!
• Can members (methods, etc.) within a class be developed in parallel? Unlikely, because usually, there are dependencies!!
Two approaches:
• Top-down • Bottom-up
© 2019 Ritwik Banerjee 12
Top-down development
• Emphasize planning and a complete overall understanding of the system.
• No coding begins unless and until adequate details have been reached in the design (at least some part of the system, if not entirely).
• Implementation begins by attaching test stubs, which are programs that simulate the behavior of some software component.
Delays testing of functional units.
Simplifies system integration.
Critical high-level errors are easier to identify.
• This was the traditional approach, but stopped being popular in the mid-1980s.
Bottom-up development
• Emphasizes coding and testing as soon as the first module is specified.
– May lead to a situation where a module is being developed without a very clear idea of how it links to other modules in the larger system.
+ As a designer and developer, you are forced to think abstractly, and in general terms.
+ This usually leads to reusable code, and the resulting system is highly decoupled!
• These days, we usually combine top-down and bottom- up approaches.
• A project being developed in isolation may be mostly top-down.
• Otherwise, developers will try to leverage pre- existing modules, giving the development a bottom- up flavor.
Test-driven development
15
Do
• design modular classes and methods.
• decide before coding:
• what needs to be coded
• what needs to be tested
• design test cases for important methods.
• while coding, test incrementally as you implement. Do not
• code without a design
• design without a plan for testing
• write large amounts of untested code (some experienced developers will tell you that 10 lines is “large” in this context)
• write large methods/functions (this indicates lack of modularity)
© 2019 Ritwik Banerjee
Levels of testing
• No matter what model (top-down, bottom-up, or mixed) of development is adopted, there are tests (of correctness, efficiency, etc.) to be conducted at multiple levels. The following is a widely recognized hierarchy of such levels:
• Unit testing is the process of testing each unit (module, class, function) separately, in isolation.
• Integration testing is the process of testing the interfaces between various modules.
• Regression testing is done after modifications, to ensure that the original intended behavior or the program is preserved.
• System testing is done to test the overall behavior of the program in an integrated environment.
• Evaluates the program’s compliance with specified requirements.
Black-box testing
• It is impossible to perform exhaustive testing!
• So, test cases are sampled.
• A small but representative sample of all input combinations.
• This is often done by what is called black-box testing, which is the process of testing using cases generated based on the program specifications, and not the implementation.
• Tests are based on “what” the software is supposed to do but not “how” it does so.
• As a methodology, black-box testing can be applied to almost any type of testing, including unit, integration and system testing.
• System testing is usually considered black-box testing since it should require no knowledge of the internal design to perform system testing.
© 2019 Ritwik Banerjee
17
Black-box testing design techniques
18
Use Case Testing
• where test cases are sampled in order to cover the system on a transaction by transaction basis from start to finish
Boundary-Value Analysis
• A boundary value is an input that is right at the edge of the set of standard acceptable input values for a system. Moving even the slightest bit over to the “other side” will produce different behavior from the system.
• For example, how will a program that adds integers perform when the sum is exactly equal Integer.MAX_VALUE?
© 2019 Ritwik Banerjee
Glass-box testing
• Also called white-box, clear-box, transparent-box, or structural testing.
• Tests the internal workings and structure of the system, i.e., the “how” of the software, not its external behavior.
• Unit, integration, and system testing can be done with glass-box testing.
Due to internal knowledge of code, can zero in on weaknesses.
Makes tests traceable from source.
Are easy to automate, and hence much more testing can be done.
Can only be done by someone with internal knowledge of the code.
Focuses on existing functionality, so missing features may not be discovered.
© 2019 Ritwik Banerjee
19
Unit testing
Test small units of a Java application
• Classes
• Methods
Unit testing is usually an automatic process.
• Once implemented, can be run again and again.
Unit testing frameworks in Java:
• JUnit (our weapon of choice) • Arquillian
• TestNG
Unit testing in Python:
• Theunittestmodule
© 2019 Ritwik Banerjee 20
Unit testing terminology
• A test case is the individual unit of testing. It checks for a specific response to a particular set of inputs.
• A test suite is a collection of test cases, test suites, or both. It is used to aggregate tests that should be executed together.
• Sometimes, there may be additional preparation needed to perform one or more tests (connecting to a database, initializing variables, etc.). These are represented by a test fixture.
• A test runner is a component that orchestrates the execution of tests and provides the outcome to the user.
• A test runner may use a GUI, a terminal, or return a special value to indicate the results of executing the tests.
A simple unit test
• Consider this class, which has just the one method:
public class StringUnit {
public String concatenate(String a, String b) {
return a + b; }
}
• Testing a class means testing all its public methods.
• Sincethereisjustthemethodconcatenate()inthisclass, we will write the corresponding testConcatenate() method.
• Usually, a test method tests a single method in the target class. © 2019 Ritwik Banerjee
22
A simple unit test
• Using JUnit, the test would be something like import org.junit.Test;
import static org.junit.Assert.*; public class StringUnitTest {
@Test
public void testConcatenate() {
StringUnit u = new StringUnit();
String r = u.concatenate(“one”, “two”); assertEquals(“onetwo”, r);
} }
© 2019 Ritwik Banerjee
23
Assertion
• Assert methods are the secret ingredient of unit testing in Java
• From org.junit.jupiter.api.Assertions:
1. assertArrayEquals()
2. assertEquals()
3. assertTrue(), assertFalse() 4. assertNull(), assertNotNull() 5. assertSame(), assertNotSame() 6. assertThat()
Assertion
• Assert methods are the secret ingredient of unit testing in Java
• From org.junit.jupiter.api.Assertions:
1. assertArrayEquals()
• To assert whether or not two arrays are equal. 2. assertEquals()
• To assert whether or not two objects are equal.
• Uses the equals(Object o) method.
• Remember that if equals(Object o) is not overridden, reference equality will be used (==).
• If you override equals(Object o), you must also remember to override the hashCode() method.
3. assertTrue(), assertFalse() 4. assertNull(), assertNotNull() 5. assertSame(), assertNotSame() 6. assertThat()
Assertion
• Assert methods are the secret ingredient of unit testing in Java
• From org.junit.jupiter.api.Assertions:
1. assertArrayEquals()
2. assertEquals()
3. assertTrue(), assertFalse()
• If a condition – passed as parameter – is true (resp., false), throws an error.
4. assertNull(), assertNotNull() 5. assertSame(), assertNotSame() 6. assertThat()
Assertion
• Assert methods are the secret ingredient of unit testing in Java
• From org.junit.jupiter.api.Assertions:
1. assertArrayEquals()
2. assertEquals()
3. assertTrue(), assertFalse() 4. assertNull(), assertNotNull()
• If an object – passed as parameter – is not null (resp., null), throws an error.
5. assertSame(), assertNotSame() 6. assertThat()
Assertion
• Assert methods are the secret ingredient of unit testing in Java
• From org.junit.jupiter.api.Assertions:
1. assertArrayEquals()
2. assertEquals()
3. assertTrue(), assertFalse() 4. assertNull(), assertNotNull() 5. assertSame(), assertNotSame()
• To assert whether or not two objects refer to the same object.
• That is, uses reference equality (==). 6. assertThat()
• compares an object – passed as parameter – to an org.hamcrest.Matcher to see if the given object matches the condition defined by the matcher.
Matchers & Assertions
• Matchers offer a very useful way of testing assertions with the assertThat method, which is overloaded (in org.hamcrest.MatcherAssert):
public static
Matcher m)
public static
Matcher m) public static void assertThat(String reason,
boolean assertion)
• Several important matchers are available in org.hamcrest.Matcher (we
can also make our own custom matchers).
© 2019 Ritwik Banerjee 29
Some Abstract Matchers
allOf(Iterable
• conjunction (i.e., “and”) of all the elements in the iterable parameter.
anyOf(Iterable
• disjunction (i.e., “or”) of all the elements in the iterable parameter.
any(Class
• matches when the examined object is an instance of the specified type.
is(T value), equivalent to is(equalTo(x))
• matches if there is logical equality, based on the object’s equals(Object o) method.
not(Matcher
© 2019 Ritwik Banerjee 30
I/O components are usually
read from an InputStream, or
written to an OutputStream.
Unit testing In order to test these, you need control over the data that is
with I/O
read or written.
Next, we will look into unit testing (i) input, and (ii) output.
Unit testing input components
public class TokenUnit {
List
void read(InputStream input) throws IOException { StringBuilder builder = new StringBuilder(); int data = input.read();
while (data != -1) { // -1 denotes end of stream
if (((char) data) != ‘,’) builder.append((char) data);
else { tokens.add(builder.toString());
// empty the string builder
builder.delete(0, builder.length());
}
data = input.read();
}
}
void throwIllegalArgumentException() { throw new IllegalArgumentException();
} }
© 2019 Ritwik Banerjee
32
Unit testing input components
public class TokenUnitTest {
@Test
public void testRead() throws IOException {
TokenUnit unit = new TokenUnit();
byte[] data = “123,456,789”.getBytes(); InputStream input = new ByteArrayInputStream(data); unit.read(input);
assertEquals(“123”, unit.tokens.get(0)); assertEquals(“456”, unit.tokens.get(1)); assertEquals(“789”, unit.tokens.get(2));
} }
© 2019 Ritwik Banerjee
33
Unit testing output components
public class TokensOutputUnit {
List
public void write(OutputStream output) throws IOException { for(int i = 0; i < tokens.size(); i++) {
} }
© 2019 Ritwik Banerjee
if(i > 0) output.write(‘,’);
output.write(tokens.get(i).getBytes()); }
34
Unit testing output components
public class TokensOutputUnitTest {
@Test
public void testWrite() throws IOException { TokensOutputUnit unit = new TokensOutputUnit(); ByteArrayOutputStream output =
new ByteArrayOutputStream();
unit.tokens.add(“one”); unit.tokens.add(“two”); unit.tokens.add(“three”); unit.write(output);
String string = new String(output.toByteArray());
assertEquals(“one,two,three”, string); }
}
© 2019 Ritwik Banerjee
35
Running a Java program with/out assertions
• Bydefault,assertionsaredisabledatruntime, but they can be enabled by using the flag
-enableassertions | -ea
• Fromthecommandline,runyourtestas java -ea TokensOutputUnitTest
• Formorecomplexprograms,assertionscanbe selectively enabled/disabled at class and package levels
java –ea:packagename
–da:SomeOtherClass
TokensOutputUnitTest
© 2019 Ritwik Banerjee 36
Exception handling and assertions
public void setRadius(double d) {
// wrong; throw an exception instead
assert d >= 0;
this.radius = d;
}
• First of all, this is not a choice. Assertions should never replace exception handling!
• Exception handling is meant to deal with unusual circumstances during execution. Proper exception handling ensures robustness.
• Assertions, on the other hand, are meant to ensure/test correctness of a program.
• Valid arguments that may be passed to a method are within the method contract.
• Never use assertions to check validity of arguments!
© 2019 Ritwik Banerjee 37
Testing for exceptions
• Sometimes, we may want to test whether or not a code throws the correct exception when given an invalid input.
• We can use one of the overloaded assertThrows() methods
public static
Executable executable)
public static
String message)
public static
Supplier
© 2019 Ritwik Banerjee 38
public static
}
.reduce((o1, o2) -> from_start ?
o1.compareTo(o2) <= 0 ? o1 : o2 :
.orElse(null);
o1.compareTo(o2) < 0 ? o1 : o2)
Testing for exceptions
assertThrows(RuntimeException.class, // expected type () -> StreamUtils.least(empty, true), // executable String.format(“1(%s, %b)”, empty.toString(), true)) // message
© 2019 Ritwik Banerjee 39
Combining multiple tests
© 2019 Ritwik Banerjee 40
• The unit you are testing (say, a method) should be tested with multiple test inputs.
• But instead of creating one separate test method per input, it usually makes more sense to have one test method per actual method in your code.
• In this case, you would want to combine multiple test inputs inside one test method.
• However, if the !th test fails, the unit testing method reports the whole test as a failure. As a result, it does not continue to the !+1 st test.
@Test
void least() {
System.out.println(“least(): [5 tests; 1 point each]”); /* test 1 */
assertThrows(RuntimeException.class,
}
// expected type
// executable String.format(“1(%s, %b)”, empty.toString(), true)); // message
/* test 2 */
assertEquals(”I”, // expected
StreamUtils.least(strings1, true), // actual String.format(“%s || %b”, strings1.toString(), true)); // message
/* test 3 */
assertEquals(”a”),
StreamUtils.least(strings1, false),
String.format(“%s || %b”, strings1.toString(), false));
/* test 4 */
assertEquals(…); /* test 5 */ assertEquals(…);
() -> StreamUtils.least(empty, true),
© 2019 Ritwik Banerjee 41
@Test
void least() {
System.out.println(“least(): [5 tests; 1 point each]”); assertAll(
); }
() -> assertThrows(RuntimeException.class,
() -> StreamUtils.least(empty, true),
String.format(“%s || %b”, empty.toString(), true)),
() -> assertEquals(“I”, StreamUtils.least(strings2, true),
String.format(“%s || %b”, strings1.toString(), true)),
() -> assertEquals(“a”, StreamUtils.least(strings2, false),
String.format(“%s || %b”, strings1.toString(), false)),
() -> assertEquals(myclassobjects.get(3), StreamUtils.least(myclassobjects, true),
String.format(“%s || %b”, myclassobjects.toString(), true)),
() -> assertEquals(StreamUtilsSolution.least(doubles, true),
StreamUtils.least(doubles, true),
String.format(“%s || %b”, doubles.toString(), true))
assertAll() takes a sequence of assertions, and uses Stream#collect() all the failures into a List
© 2019 Ritwik Banerjee 42
Test fixtures
• A test fixture represents the preparations needed to perform the actual tests. This may include any cleanup required after the tests are over.
• The purpose of a test fixture is to ensure that there is a fixed environment in which tests are run so that the tests and their results are reproducible. In other words, a test fixture provides a controlled environment.
• Examples of fixtures include
• Preparation of input data and setting up mock
objects
• Establishing a connection to a database
• Closing a connection to a database after tests are finished running
• Something as simple as a fixed string, which is used as input for a method
If you need to initialize the same data for every test (i.e., every method annotated with @Test), you need to do this in the @Before setUp() method.
and
annotations in
JUnit
Test fixtures
• The @Before setup() method is called before each @Test method.
If there is any data that needs to be ‘cleaned up’, then you need to do this in the @After tearDown() method.
• The @After tearDown() method is called after each @Test method.
Execution
order with
tests and
fixtures
Suppose there is a test class Tests with two methods test1 and test2 for testing, but each method requires a resource (e.g., a file) to be opened for the test to work, and then closed once the test is finished running.
The execution order will be
1.Tests @Before setUp() 2.Tests @Test test1() 3.Tests @After tearDown() 4.Tests @Before setUp() 5.Tests @Test test2() 6.Tests @After tearDown()
Often, you will face a situation where opening or closing the resource is an expensive operation (e.g., takes time or leads to a spike in memory-usage).
In this case, it may make more sense to open the resource only once, run all the tests, and then close the resource after all tests are finished running.
expensive
fixtures
Working with
This makes the fixtures class-specific instead of method- specific, and the annotations reflect this. The execution order will then become
1.Tests @BeforeAll setUpClass() 2.Tests @Test test1()
3.Tests @Test test2()
4.Tests @AfterAll tearDownClass()
A test class prototype
import org.junit.jupiter.api.*;
import static org.junit.jupiter.api.Assertions.fail; public class StandardTests {
@BeforeAll
static void setupAll() { } @BeforeEach void setup() { } @Test void succeedingTest() { }
@Test void failingTest() { fail(“a failing test”);
}
@Test
@Disabled(“for demonstration purposes”) void skippedTest() { /* not executed */ }
@Test void abortedTest() { Assumptions.assumeTrue(“abc”.contains(“Z”)); fail(“test should have been aborted”);
}
@AfterEach
void tearDown() { }
@AfterAll
static void tearDownAll() { } }
© 2019 Ritwik Banerjee 47
Source: https://junit.org/junit5/docs/current/user-guide/#writing-tests-classes-and-methods
Unit testing Python code
• A test case in unit testing is an individual unit of testing. We will casually use this to mean individual test input-output pairs, but formally, a test case is a single class that contains multiple such “cases”.
• Treating a single class as a test case is meaningful for larger programs because multiple such classes are grouped together to form a test suite.
• For CSE 216, though, we will not create such test suites.
• Unit testing in Python is done using the unittest module, which is heavily inspired by Java’s Junit framework.
A basic unit test class in Python
import unittest
class TestStringMethods(unittest.TestCase):
def test_upper(self): self.assertEqual(‘foo’.upper(), ‘FOO’)
def test_isupper(self): self.assertTrue(‘FOO’.isupper()) self.assertFalse(‘Foo’.isupper())
def test_split(self):
s = ‘hello world’
self.assertEqual(s.split(), [‘hello’, ‘world’])
# check that s.split() fails when the separator is not a string with self.assertRaises(TypeError):
# noinspection PyTypeChecker
s.split(2)
if __name__ == ‘__main__’: unittest.main()
© 2019 Ritwik Banerjee
49
Source: https://docs.python.org/3/library/unittest.html