CS246-F20-02-TestingOverview
Lecture 2.2
• Unit and integration testing
• Black-box and white-box testing
• Other kinds of testing
• Static and dynamic analysis
CS246
Scale of testing
• Unit testing tests individual components, such as a single class or
a file or a set of closely related functions from a single API
– They often require some technical infrastructure (scaffolding) to be able to
create and run the tests
– JUnit is the well known Java unit testing framework
– There are several options for C++: Google, Boost, CPPUnit
– Feel free to read up on this; we won’t require its use in CS246
• Integration testing tests functionality at a coarser level of
granularity (i.e., subsystems comprised of multiple units)
– Continuous integration, gradual scaling up, not “big bang merge”
– System-level testing is the coarsest level of integration testing; it tests the
functionality visible to the external user (like Marmoset)
https://twitter.com/ThePracticalDev/status/892788721350836225/ https://twitter.com/ThePracticalDev/status/845638950517706752
https://twitter.com/yogthos/status/951905438727057408?lang=en https://twitter.com/tiggex/status/953604154450173952
Two central metaphors of testing
1. Black-box testing
aka specification testing
– Test a class / method / subsystem against what it is
supposed to do, but without looking at how the code
achieves it
– e.g., test vector.erase(element) on
• element present, not present, present multiple times,
• vector of length zero, one, two, medium-sized length, long length
Two central metaphors
2. White-box testing
aka structural testing aka glass-box testing
– Try to exercise all parts of the underlying implementation
– Test against how the code is built, rather than what it is
supposed to do
• Kick the tires, push all the buttons at least once
• Can you exercise all possible statements, paths?
• for loop: execute 0, 1, K, N-1 times
• Evaluate all possible predicates: if (A || B)
Other testing topics
• Regression testing
– “A bug is simply a test case you forgot to write.”
– Keep the test suites maintained along with the code; when you check
in new code, run against the regression suites to make sure you didn’t
break anything
– Adding new features means adding new regression tests
– Changing existing design / functionality means changing existing (or
adding new) regression tests
– Regression tests accumulate over time; testing takes longer and
longer; developers fear to remove them tho
• Moral: Testing is a core part of the system; treat it that way
Other testing topics (CS447)
• Usability testing
– Very different from code-oriented testing
– Watch users interact with system, interviews, evaluations, look for
usage patterns, common mistakes, frustrations, …
• Testing non-functional requirements (NFRs)
– e.g., performance, robustness, security
– Need special-purpose techniques, often at the system level
– Often, can be planned but not performed until system is more or less
completed
Static vs. dynamic analysis
• Static: Some tools can take the code and analyze it without
executing it, looking for likely problems
e.g., Look for accessing an object thru a freed pointer
Tools: lint, FindBugs, Coverity, KLOCworks, UnderstandForC++
• Dynamic: Other tools can take input from (or watch /
instrument) an executing system to produce reports
e.g., look for unexpected bottlenecks
Tools: valgrind, gprof
Testing in CS246
• We don’t go very far into testing on CS246 assignments
– Marmoset supports only system-level (end-user) testing
• You might want to try to set up a regression test script for the
project, since you will be working in small groups
– Make sure your teammate doesn’t break something you got right, and
vice versa
• CS447 is all about testing and other kinds of QA!
End
CS246
2. A few thoughts on testing
CS246 Fall 2020
Prof. Mike Godfrey
University of Waterloo