CIS 3760: Software Engineering Software Quality –
Defect Detection
Instructor: Prof. Stacey Scott
School of Computer Science
University of Guelph https://www.uoguelph.ca/computing/people/stacey-scott stacey.scott@uoguelph.ca
© 2017-20 Stacey D. Scott – with content from J. McCuaig, used with permission
Required Reading
•
“Quality Assurance, Quality Control and Testing — the Basics of Software Quality Management” by AltexSoft
•
Read this for Thursday. We will be discussing it.
– https://www.altexsoft.com/whitepapers/quality-assurance- quality-control-and-testing-the-basics-of-software-quality- management.
CIS 3760
Software Engineering
2
Detecting Defects (Testing)
3
Taxonomy for Defect Handling Techniques
Methodology
Configuration Management
Exception Handling
Modular Redundancy
Static Analysis Methods
Verification/ Formal Methods
Unit Testing
Integration Testing
System Testing
Note: this is an incomplete taxonomy; other techniques exist, including usability testing
Defect Prevention
Defect Detection
Defect Tolerance
CIS 3760
Software Engineering
Slides adapted from Bruegge & Dutoit, Pearson online resources
4
Testing
Debugging
Defect Handling
Outline
• Testing basics & Types of software testing
• Unit testing basics – driversandstubs
• Integration testing basics – Topdownvs.bottomup
• White-box vs. Black-box testing – Strategiesfordevelopingtestcases
CIS 3760
Software Engineering
5
Software Testing
Defn:
Testing is the process of exercising a program with the specific intent of finding errors prior to delivery to the end user.
CIS 3760
Software Engineering
6
– Roger Pressman, 2004
Slides adapted from Pressman, McGraw-Hill online resources
Observations on Testing
• It is impossible to completely test any nontrivial module or system
– Practicallimitations:Completetestingisprohibitiveintimeand cost
• “Testingcanonlyshowthepresenceofbugs,not their absence” (Dijkstra).
• Testing is not for free, so it should be done carefully, using systematic processes
• Goal of testing should be to “break” the code…different mindset than development of “correct” code
CIS 3760
Software Engineering
7
Slides adapted from Bruegge & Dutoit, Pearson online resources
Testing vs. Debugging
“Some programmers use the terms “testing” and “debugging” interchangeably, but careful programmers distinguish between the two activities.
Testing is a means of detecting errors.
Debugging is a means of diagnosing and correcting the root causes of errors that have already been detected.”
– McConnell, Code Complete 2nd ed., p. 500 [emphasis added]
CIS 3760
Software Engineering
8
Who Tests the Software?
CIS 3760
Software Engineering
9
Slides adapted from Pressman, McGraw-Hill online resources
Testing Takes Creativity
• To develop an effective test, one must have:
– Detailedunderstandingofthesystem
– Applicationandsolutiondomainknowledge
– Knowledge of the testing techniques
– Skilltoapplythesetechniques
• Testing is best done by independent testers
– Weoftendevelopacertainmentalattitudethattheprogram
should behave in a certain way when in fact it does not
– Programmers often stick to the data set that makes the program work
– Aprogramoftendoesnotworkwhentriedbysomebodyelse
CIS 3760
Software Engineering
10
Slides adapted from Bruegge & Dutoit, Pearson online resources
Types of Software Testing
System testing
1: Software system
Note, boundaries are not precise.
3: Within a subsystem: division into different classes / packages
Unit testing is typically done on classes created by single devs/team of devs.
Component testing
Component testing done on class/ package created by multiple devs or teams.
Unit testing
4: Data and functions/ routines within classes / packages
CIS 3760
Software Engineering
5: Internal method design
11
Integration testing
2: Subsystems
Typical focus of developer testing
1: Software system
CIS 3760
Software Engineering
5: Internal method design
12
Integration testing
2: Subsystems
Component testing
3: Within a subsystem: division into different classes / packages
Unit testing
4: Data and functions/ routines within classes / packages
Testing Activities
Unit Testing
Integration Testing
System Testing
Acceptance Testing
Developer / Tester / SQA
Client
CIS 3760
Software Engineering
13
User
Usability Testing
Slides adapted from Bruegge & Dutoit, Pearson online resources
Purpose and Resources for Testing Activities
Object-level Design
System-level Design
Requirements Analysis
Client Expectations
Unit Testing
Integration Testing
System Testing
Acceptance Testing
Finds defects in objects and/or subsystems with respect to use cases
Finds defects by testing individual components in combination
Tests whole system, with respect to use cases / user stories
Tests whole system, with respect to project agreement (or “definition of done” for user 14
Developer / Tester / SQA
Client
CIS 3760
Software Engineering
stories)
Slides adapted from Bruegge & Dutoit, Pearson online resources
Purpose and Resources for Testing Activities
Object-level Design
System-level Design
Requirements Analysis
Client Expectations
Finds defects in
User
user interface design of the system
User Interface Design Usability Testing
Unit Testing
Integration Testing
System Testing
Acceptance Testing
Developer / Tester / SQA
Client
CIS 3760
Software Engineering
15
Slides adapted from Bruegge & Dutoit, Pearson online resources
V-Model of Testing: Mapping each SE stage to validation (testing) activity
Q: This models looks very Waterfall-y. What would this look like in Scrum context? What are the “validation” activities called?
CIS 3760 Source: https://www.plutora.com/wp-content/ Software Engineering uploads/2018/01/Integration-Testing-V-Model.png
16
V-Model for Agile Development Testing
Special Note: Testing is an iteration activity and part of the definition of “done”.
CIS 3760 Source: https://www.modernanalyst.com/Resources/Articles/tabid/115/ID/1967/A-17 Software Engineering Proposal-for-an-Agile-Development-Testing-V-Model.aspx
Types of Testing
Unit Testing
System Testing
– Individual components (class or subsystem) are tested
– The entire system is tested
– Carried out by developers
– Carried out by developers
– Goal: Confirm that the component or subsystem is correctly coded and carries out the intended functionality
– Goal: Determine if the system meets the requirements / user needs
Integration Testing
Acceptance Testing
– Groups of subsystems (collection of subsystems) and eventually the entire system are tested
– Evaluates the system delivered by developers
– Carried out by developers
– Carried out by the client. May involve executing typical transactions on site on a trial basis
– Goal: Test the interfaces among the subsystems
– Goal: Demonstrate that the system meets the requirements and is ready to use
CIS 3760
Software Engineering
Slides adapted from Bruegge & Dutoit, Pearson online resources 18
Outline
Testing basics & Types of software testing Unit testing basics
– driversandstubs
• Integration testing basics – Topdownvs.bottomup
• White-box vs. Black-box testing – Strategiesfordevelopingtestcases
CIS 3760
Software Engineering
19
We will focus on
1: Software system
Unit Testing
During implementation, you should: 1. Write a routine or class
2. Check it mentally
3. Have it peer reviewed and/or
3: Within a subsystem: division into different classes /
4. Test it
CIS 3760
Software Engineering
20
Unit testing
4: Data and functions/ routines within classes / packages
2: Subsystems
packages
5: Internal routine (function) design
Test Drivers and Stubs
• Driver:
– Aroutine/programthatcallsthe
TestComponent
Driver
– Controlsthetestcases(theprogram executing the test)
• Stub:
– AcomponenttheTestComponent
TestComponent
depends on
– Partial implementation of some other system component / package
Stub
21
– Returnsfakevalues
CIS 3760
Software Engineering
Slides adapted from Bruegge & Dutoit, Pearson online resources
Unit Test Environment
CIS 3760
Software Engineering
22
Slides adapted from Pressman, McGraw-Hill online resources
(see defn later in slides)
Unit testing frameworks
• Software frameworks that facilitate unit testing • Sample frameworks:
– JUnit(http://junit.org)(seeUnitTestingtutorialslides from lab)
– PyUnit(http://pyunit.sourceforge.net)
– Check(C)(http://check.sourceforge.net)
– APISanityChecker(C/C++)(https://lvc.github.io/api-sanity- checker)
– UnitJS(http://unitjs.com)
– Many,manymore…
CIS 3760
Software Engineering
23
Outline
Testing basics & Types of software testing Unit testing basics
drivers and stubs
Integration testing basics
– Topdownvs.bottomup
• White-box vs. Black-box testing – Strategiesfordevelopingtestcases
CIS 3760
Software Engineering
24
Integration vs. System Testing
System Testing
Integration Testing
1. We check system as a whole. (input-output testing)
We check the interfacing between connected components.
2. Used for functional and non- functional testing (performance, usability etc).
Generally limited to functional aspects of the integrated components.
3. Testing is limited to evaluation of requirements, hence, it includes black-box testing techniques only.
Interfacing logic typically required to perform this testing, hence, it includes white-box testing (and possibly also black-box techniques).
4. Types of system testing: Functional, Performance, Usability, Reliability, Security, Scalability, and Installation testing, etc.
Types of integration testing: Top down, bottom up, big bang, and hybrid integration. (see video next slide)
CIS 3760
Software Engineering
Source: https://artoftesting.com/manualTesting/difference- between-system-and-integration-testing.html
25
Integration Testing: Video Explanation
CIS 3760
Software Engineering
26
Note about integration during Scrum sprints
• When incorporating new features during a sprint to your existing, working application integration testing occurs…this is “big bang testing”…you test all components when they are ready. (i.e. this does not involve drivers and stubs, which are needed when you are trying to test yet-to-be developed features that your code under test connects to).
• The advantage of the Scrum (or any agile) approach is that you are doing integration throughout the whole project, as new features are developed. Thus, you can uncover assumptions made by developers working on different components early in the project.
CIS 3760
Software Engineering
27
Integration Testing: Example
Music system
CIS 3760
Software Engineering
28
Slides adapted from Pressman, McGraw-Hill online resources
Integration Testing: Top Down Integration
Top Down Testing – This form of testing involves testing the high level module(s) first, then testing lower level modules as development is completed and they are integrated.
Stubs are used to simulate the data response of lower level modules until they are completed and integrated.
CIS 3760
Software Engineering
29
Slides adapted from Pressman, McGraw-Hill online resources
Integration Testing: Bottom Up Integration
Bottom Up Testing – Lower level modules are tested first to ensure the individual modules are working correctly before they are integrated with their parent module.
Drivers are used to simulate the higher level modules data response until the development of the higher level module is completed and integrated.
CIS 3760
Software Engineering
30
Working modules are grouped into builds and integrated
Good for finding bugs
Slides adapted from Pressman, McGraw-Hill online resources
Test Scenarios and Test Cases
• To conduct any of the previous test types, you need to create appropriate test scenarios and test cases
• Test scenario: Any system functionality that can be tested. – Examples for an eCommerce system:
Test Scenario 1: Check the Search Functionality Test Scenario 2: Check the Login Functionality
(https://www.guru99.com/test-case-vs-test-scenario.html)
• Testcase-asetofconditionsorvariablesunderwhicha
tester will determine whether a system under test satisfies
requirements or works correctly (http://softwaretestingfundamentals.com/test-case)
– For each test scenario, multiple test cases are usually needed to validate the scenario, for instance, using multiple possible input and output combinations or testing the system under different states (see video on next slide for example)
CIS 3760
Software Engineering
31
Designing Test Cases Video Explanation
How to write a test case? https://youtu.be/BBmA5Qp6Ghk
CIS 3760
Software Engineering
32
Outline
Testing basics & Types of software testing Unit testing basics
drivers and stubs
Integration testing basics
Top down vs. bottom up
White-box vs. Black-box testing
– Strategiesfordevelopingtestcases
CIS 3760
Software Engineering
33
Black-box vs. White-box testing
• “Black-box testing”
refers to tests in which
the tester cannot see (or is not focused on) the inner workings of the item being tested
output
– Typically performed by external testers
input
– Focusedontheinput/outputbehaviour–foranygiven input, what output is expected?
• “White-boxtesting”
refers to tests in which
the tester is focused on the inner workings of the item being tested
input
– Typicallythekindoftestingyouwilldoasadevelopertotest your own code
CIS 3760
Software Engineering
34
logic path
logic path
output
Determining Test Cases (for Unit Testing)
• Black-box testing
input
logic path
output
– Design potential scenarios, with appropriate sample data, to test the correct functioning of a function / routine
– Where to get these scenarios? Requirements. Logic. Creativity.
• White-box testing
input
logic path
output
– Can do above, but can also do much more
– Goal:thoroughness(testcoverage).Everystatementinthe component is executed at least once
– StatementTesting,LoopTesting,PathTesting,BranchTesting
– See“Tricks”forwritingTestCasesinnextslide
CIS 3760
Software Engineering
35
Review: V-Model of Testing
Q2: What is the difference b/w an integration test
and a system test?
CIS 3760 Source: https://www.plutora.com/wp-content/ Software Engineering uploads/2018/01/Integration-Testing-V-Model.png
36
Q1: What is a unit test?
Review
• What is a unit test?
• What is the difference between integration and system testing?
• What type of testing do ugrad computing students typically do?
CIS 3760
Software Engineering
37
Discussion of “Quality Assurance, Quality Control, and Testing” reading
• Placement of QA personnel – Embeddedinagileteam?
– SeparateQAteam?
• Design and execution of test strategies
– Advantagesofstatictestingmethodsoverdynamictestingmethods? – Defectprevention:staticordynamictesting?
• Levels of testing
– Whichlevel(s)oftestinghaveyoubeendoinginyourCIS3760 project?
– Whatleveloftestingisthesprintdemo?
– Inwhichlevelsoftestingmightyouusestubsand/ordrivers?
– Howdostubsanddriversrelatetotheconceptsofbottom-upand
top-down integration testing?
38
CIS 3760
Software Engineering
Additional Readings / Resources
• Programming Fundamentals: Software Testing/QA Course, LinkedIn Learning (Instructor: Meaghan Lewis): https://www.linkedin.com/learning- login/share?forceAccount=false&redirect=https%3A%2F%2Fwww.li nkedin.com%2Flearning%2Fprogramming-foundations-software- testing-qa%3Ftrk%3Dshare_ent_url&account=56996833
• [Herman Bruyninckx] Embedded Control Systems Design, WikiBook, Learning from Failure: http://en.wikibooks.org/wiki/Embedded_Control_Systems_Design/L earning_from_failure
• Joanne Lim, An Engineering Disaster: Therac-25
http://www.bowdoin.edu/~allen/courses/cs260/readings/therac.pdf
CIS 3760
Software Engineering
39
Slides adapted from Bruegge & Dutoit, Pearson online resources
Additional Material (Optional)
40
“Tricks” or Methods for Writing Test Cases (for Part B in Assign. 4 – see also McConnell Ch 22.3)
• Structured Basis Testing
– Testingeachstatementintheprogram/componentatleast once
– Foranybranching(if/case),orlooping,youneedtovarythe testing according to how complicated the expression is (all possible evaluation options))
• Data-Flow Testing
– Testing all combinations of data assignment, use, and destruction
• Boundary Analysis
– Testingfor“off-by-one”errors
– Foreveryboundary,testbelow,in,andabovetheboundary
CIS 3760
Software Engineering
41
Example of Boundary Analysis Testing
(For every boundary, test below, in, and above the boundary)
Consider an application that has an input box accepts numbers between 1-1000, inclusive. Valid range 1-1000, Invalid range 0 and Invalid range 1001 or more.
Q: What are the boundary analysis test cases?
CIS 3760
Software Engineering
42
Write Test Cases for Valid partition value, Invalid partition value, and exact boundary value.
Test Case 1: Consider value less than valid partition: value=0 Test Case 2: Consider value exactly lower boundary: value=1 Test Case 3: Consider values between 2 and 999: 2 & 999
Test Case 4: Consider value exactly upper boundary: value=1000 Test Case 5: Consider value above valid partition: value=1001
CIS 3760
Software Engineering
43
•
White-box Testing
Both types of testing are needed
•
Black-box Testing
– Number of possible logical paths
Comparison of White & Black-box Testing
– –
Potentially infinite number of paths have to be tested
• White-box testing and black box testing are the extreme ends of a testing continuum
–
Cannot detect missing requirements
– – –
Potential explosion of test cases (valid & invalid data)
– Nature of input data
White-box testing often tests what is done, instead of what should be done
• Any choice of test case lies in between and depends on the following:
Often not clear whether the selected test cases uncover a particular error
– Complexity of algorithms and data structures
Does not discover extraneous use cases (“features”)
CIS 3760
Software Engineering
Slides adapted from Bruegge & Dutoit, Pearson online resources 44
– Amount of computation
Most important thing to remember when testing your code
• Goal is to FIND defects… • NOT to fix the defects…
• Very common mistake developers make:
– Starttestingcode,firstdefect/bugtheyfind,theyputtheir “dev” hat back on and start fixing the bug, and
– Then,forgettocomebacktothetesting…thuscodeisnot completely tested
CIS 3760
Software Engineering
45
Review
• What is the difference between black-box and white- box testing?
• What resources are used to inform the test cases for black-box and white-box test cases?
• What strategies can be used to ensure thorough coverage for white-box tests?
– Is testing each line of code once sufficient?
• What are drivers and stubs? What types of tests use them?
CIS 3760
Software Engineering
46
Current Software Disaster in Canadian News
• https://www.itworldcanada.com/article/phoenix-payroll-system-timeline-of-the-governments-problems/396407
• https://www.itworldcanada.com/blog/phoenix-payroll-report-by-michael-wernick-the-clerk-of-the-privy-council/385370
• http://www.cbc.ca/news/canada/topic/Tag/Phoenix%20Falling
• https://www.theCgIlSob3e7a6n0dmail.com/politics/article-phoenix-pay-system-problems-on-track-to-cost-government-22-billion/
47
Software Engineering
Current Software Disaster in Canadian News: The Phoenix Payroll System
“More than half of the roughly 300,000 people employed by the federal government have faced pay problems since the Phoenix system was introduced more than two years ago.
Workers have been overpaid, underpaid or
not paid at all for months at a time.”
– CBC News, Oct 22, 2018
CIS 3760
Software Engineering
48
CIS 3760
Software Engineering
https://spectrum.ieee.org/riskfactor/computing/software/canadian-
49
“Ferguson’s audit describes what is essentially a manual for senior managers desiring to sabotage IT projects. … the estimated software development cost was C$119 M more than the C$155 M originally budgeted.”
‐ force‐fit project into the existing budget: reducing its functionality, testing, schedule, dev staff
‐ system was never end‐to‐ end tested
‐ pilot rollout was canceled
governments-phoenix-pay-system-an-incomprehensible-failure