—ISYS1087/5 Software Testing Chapter 4
Test Techniques
A/Prof I. Balbin
Lecture Session: Weeks 5, 6 & 7
Aim: we examine techniques for testing software by executing the test objects on a computer
Lecture Aims
We present different techniques, with examples, for specifying test cases and for defining test exit criteria.
These test design techniques are divided into three categories: ▪ black box testing,
▪ white box testing, and
▪ experience-based testing.
Testing by execution
Dynamic Execution means that we execute test cases ON some system
If there is code, we inject the test cases using a test harness system (for example: spiratest or selenium) into the code and observe and manage the results of the test
With integration and component testing, often this cannot be achieved without some additional software that ▪ may not yet have been written
▪ isn’t currently available
▪ requires a driver
Stubs simulate the input/output behaviour of the part of the program that usually would be called by the test object
In the diagram on the next page,
▪ PoC = Point of Control
▪ PoO = Point of Observation
• The tester must often create the test bed, or the tester must expand or modify standard (generic) test beds, adjusting them to the interfaces of the test object
• The objective of testing is to show that the implemented test object fulfils specified requirements as well as to find possible faults and failures
• Unstructured testing “from your gut feeling” does not guarantee that as many as possible, maybe even all, different situations supported by the test object are tested
Recall the formal test plan document (IEEE 829)
• Test Plan Identifier • References
• Introduction
• Test Items
• Features To Be Tested
• Features Not To Be Tested • Approach
• Pass/Fail Criteria
• Suspension Criteria
• Test Deliverables
• Testing Tasks
• Environmental Needs
• Responsibilities
• Staffing and Training Needs • Schedule
• Risks and Contingencies
• Approvals
Structured Approach
1. Determine conditions and preconditions for the test and the goals to be achieved. 2. Specify the individual test cases.
3. Determine how to execute the tests (usually chaining together several test cases) The degree of formality depends on several factors, including but not limited to:
• the application area of the system (for example, safety-critical software), • the maturity of the development and test process,
• time constraints, and
• knowledge and skill-level of the project participants
…
At the beginning,
➡ the test basis is analysed to determine what must be tested (for example, that a particular
transaction is correctly executed)
➡ the test objectives are identified, for example, demonstrating that requirements are met.
➡ the failure risk should especially be taken into account.
➡ the tester identifies the necessary preconditions and post conditions for the test, such as what data should be in a database.
Traceability between specifications and test cases allows an analysis of the impact of the effects of changed specifications on the test, that is, the necessity for creation of new test cases and removal or change of existing ones.
Traceability also allows checking a set of test cases to see if it covers the requirements. Thus, coverage can be a criterion for test exit.
The number of test cases can soon reach hundreds or thousands. Only traceability makes it possible to identify the test cases that are affected by specification changes.
Start with blank sheet?
The expected results (output, change of internal states, etc.) should be determined and documented before the test cases are executed. Otherwise, an incorrect result can easily be interpreted as correct, thus causing a failure to be overlooked.
Test cases should be grouped so that a whole sequence of test cases is executed (test sequence, test suite or test scenario).
Such a test sequence is documented in the test procedure specifications or test instructions.
▪ This document commonly groups the test cases by topic or by test objectives.
…
• Test priorities and technical and logical dependencies between the tests and regression test cases should be visible
• The test execution schedule (assigning tests to testers and determining the time for execution) is described in a test schedule document
• A test script contains instructions for automatically executing the test sequence, usually in a programming language or scripting language,
• The test script may contain the corresponding preconditions as well as instruction for comparing the actual and expected results.
• For example: JUnit is an example of a framework that allows easy programming of test scripts in Java.
xunit
A framework is a collection of classes, procedures, and macros
xunit is set of “frameworks” for programming and automated execution of test-cases
x stands for programming language
▪ most Famous is JUnit for Java
▪ exists for almost all programming languages ▪ CUnit, CppUnit, DUnit, JUnit, NUnit, …
Basic use of a framework
Elaborating
Assertions
▪ Boolean expression that compares expected and actual results ▪ The basic and smallest building-block
▪ General: ASSERT (expected, actual)
Test Case
▪ A class that extends “TestCase”s
▪ A composition of concrete test procedures
▪ May contain several assertions and test for several test objectives ▪ E.g all test of a particular function
Test Suite
▪ Collection of related test cases
▪ Can be executed automatically in a single command
Generally
xunit advice
Make them fast;
➡ If slow, developers won’t run them.
• Smoke test suites
• Complete test suites
All developers must know about them;
➡ Everyone who touches the code must run the tests.
➡ Add to common code-repository
Make test-code as nice & readable as implementation code
▪ Documentation, Maintainability
Keep test code in a separate directory
Keep both tests-sources and implementation-source in version control Don’t checkin unless version passes all tests
Test Case Design Techniques
Two broad categories
Black Box
▪ You don’t know about the code
▪ Sometimes called requirements testing. In Agile, consider User Acceptance Testing ▪ Also known as function, behavioural or specification based testing
▪ Formal Logic based testing is used in certain application areas
White Box
▪ Code is used
▪ internal flow is observed
▪ cases guided by program structure
▪ Gives rise to the issue of coverage
▪ Sometimes called glass box or structural testing
Pictorially
Stages of testing
White box testing can be used for
▪ component testing
▪ integration testing
▪ makes very little sense for system testing Some Black box testing is not systematic
▪ intuitive testing
▪ experience-based testing
As we will see, sometimes we need to have a bit of White and Black: that is, Grey Box testing
Black Box: Methods for choosing test cases
Equivalence Class Partitioning
An equivalence class is a set of data values that the tester assumes are processed in the same way by the test object.
Testing one representative of the equivalence class is considered sufficient because it is assumed that for any other input value of the same equivalence class, the test object will show the same reaction or behaviour.
Besides equivalence classes for correct input, those for incorrect input values must be tested as well
Example: Valid sales prices
Recall, in our Virtual Show Room, that the following text is part of the description of the requirements in respect of the Sales Price:
“For a sales price of less than $15,000, no discount shall be given. For a price up to $20,000, a 5% discount is given. Below $25,000, the discount is 7%, and from $25,000 onward, the discount is 8.5%.”
We therefore derive four distinct valid equivalence classes
Test Coverage
• The degree to which a test item has been exercised by a test suite, as a percentage. For example, what is the “statement” coverage.
• 100% coverage doesn’t mean 100% tested
• Just because you execute/test a line of code doesn’t mean your test case found a
defect arising from that line of code.
• Coverage doesn’t tell us about things that are missing. It works on what has been
written.
• Coverage doesn’t tell us about the quality of the tests or the system
• In each of the blackbox techniques we will describe eg Equivalence partitioning, Boundary analysis, decision tables, state transition tables, we will need to identify the element that is being covered. Eg % of states in state transition
Test Coverage
When we say “What percentage of tests have been run” that is NOT Test coverage. Some may call this “test completeness”
• Coverage is about an item to be covered.
• It is a test management metric designed to give us confidence that we have covered the risky functional and non
functional requirements.
• Coverage is when we want to know the percentage covering of “something” by a test.
• Test coverage is an important metric that defines the extent to which the software product’s complete functionality is covered. It indicates the completion of testing activities and can be used as a criterion for concluding testing
Test coverage can be viewed through the lenses of
• feature—check whether you covered with tests all the functionality
• GUI icon—each interface has its own menu, screens, buttons, and pull-downs. The point of this measurement is to see if we have tests for every item of an interface
• structure—eg tests for required, solution coverage (branches)
• scenario: testers use the reports of user experience to see what functions should be approached
• web script, page, app—indication of the risk level of all these “web artefacts” helps to lower the opportunity of risk appearance. For successful deployment of test coverage, you need to see the bug risks
Coverage support
Practically, we use tool support.
1. Decide on a structural element—the coverage item—that is to be counted
2. Count these elements
3. Inject the code
4. Run the tests for this item to be measured
5. Determine based on the output, the percentage of of items that were exercised by the injected tests
Remember: you can test the same item with different data and only a test case with this data may may show up a defect even though the structure/item has been covered.
Equivalence Classes for invalid sales
prices
It is also possible to work backwards from equivalence classes of output. More expensive: for each output there can be multiple inputs.
The best test values are those verifying the boundaries of the equivalence classes. There are often misunderstandings or inaccuracies in the requirements at these spots because our natural language is not precise enough to accurately define the limits of the equivalence classes
Experience counts
If it is assumed that the program’s reaction on an incorrect value is always the same (e.g., an exception handling that delivers the error code NOT_VALID), it is sufficient to map all incorrect values to NaN for Not a Number).
Floating-point numbers are part of this equivalence class: it is expected the program displays an error message to inputs such as 3.5. In this case, the partitioning method does not require further subdivision because the same reaction is expected for every wrong input.
.
However, an experienced tester will always include a test case with a
floating-point number in order to determine if the program rounds the number
and then uses the corresponding integer number for its computation
Positive and Negative and Zero Values
Negative and positive values are often handled differently, it is sensible to further partition the valid equivalence class. Zero is also an input, which often leads to failure.
• we should test the boundary values of: MIN_INT, -1, 0, MAX_INT.
• for invalid values, no boundary values can be given.
• including the boundary values for the integer parameter extras results in the following seven values to be tested:
{”f”, MIN_INT, -123, -1, 0, 654, MAX_INT}
• For each of these, the predicted outputs or reactions of the test object must be defined, to decide if there was a failure.
Guide to determining equivalence classes
1. For the inputs and outputs, identify restrictions and conditions from the specification.
2. For every restriction or condition, partition into equivalence classes:
3. If a continuous numerical domain is specified, then create one valid and two invalid equivalence classes.
4. If a number of values should be entered, then create one valid (with all possible correct values) and two invalid equivalence classes (less and more than the correct number).
5. If a set of values is specified where each value may possibly be treated differently, then create one valid equivalence class for each value of the set (containing exactly this one value) and one additional invalid equivalence class (containing all possible other values).
6. If there is a condition that must be fulfilled, then create one valid and one invalid equivalence class to test the condition fulfilled and not fulfilled.
7. If there is any doubt that the values of an equivalence class are treated “equally”, the equivalence class should be divided further into subclasses.
Test case design
• The representative values of all valid equivalence classes should be combined to test cases. Any of those combinations of values for variables builds a valid test case.
• The representative value of an invalid equivalence class is combined only with representatives of other valid equivalence classes (the first one will likely generate an exception so the other(s) will be wasted).
• How many test cases will there be?
Test case explosion?
The number of valid test cases is the product of the number of valid equivalence classes per parameter!
Since it is seldom possible to use that many test cases, more heuristic rules are necessary to reduce the number of valid test cases
Rules of thumb
1. Combine the test cases and sort them by frequency of occurrence (typical usage profile or as advised by the client). Prioritise the test cases in this order so only the relevant test cases (or common combinations) are tested
2. Test cases including boundary values or boundary value combinations are preferred
3. Combine every representative of one equivalence class with every representative of the other equivalence classes (i.e. pairwise combinations instead of complete combinations) (Pairwise Combination: explained next slide!)
4. Ensure that every representative of an equivalence class appears in at least one test case. This is a minimum criterion.
5. Representatives of invalid equivalence classes should not be combined with representatives of other invalid equivalence classes
Pairwise Combination of Equivalence Classes
Consider testing multi-platform DVD playing software:
• Combinatorially, this would be 4 * 2 * 3 * 3 = 72 test combinations.
• To reduce this, start with a pairwise combination of two longest
rows (Platform and Hardware)
➡ we could have used (Platform and Display)
• So, for each value of Platform, we write down each value of Hardware
This gives us 4 * 3 = 12 rows.
Next, fill in the values of External display so that we cover all its pairings with both Platform and External display at the same time, down the column as in
Comment: this greatly reduces the number of cases to test and purportedly hits some 75%+ of actual defects. Research papers however show that though this is common practice, there is no scientific support and random choice may perform as well!
VSR DreamCar Example
Task: test the calculate_price function. Known information
double calculate_price {
double baseprice,
double special_price,
double extra_price,
int extras,
double discount
}
// base price of vehicle
// special model addition
// price of extras
// number of extras
// dealer discount
Now, we need to define the domain for every input parameter, and have valid and invalid values for each parameter for the equivalence classes based on the interface specifications.
double calculate_price (double baseprice, specialprice, extraprice; int extras, double discount) {
double addon_discount, result;
if (extras >= 3) else if (extras >= 5) else
addon_discount = 10;
addon_discount = 15;
addon_discount = 0;
if (discount > addon_discount) addon_discount = discount;
result = baseprice/100.0*(100-discount)
+ specialprice
+ extraprice/100.0*(100-addon_discount); return result;
}
Raw
we know more based on the functional specification: prices must be positive; extras must be positive or zero and controls the discount such that 10% if
extras >= 3 and 15% if extras >= 5 . There is no upper limit.
(dealer) discount is a percentage, and may be confirmed with the client. That is, between 0 and 100.
The more details and questioning from the tester the better refined and defined
the equivalence classes are, and the more accurate the testing will be
.
Partitioned
There are 18 equivalence classes in total.
We choose representative values. Mathematically, it is best to ensure they don’t overlap (takes time).
For efficiency, choose invalid values that are plausible and of low complexity.
We have 1 x 1 x 1 x 3 x 1 = 3 valid test cases plus 2 + 2 + 2 + 2 + 3 = 11 invalid cases for a total of 14 test cases
The next step is to work out what the expected output should be
resultant
discount is a general discount.
there is a discount off the supplementary equipment: 10% if >= 3 and 15% if >= 5 Dealer Discount is a % such that if sale < 15K no dealer discount, up to 20K 5%, up to 25K 7% and > 25K 8.5%
Exit Criteria for Equivalence Coverage
EC-coverage = (# of tested EClasses/total number of EClasses) × 100% In our example: EC-coverage = (14/18) × 100% = 77.78%
Test coverage may well be defined beforehand in the specifications. If, for example, 80% was desired, we know that our result of 77.78% can’t be improved anyway.
Equivalence classes should not be determined solely for inputs and outputs of methods and functions.
▪ They can also be prepared for internal values and states, time-dependent values (for example, before or after an event), and interface parameters.
▪ The method can thus be used in any test level.
We haven’t looked at dependencies or interactions between conditions. This is expensive and is done through domain analysis. (also called “context sensitive testing”)
Boundary Value Analysis
Boundary value analysis checks the borders of the equivalence classes.
On every border, the exact boundary value and both nearest adjacent values (inside and
outside the equivalence class) are tested.
The minimal possible increment in both directions should be used.
For floating-point data, this can be the defined tolerance. Therefore, three test cases result from every boundary.
If the upper boundary of one equivalence class equals the lower boundary of the adjacent equivalence class, then the respective test cases coincide
In some cases there does not exist a “real” boundary value because the boundary value belongs to an equivalence class.
▪ In such cases, it can be sufficient to test the boundary with two values: one that is just inside the equivalence class and another that is just outside the equivalence class
VSR Example
Equivalence Classes 3 & 4 are vEC3: 20000 < x < 25000 vEC4: x >= 25000.
For testing the common boundary (2500) the values 24999 and 25000 are chosen (assume only whole dollars). The value 24999 lies in vEC3 and is the largest possible value. The value 25000 is the least possible value in vEC4.
The values 24998 and 25001 do not give any more information.
When are the values 24999 and 25000 sufficient and when should we additionally use the
value 25001?
▪ 25001 will find the fault if for a totally wrong implementation eg if (x <> 25000)
▪ Can also be found in a code review because it does not check the boundary of a value area
Boundary Values for valid equivalence classes
which reads as delta, means “just a little bit”
extra
Inside the valid equivalence classes ([..]) we have 4 + 4 + 4 + 9 + 4 = 25 baseprice specialprice, extraprice and discount, each have 4 values
Recall: If 3 or more extras (not part of the special model) are chosen, discount = 10% on these particular items. If 5 or more extras discount is 15%. This gives us the ECs [0,1,2] and [3,4] and [5,6,maxint-1,maxint] (which corresponds to the 9 in the calculation above.
We had already included “1” and “3” (see test cases 1 & 2 four slides back) … see over
δ
We already had tested for extras = 1 and 3 above, so instead of 4 + 4 + 4 + 9 + 4 = 25 cases, we remove 2 and have 23
23 test cases in total
leading to the following test cases “extract” which includes the output result (we haven’t include boundary values outside the equivalence classes
The expected results of a boundary value test are often not clearly visible from the specification.
The experienced tester must then define reasonable expected results for her test cases
observations
Test 15 is not reasonable. Perhaps in a system test where such a thing might happen
Test 16 is like 15 but is about precision, but we prefer values like 0.005
Test 17 looks strange, discount of 99.99? prices are added. We might want to run
tests on the individual values with MAX_DOUBLE. Test 18 to 20 does that
We still need test cases for boundary values (extras = 5, 6, MAX_INT-1, MAX_INT and discount = 100.00) as per previous slide. More test cases are needed!
Testers need to speak to the client
If the tester communicates with the customer before finalising the test cases, and the value ranges of the parameters are then specified more precisely, the test will be less expensive (it is not smart to do “brute force all possible test automation”
The customer has now given the following information after discussion:
• The base price is between 10000 and 150000.
• The extra price for a special model is between 800 and 3500.
• There are a maximum of 25 possible extras, whose prices are between 50 and 750. • The dealer discount is maximum 25%
This simplifies matters for both the programmer and tester so that:
The above valid boundary values may be freely combined to test cases. One test case is needed for each value outside the equivalence classes.
The maximum price for extra items can’t be specified exactly because the dependence between the number of extras and the total price can’t be considered with equivalence classes. We used 25 × 750 = 18750 for extraprice because there is a maximum of 25 extras which can each cost up to $750
Heuristic guide for good practice
• For an input domain, the boundaries and adjacent values outside the must be considered. Domain: [-1.0; +1.0], test data: -1.0, +1.0 and -1.001, +1.001. Precision is problem specific
• If an input file has a restricted number of data records (for example, between 1 and 100), the test values should be 1, 100 and 0, 101
• If outputs are the concern and output test object is an integer between 500 and 1000. Test outputs should be: 500, 1000, 499, 1001. It can be difficult to identify input test data to get the required outputs. Generating invalid outputs may even be impossible, but you may find defects by thinking about it
• For ordered sets, the first element and the last element are of special interest for the test
• If data structures are given as input or output (for instance, an empty list or zero), tables can be considered as boundary values
…
• For numeric calculations, values that are close together, as well as values that are far apart, should be taken into consideration as boundary values
• For invalid equivalence classes, boundary value analysis is only useful when different exception handling for the test object is expected, depending on an equivalence class boundary
• Extremely large data structures, lists, tables, etc. should be chosen. For example, you should exceed buffer, file, or data storage boundaries, in order to check the behaviour of the test object in extreme cases
• Empty and full lists and the first and last elements are of interest because they often show failures due to incorrect programming (Off-by- one problem)
Text exit criteria
Similar to Equivalence Classes. BV = Boundary Values
BV-Coverage = (# of tested BV / total BV) × 100%
• Overlapping values of adjacent equivalence classes are counted as only one boundary value because only one test case with the respective input value is used
• Boundary value analysis should be used together with equivalence class partitioning because faults can be found more often at the boundaries of the equivalence classes than far inside the classes
• The technique requires a lot of creativity to define appropriate test data at the boundaries. It is often neglected; it is not trivial to do.
State Transition Testing
Till now we have looked at values of variables. Is that enough? We also should be able to see the conditions under which a variable changes and model this and thereby test it. A formalism to do so is through a finite state machine (FSM)
A FSM is an abstract machine for which
▪ the number of states and input symbols are both finite and fixed
▪ consists of states (nodes), transitions (links), inputs (link weights), and outputs (link weights)
▪ There are a finite number of internal configurations, called states
▪ The state of a system implicitly contains the information that has resulted from earlier inputs and that is necessary to find the reaction of the system to new inputs
Example: A FSM for a stack
Note: one needs to specify what happens if we push when full. Calling top and pop in the empty state has also not been specified
…
Different levels of test intensity can be defined for a state transition test. A minimum requirement is to get to all possible states.
▪ In the stack example, these states are empty, filled, and full.
Test 1: initialise [empty], push [filled], push, push, push [full]
▪ Though all states have been seen, all functions have not
Test 2: initialise [empty], push [filled], top, pop [empty], delete
▪ Here we have all the functions, but we don’t have all the states (full!)
State transitions allow us to model what-if-then, and this can be traced back to requirements (and to fill in gaps)
From FSM to transition tree
In order to see the (finite) test cases it is desirable to have finite sequences of traversal ie without cycles. We
achieve this by transforming to a tree structure where 1.all states must be reached and
2.all transitions from the transition diagram must appear
Algorithm (input: state transition diagram; output: transition tree):
1. The initial or start state is the root of the tree
2. For every possible transition from the initial state to a following state in the state transition diagram, the transition tree receives a branch from its root to a node, representing this next state
3. step 2 is repeated for every leaf in the tree (every newly added node) until one of the following two end conditions is fulfilled:
a) The corresponding state is already included in the tree on the way from the root to the node. This end condition corresponds to one execution of a cycle in the transition diagram.
b) The corresponding state is final and therefore has no further transitions to be considered
Transition tree for stack
Note how we have Empty -> push -> filled and then push again to filled. This is 2 a) so we stop at that second filled ditto for the push full
What about against specifications?
• It is a violation of the specification if functions are called in states where they should not be used (e.g., to delete the stack while in the “full” state).
• A robustness test must be executed to check how the test object works when used incorrectly.
• It should be tested to see whether unexpected transitions appear.
➡ The test can be seen as an analogy to the test of unexpected input values
• The transition tree should be extended by adding a branch for every function from every node. This means that from every state, all the functions should be executed or at least an attempt should be made to execute them. This results in the extended tree
• It definitely makes sense to define which reactions to expect when trying pop and top calls for an empty stack. A reasonable reaction would be, for example, an error message
• If we have a GUI then we could use this idea for system testing
The nodes are windows that move through states which are buttons that are pressed etc
Summarising
To completely define a state-based test case, the following information is necessary:
• The initial state of the test object (component or system) • The inputs to the test object
• The expected outcome or expected behaviour
• The expected final state
Further, for each expected state transition of the test case, the following aspects must be defined:
• The state before the transition
• The initiating event that triggers the transition
• The expected reaction triggered by the transition • The next expected state
Heuristic Guide
• Evaluate the state transition diagram from a testing point of view when writing the specification. If there are a high number of states and transitions, indicate the higher test effort and push for simplification if possible.
• Check the specification, as well, to make sure the different states are easy to identify and that they are not the result of a multiple combination of values of different variables.
• Check that the state variables are easy to display from the outside.
.
include functions that set, reset, and read the state for use during testing
It is a good idea to
Test Exit Criteria
Criteria for test intensity and for exiting can also be defined for state transition testing: • Every state has been reached at least once.
• Every transition has been executed at least once.
• Every transition violating the specification has been checked
For highly critical applications, more stringent state transition test completion criteria can be defined:
• All combinations of transitions
• All transitions in any order with all possible states, including multiple executions in a row
Cause effect graphing
• We have looked at techniques which observe the effect of input values independently in order to generate test cases
• Dependencies between variables and the effect on output is something Myers proposed in defining a method logical relationships between the causes and their effects in a component or a system.
➡ These relationships should be found in the specifications
• The causes and effects are boolean expressions (evaluating to true or
false)
• Consider now an ATM
ATM Example: withdrawing money
Causes/Conditions:
• The bank card is valid.
• The PIN is entered correctly.
• The maximum number of PIN inputs is three.
• There is money in the machine and in the account. Conditions/Actions:
• Reject card.
• Ask for another PIN input.
• “Eat” the card.
• Ask for an other amount.
• Pay the requested amount of money.
ATM Cause effect graph
Transform to a decision table
1. Choose an effect.
2. Looking in the graph, find combinations of causes that do have this effect
and combinations that do not have this effect.
3. Add one column into the table for every one of these cause-effect
combinations. Include the caused states of the remaining effects.
4. Check to see if decision table entries occur several times, and if they do, delete them.
(a little vague, I will explain!)
Create the matrix
• We have 4 conditions/causes (Bank card Valid through to Money available). • Each of these is boolean, and can therefore have 2 possible values of True
or False.
➡ So, we have 24 = 16 combinations of True and False combinations.
decision table creation process
Decision Table
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Causes/ Conditions
bank card valid?
pin correct?
third pin attempt?
money available?
Effects/ Actions
reject card
ask for new pin
“eat” card
ask for new amount
pay cash
Decision Table
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Conditions
bank card valid?
T
T
T
T
T
T
T
T
F
F
F
F
F
F
F
F
pin correct?
T
T
T
T
F
F
F
F
F
F
F
F
T
T
T
T
third pin attempt?
T
T
F
F
T
T
F
F
F
F
T
T
F
F
T
T
money available?
T
F
T
F
T
F
T
F
F
T
T
F
F
T
T
F
Actions
reject card
ask for new pin
“eat” card
ask for new amount
pay cash
Decision Table
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Conditions
bank card valid?
T
T
T
T
T
T
T
T
F
F
F
F
F
F
F
F
pin correct?
T
T
T
T
F
F
F
F
F
F
F
F
T
T
T
T
third pin attempt?
T
T
F
F
T
T
F
F
F
F
T
T
F
F
T
T
money available?
T
F
T
F
T
F
T
F
F
T
T
F
F
T
T
F
Actions
reject card
👍
👍
👍
👍
👍
👍
👍
👍
ask for new pin
👍
👍
👍
👍
“eat” card
👍
ask for new amount
👍
👍
pay cash
👍
👍
Decision Table
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Conditions
bank card valid?
T
T
T
T
T
T
T
T
F
F
F
F
F
F
F
F
pin correct?
T
T
T
T
F
F
F
F
F
F
F
F
T
T
T
T
third pin attempt?
T
T
F
F
T
T
F
F
F
F
T
T
F
F
T
T
money available?
T
F
T
F
T
F
T
F
F
T
T
F
F
T
T
F
Actions
reject card
👍
👍
👍
👍
👍
👍
👍
👍
ask for new pin
👍
👍
👍
“eat” card
👍
ask for new amount
👍
👍
Wha
t is
common a
bout those
pay cash
👍
👍
g
rou
ped by
colour? 🔍
Now nicely reduced to five test cases!
There are programs you can use to plug in a decision table and it will check and reduce the table to the essentials as we have done. In Java, for example Drools a BRMS (Business rule management system)
We can still refine this with one more step where we use “-“ to indicate a value we don’t care about
Decision Table
1
2
5
6
9
Conditions
bank card valid?
T
T
T
T
F
pin correct?
T
T
F
F
F
third pin attempt?
T
T
T
T
F
money available?
T
F
T
F
F
Actions
reject card
👍
ask for new pin
👍
“eat” card
👍
ask for new amount
👍
pay cash
👍
Final version
Decision Table
TC1
TC2
TC3
TC4
TC5
Conditions
bank card valid?
T
T
T
T
F
pin correct?
T
T
F
F
—
third pin attempt?
—
—
T
T
—
money available?
T
F
—
—
—
Actions
reject card
👍
ask for new pin
👍
“eat” card
👍
ask for new amount
👍
pay cash
👍
👀
TC means Test Case
Test exit criteria
• A minimum requirement is to execute every column in the decision table by at least one test case.
➡ This verifies all sensible combinations of conditions and their corresponding effects
• The systematic and very formal approach in defining a decision table with all possible combinations may show combinations that are not included when other test case design techniques are used
• Errors can result from optimisation of the decision table when input and conditions are mistakenly left out
• The table may grow quickly and lose readability
➡ In practice, one needs tool support otherwise testers need their own debuggers (yuck!)
Use-case testing
• There are perhaps too many notations to cover!
• The idea though is they specific business rules that should be tested
• They are especially useful for system and acceptance testing
• “Alternatives” are usually expressed using the extend relation. For example: For example, imagine Withdraw Cash is a use case of an ATM. Assess Fee would extend Withdraw Cash and describe the conditional “extension point” user doesn’t bank at the ATM’s owning bank
• A possible criterion is that every use case or every possible sequence of use cases in the diagram is tested at least once by a test case.
➡ Since alternatives and extensions are use cases too, this criterion also requires their execution.
Other black box techniques
Syntax testing describes a technique for identifying test cases that can be applied if a formal specification of the syntax of the inputs is available.
➡ Syntax testing would be used for testing interpreters of command languages, compilers, and protocol analysers, for example.
➡ The syntax definition is used to specify test cases that cover both the compliance to and violation of the syntax rules for the inputs
A compiler is a syntax tester!
…
Random testing generates values for the test cases by random selection. If a statistical distribution of the input values is given (e.g., normal distribution), then it should be used for the selection of test values. This ensures that the test cases are preferably close to reality, making it possible to use statistical models for predicting or certifying system reliability
➡ When? Absence of specifications
➡ Why? It is also used as way to avoid biased testing.
…
The term smoke test is often used in software testing.
• It is very useful and can be neglected because it isn’t formal!
• A smoke test is commonly understood as a “quick and dirty” test that is primarily aimed at verifying the minimum reliability of the test object.
• The test is concentrated on the main functions of the test object.
• The output of the test is not evaluated in detail. It is checked only if the test object
crashes or seriously misbehaves.
• A test oracle (source for expected result) is not used, which contributes to making this
test inexpensive and easy
• A smoke test is often used to decide if the test object is mature enough to proceed
with further testing designed with the more comprehensive test techniques.
• Smoke tests can also be used for first and fast tests of software updates
Closing remarks about black box testing
• The basis of all black box techniques is the requirements or specifications of the system or its components and how they collaborate
• Black box testing will not find problems where the implementation is based on incorrect requirements or faulty design specification because there will be no deviation between faulty specification/design and observed results
➡ The test object will execute as the requirements or specifications require, even when they are wrong
• Testers need to be on the alert for wrong specs. ➡ How?
• Black box testing cannot reveal extra functionality that exceeds the specifications
• Verification of the functionality of the test object is the thrust of black box testing. It is indisputable that the highest priority is that the software work correctly. Thus, black box techniques should always be applied
White box testing
• The code must be available
• The expected results should be determined using the requirements or
specifications, not the code.
• The foundation of white box techniques is to execute every (or. pre-agreed %)
part of the code of the test object at least once.
• Flow-oriented test cases are identified, analysing the program logic, and then
they are executed.
• The primary goal of the technique is then to achieve a previously defined coverage of the statements during testing
White box techniques
or branch testing Testing of conditions
▪ Condition testing (simple condition test or coverage)
▪ Multiple condition testing (minimal multi condition test/coverage) ▪ Condition determination testing
Path testing
Note: ISTQB Foundations only describes the techniques in bold ↑
Statement testing
Decision testing
Example
Every node (statement) can be reached by the minimal test case
a, b, f, g, h, d, e
We compare the expected result with the specification (not code!)
Test Exit Criteria (Also called C0 (C zero) coverage)
Statement Coverage =
(# of executed statements / total # of statements ) * 100%
Value of technique
• If complete coverage of all statements is required and some statements cannot be executed by any test case, this may be an indication of unreachable source code (dead statements/unreachable code)
• An if statement without an else clause will meet up after the endif. This may or may not indicate some empty statements for the else.
• Tools are used to measure the statement coverage
example
if condition then s1
endif
s3
if condition then s1
else
s2
endif
s3
Decision branch testing
Here we want to traverse not just the nodes (statements) but also all the edges, each conditional path.
Contrary to statement coverage, for branch coverage it is not interesting if, for instance, an IF statement has no ELSE-part. It must be executed anyway. (the dotted d in the previous slide)
for loops, both
▪ execution of the loop body,
▪ bypassing the loop body and
▪ returning to the start of the loop
need to be tested Returning to our example …
a,b,f,g,h,d,e was statement coverage
For decision branch coverage:
▪ edges c, i and k need to be engaged
▪ c and k are empty branches of a condition and i is the return to beginning of the loop.
So, we have:
▪ a,b,c,d,e
▪ a,b,f,g,i,g,h,d,e ▪ a, k, e
Some edges unavoidably have been executed more than once: eg edges a and e. This is unavoidable.
Test exit criteria
It is reasonable to record which branches have been executed in which test case in order to find wrong execution flows. This helps to find faults, especially missing code in empty branches
Branch Coverage = C1 coverage
(# of executed branches / total # of branches) × 100%
If we execute
▪ a,b,f,g,h,d,e
▪ a,b,c,d,e
▪ a,b,f,g,i,g,h,d,e
Then we miss out on the k branch, and have 9/10 90% branch coverage but 100% statement coverage.
The exit criteria will need to specify an acceptable percentage.
Branch coverage of 100% guarantees 100% statement coverage, but not vice versa. Thus, branch coverage is a stronger criterion and 100% branch coverage means a test is deemed sufficient.
Test of conditions
An example for a combined condition is x > 3 OR y < 5.
The condition consists of two conditions (x > 3 ; y < 5) connected by the logical operator OR (⌵).
The goal of condition testing is that each partial condition is evaluated once, resulting in each of the logical values.
The test data x = 6 and y = 8 results in the logical value true for the first condition
(x > 3) and the logical value false for the second condition (y < 5). The logical value of the complete condition is true (true OR false = true).
The second pair of test data with the values x = 2 and y = 3 results in false for the first condition and true for the second condition. The value of the complete condition results in true again (false OR true = true).
Both parts of the combined condition have each resulted in both logical values. ▪ The result of the complete condition, however, is equal for both combinations.
Condition coverage is therefore a weaker criterion than statement or branch coverage because it is not required that different logical values for the result of the complete condition be included in the test
▪ consider if there is defect in the code for “y” (in general, y is an boolean expression)
Multiple Condition Testing and Coverage
Multiple condition testing requires that all true-false combinations of the atomic partial conditions be exercised at least once.
Four combinations of test cases are possible with the test data from the previous example for the two conditions
(x > 3 ; y < 5)
▪ x=6(T),y=3(T),x>3ORy<5(T) ▪ x=6(T),y=8(F),x>3ORy<5(T) ▪ x=2(F),y=3(T),x>3ORy<5(T) ▪ x=2(F),y=8(F),x>3ORy<5(F)
Comments
• The evaluation of the complete condition results in both logical values.
• Multiple condition testing meets the criteria of statement and branch coverage.
• It is a more comprehensive criterion that also takes into account the complexity of combined conditions.
• This is a very expensive technique due to the growing number of atomic partial conditions that make the number of possible combinations grow exponentially (to 2n, with n being the number of atomic partial conditions)
• A problem results from the fact that test data cannot always generate all combinations.
See next example ...
Example
For the combined condition of 3<=x AND x<5 not all combinations with the according values for the variable x can be produced because the parts of the combined condition depend on each other
x=4: 3 <= x(T), x<5(T), 3 <= x AND x<5(T) x=8: 3 <= x(T), x<5(F), 3 <= x AND x<5(F) x=1: 3 <= x(F), x<5(T), 3 <= x AND x<5(F) x=?: 3 <= x(F), x<5(F),
combination not possible because the value x shall be smaller than 3 and greater than or equal to 5 at the same time!
Condition Determination Testing / Minimal Multiple Condition Testing
Guidelines:
Not all combinations must be included; however, do include every possible combination of logical values where the modification of the logical value of an atomic partial condition can change the logical value of the whole combined condition
Equivalently, for a test case, every atomic partial condition must have a meaningful impact on the result
▪ Test cases in which the result does not depend on a change of an atomic partial condition need not be designed
Example: (x > 3 OR y < 5)
22 possibilities
1. x=6(T),y=3(T),x>3ORy<5(T) 2. x=6(T),y=8(F),x>3ORy<5(T) 3. x=2(F),y=3(T),x>3ORy<5(T) 4. x=2(F),y=8(F),x>3ORy<5(F)
For the first combination: If the logical value is wrongly calculated for the first condition (an incorrect condition is implemented), then the fault can change the logical value of the first condition part from true (T) to false (F). But the result of the complete condition stays unchanged (T). The same applies for the second partial condition.
For the first combination, incorrect results of each partial condition have no effect on the result of the complete condition and failures will not be visible. Consequently, the test with the first combination can be left out.
What about the other three cases?
Advice
Because of the weak significance, condition testing should not be used.
For complex conditions, condition determination testing should be applied because the complexity of the conditional expression is taken into account for test case design.
▪ The method also subsumes statement and branch coverage; they need not be checked in addition.
Combinations of logical expressions are especially defect prone. Thus, a comprehensive test is very important
▪ It may be very expensive to choose the input values in such a way that a certain part of the condition gets the logical value required by the test case
The intensive test of complex conditions can possibly be omitted if they have been subjected to a review where correctness is verified
What happens here?
...
Flag = (A || (B && C)); If (Flag)
...; else ...; ...
Seems like a simple condition but it is not! Compilers may swap the order
Compilers may only execute one partial condition!
Path testing and coverage
Recall that we (black box) derived test cases using equivalence classes for valid and invalid values of sales price.
Code + Control Flow
test cases
price = calculate_price(10000.00,2000.00,1000.00,3,0); test_ok = test_ok && (abs(price–12900.00) < 0.01);
price = calculate_price(25500.00,3450.00,6000.00,6,0); test_ok = test_ok && (abs(price-34050.00) < 0.01);
Corresponds with:
▪ TestCase01:a,b,c,j,m,n ▪ TestCase02:a,b,c,j,m,n
Notice they are two test cases with the same path because “c” doesn’t specify which statement.
The two tests covered 6 out of 14 branches = 43%
...
To increase test coverage, the following further test cases are specified:
price = calculate_price(10000.00,2000.00,1000.00,0,10); test_ok = test_ok && (abs(price–12000.00) < 0.01);
price = calculate_price(25500.00,3450.00,6000.00,6,15); test_ok = test_ok && (abs(price–30225.00) < 0.01);
These test cases cause the execution of the following edges of the graph:
• Test Case 03: a, d, g, h, i, j, k, l, n
• Test Case 04: a, b, c, j, k, l, n
• This is 83% of branches
• Note though that edges e and f have not yet been executed
• But this implies we would need extras not to be >= 3 and then extras >= 5. Impossible! extras can’t be < 3 and >= 5. There is a defect in the code!
General observations about white box testing
• Adequate test case design techniques can be chosen and applied depending on the complexity of the program structure.
➡ The intensity of the test depends on the source code and the selected technique
• The white box techniques are appropriate for the lower test levels. Not sensible for system
testing, for example.
• The concept of coverage can be applied to other test levels above the code level. For example, during an integration test, we can assess what percentage of modules, components, or classes are executed during the test. This results in module, component, or class coverage
• White box techniques can verify only code that exists, that is, requirements that are implemented in the program, not code that should be there but isn’t.
➡ In other words, it can’t help us with missing requirements which needs other techniques
Exploratory Testing
• The basis of this method is the skill, experience, and knowledge of the tester to select test cases that uncover expected problems and their symptoms (failures).
• A systematic approach is not used.
• The test cases are based on where faults have occurred in the past or the tester’s ideas of
where faults might occur in the future.
• This type of test case design is also called error guessing and is used very often in practice
• The technique is also applicable when time is severely restricted because it uses much less time than other techniques.
• The approach is mainly based on the intuition and experience of the tester.
• Based on my research it is used 10-20% of the time
…
• The possible elements of the test object (its specific tasks and functions) are “explored.”
• Only a few test cases are executed and their results are analysed
• By executing the test cases, the “unknown” behaviour of the test object will be determined further. Anything considered “interesting,” as well as other information, is then used to determine the next test cases.
• It becomes clearer what the test object does and how it works, which quality problems there could be, and which test techniques and expectations to the program should be fulfilled.
• Exploratory testing is applied to certain elements of the program (certain tasks or functions).
• The elements are further decomposed. The term test charter (=a statement of test objectives, and possibly test ideas) is used for such smaller parts
Test Charter
• A Test Charter is the documentation of the goal/objective for a test session, where a test session is an uninterrupted period of time executing test cases.
• When executing test charters, the following questions are of interest ➡ Why? What is the goal of the test run?
➡ What? What is to be tested?
➡ How? Which testing method should be used?
➡ What? What kind of problems should be found
• Intuitive test case determination should not be applied as the primary testing technique.
➡ This technique should be used to complete the test cases and to support the systematic test design techniques
Applied to our running example
A new car configurator.
▪ the testers are familiar with the previous system.
▪ Many of have tested this system before. They know the weaknesses the system had and they know the problems the car dealers had with the operation of the old software (from hotline data and from discussions with car salespeople).
▪ Employees from the marketing department know for the business-related test which vehicles in which configurations are sold often and which theoretically possible combinations of extra equipment might not even be shippable.
▪
▪ The test manager knows which of the developer teams act under the most severe time pressure and even work on weekends. Hence, she will test the components from these teams more intensively
They use this experience to intuitively prioritise the systematically identified test cases
and to complete them by additional test cases.
Exit Criteria
Unlike the systematic techniques, a criterion for termination cannot be specified.
If there exists a list that may even be beneficial to developers because it indicates in advance what potential problems and difficulties might occur, then this is useful.
If the previously mentioned list exists, then a certain completeness can be verified against that list
Choosing the technique
1. The kind of test object
▪ The complexity of the program text can vary considerably. If, for example, decisions in the program are combined from several atomic conditions, branch coverage is not sufficient. A suitable technique to check the conditions should be chosen depending on the criticality and the risk in case of failure
2. Formal documentation and the availability of tools
▪ If specification or model information is available in a formal document, it can be fed directly into test
design tools, which then derive test cases.
▪ Agile will not have this benefit! 3. Conformance to standards
▪ Industry and regulatory standards may require use of certain test techniques and coverage criteria 4. Tester Experience
▪ Tester experience may lead to the choice of special techniques. A tester will, for example, reuse techniques that have led to finding serious faults earlier
…
5. Customer Wishes
▪ The customer may require the use of specific test techniques and even the test coverage to
be achieved
▪ This may lead to fewer failures in acceptance testing
6. Risk Assessment
▪ The expected risk dictates more or less the test activities, that is, the choice of techniques and the intensity of the execution
7. Additional Factors
▪ there are factors like the availability of the specification and other documentation, the knowledge and skill of the test personnel, time and budget, the test level, and previous experience with what kind of defects occur most often and with which test techniques these have been found
So what do you need to know well
You need to know and be able to do exercises surrounding the concepts: • Test Technique
• Black Box
• White Box
• Coverage
• Experience Based
• Equivalence Partitioning
• Boundary value analysis
• Decision table testing
• State transition testing
• Use Case testing
• Statement Coverage
• Decision Coverage
• Error Guessing
• Exploratory testing
• Checklist based testing
And have read the book chapter carefully. See the Chapter Review and exercises.