—ISYS1087/5 Software Testing Chapter 3
Static Testing
A/Prof I. Balbin
Lecture Session: Week 4
Aims
Lecture Aims
In this lecture we look at aspects of testing which do not involve the dynamic execution of the system or parts of the system in order to detect defects.
Static investigations like reviews and tool-supported analysis of code and documents can be used successfully for improving quality. This chapter presents the possibilities and techniques.
Dynamic techniques are the focus of the next chapter.
Static Analysis vs Static Testing
Almost the same
Analysis means some testing tool is applied without executing the code
Static Analysis is used in Security testing. Look at code for weaknesses. Hackers do this. You don’t find the weaknesses by executing.
Used in mission critical systems eg health and medical Its natural home is during reviews
How is Static Analysis different?
Opposite to dynamic testing, the test object is not provided with test data and executed but rather analysed.
This can be done using one or more persons for an intensive investigation or through the use of tools.
Such an investigation can be used for all documents relevant for software development and maintenance.
Tool-supported static analysis is only possible for documents with a formal structure.
Applicable to standard SDLC but should be incorporated into sprints
The goal of examination is to find defects and deviations from the ▪ existing ,
▪ to comply with, or even
▪ the project plan.
The basic idea is defect prevention: defects and deviations should be recognised as early as possible before they have any effect in the further development process where they would result in expensive rework
specifications
standards
Errors, Defects and Failures
• An error is a mistake by a human being
Coding errors or poor requirements or poor user story
• A defect is something that is wrong—variance between actual and expected result.
Testing is the process of seeking to find defects
Defect accepted by the Dev team becomes a bug Defect accepted by the Analysis team becomes a “bug”
• A failure is when the system does not perform as it should
Best if found by the tester
Could be found by the client
Static testing can’t find failures—nothing is executed—but it can find defects.
➡
➡ ➡ ➡
➡ ➡
…
Like dynamic testing, static analysis looks for defects in software source code and software models eg UML, user stories, requirements docs, manual etc
Unlike dynamic testing, static analysis is performed without actually executing the system
Static analysis involves analysis of the system or its components by a tool, while dynamic testing does not always involve tools
Static analysis can find defects that are hard to find or isolate in dynamic testing ▪ Examples include maintainability issues, unsafe pointer use
Isolation is easier because you find the bug, not the symptom
Typical static code analysis concerns
• Referencing a variable with an undefined value
• Inconsistent interface between modules and components • Variables that are never used
• Missing or wrong logic
• Excessive complexity
• Unreachable (dead) code
• Programming standards violations
• Security vulnerabilities
• Syntax violations of code and software models
Static testing is also unique
You cannot use Dynamic Testing to discover
▪ Code redundancy (although dynamic testing might make you look for this sort of thing)
if (condition) return true;
else
return false;
instead of simply writing
return (condition);
▪ Dead code
if (true) {
// always chosen … a situation where something will always be true
} else {
// never chosen -> Dead code
}
▪ Violations of coding standards (not to be sniffed at)
In a modern agile context
• In agile, “moving left or “shifting left” refers to testing earlier in development rather than later • Linters are a type of static analysis that works to find construct behaviour that could
potentially become run-time bugs in the future
• Check out https://www.shellcheck.net for finding bugs in your shell scripts and https:// github.com/vim-syntastic/syntastic/ a syntax checking plugin for Vim for just about any language that exists.
• Often, these need to be integrated into a development process with minimal friction. The solution needed to be compatible with current workflows which are largely centralised in GitHub. Any tool would need to fit into continuous integration and continuous delivery (CI/ CD) pipelines for releasing code changes to production quickly and regularly.
One approach to this is on the next slide
➡
LGTM
• Use LGTM https://github.com/marketplace/lgtm/ on every pull request from github, including automatically running checks on each commit and sending a CodeQL alert when an error is flagged. If an alert is triggered, the pull request is blocked from merging until resolved.
LGTM is a code analysis platform for identifying vulnerabilities and preventing them from reaching production.
LGTM automatically runs 1600+ standard analyses contributed by researchers from the Semmle Security Research Team and the customer community, including Microsoft, Google, Uber and Mozilla
➡
➡
Codacy
• Use Codacy https://github.com/marketplace/codacy/ to standardise code quality and alert to vulnerabilities.
“Codacy is an automated code analysis/quality tool that helps developers ship better software, faster. With Codacy, you get static analysis, cyclomatic complexity, duplication and code unit test coverage changes in every commit and pull request”
• As a day-to-day workflow, Codacy can immediately and automatically test for errors each time a developer creates a pull request.
When it triggers an error warning, the engineer gets immediate feedback and can work on their own time to resolve the issue.
They can also collaborate with the rest of the team directly in GitHub or Codacy’s integrated web app
➡
➡
➡
Coverity
• Use Coverity https://scan.coverity.com/
Coverity Scan is a service by which Synopsys provides the results of analysis on open source coding projects to open source code developers that have registered their products with Coverity Scan.
Find and fix defects in Java, C/C++, C#, JavaScript, Ruby, or Python open source project for free
• Run a nightly batch and run Coverity on every major feature release.
Use this static analysis for insights that might have not been flagged earlier in the process.
Coverity integrates well into a CI/CD pipeline as a last step of automated testing and helps find code defects and threats early in the development process
➡
➡
➡ ➡
Only for Agile?
Heck no! They pipe into a GitHub CI/CD but are useful across the board!
Defect Removal Metrics
Why do we have these? We need an exit criteria for our testing, of course. Calculated to give an idea of the efficiency of defect squashing prior to release.
Defect removal efficiency (DRE) is calculated prior and at the time of release as a ratio of
defects resolved to total number of defects found.
We can monitor Defect Detection Percentage using the following
dimensions:
▪ Affectedversionand ▪ Priority.
But I’m a software engineer … dammit
The importance of measurements is summarised by Lord Kelvin:
“When you measure what you are speaking about and express it in numbers, you know something about it, but when you cannot express it in numbers your knowledge about is meagre and unsatisfactory”.
The need to have appropriate measurements is summarised by Albert Einstein:
“Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted”.
Detection vs Injection
A defect is confirmed and agreed upon (not just reported)
Is a defect detected in one phase the same as a defect injected during that
same phase?
No. A defect can be detected in the System Testing phase but the cause of the defect can be due to wrong design. Hence, the injected phase for that defect is the Design phase
Saving
…
It’s important to identify the phase where a defect was found so that data can be normalised according to priority.
A good figure to aim for is at least an 85% DRE
continued
Unfortunately, there are other definitions of DRE. It is not standardised to my knowledge. How do we determine the total number of defects within a solution?
Defect Counting – combine the “number of defects found during testing” with the “number of defects identified in production, traced back to the project”.
This method will always give a lower value than the true number of defects introduced by the project as
▪ Not all defects manifest themselves as failures in Production
▪ Cosmetic and Usability related defects are often less likely to be raised in Production.
▪ In order for the DRE metric to be useful it should be derived as close as possible to when the system exited test, however many functions may not be executed in production until the system has been live for a number of years.
Perhaps we could cut off the Production Defect count “after the system has been live for three months” as rule of thumb?
In Agile
How do you find harmony between prevention processes such as static testing and agile practices?
It needs a streamlined and simplified inspection process while still capturing critical data to prevent problems from escaping to production.
Needs steps to overcome the organisational and cultural barriers that keep teams from realising the benefits of inspections aka static testing
Teams that don’t have testers or testers mindsets lose out as they skimp on static testing and miss the savings via prevention
Reviews
Reviews apply human analytical capabilities to check and evaluate complex issues. A review can be formal or informal. It is a type of static testing.
It overcomes cognitive bias.
▪ Intensive reading and trying to understand the examined documents is the key.
In general, an inspection, is a component of a review.
▪ An inspection is a static analysis technique that relies on visual examination of development
products to detect errors, violations of development standards, and other problems.
▪ Types include code inspection; design inspection, architectural inspections, Testware inspections etc
Every review has a clear goal, which is formulated beforehand.
▪ The “right” people are chosen as review participants based on the review objective as well as on their subject knowledge and skills
Planning your review
• define scope
• estimate effort/time
• select people and roles
• define entry and exit criteria (formal reviews—inspections)
• check entry criteria before it starts!
eg has gone through some linting or product author is ready or XYZ have been completed in a previous iteration
➡
Individuals—preparing
• Review all parts of the documents and artefacts in context (test basis)
• Note potential defects, recommendations and questions
• Checking rate is the number of documents/artefacts you can be expected to do in an hour.
Need to have a handle on this to manage the team/work.
• Decisions on acceptable exit criteria—# of issues, severity
• Create defect reports, fix, communicate, track changes, record new status, communicate
• Keep metrics on # of defects, average, time spent, etc this feeds into intelligence on managing the testing effort, now and on other projects. It’s not science.
➡
The verifying examinations at the end of a phase in the general V-model normally use reviews (so-called phase exit reviews).
Acceptance of user stories and more in Agile use reviews
Eliminating defects and inconsistencies leads to improved documented quality and positively influences the whole development process because development is continued with documents and artefacts that have fewer or even “no” defects
Types of Reviews
Informal: no real process (hallway chats, buddy tests, pair programming), yet useful, cheap, popular
Technical: documented and defined defect removal process, involving technical experts but
Walkthrough: “walks through” review item
Inspection: a trained moderator (other than the author) leads the inspection team (with defined roles) through a formal inspection process (rules, checklists, entry and exit criteria), which includes gathering defect removal metrics
When walkthroughs, technical reviews or inspections are performed by a peer group, the review may be called a peer review
not managers
author
Benefits of Reviews
Cheaper defect elimination. If defects are found and eliminated early, productivity in development increases because fewer resources are needed for defect identification and elimination later. These resources can instead be used for development.
Shortened development time. If defects are recognised and corrected early, costs and time needed for executing the dynamic tests (next chapter) decrease because there are fewer defects in the test object.
Because of the smaller number of defects, cost reduction can be expected during the whole product life.
▪ For example, a review may detect and clarify inconsistent and imprecise customer requests in the requirements. Foreseeable change requests after installation of the software system can thus be avoided
▪ For example, some user stories need to be modified and new ones created
…
During operation of the system, a reduced failure rate can be expected.
As the examinations are done using a team of people, reviews lead to mutual learning.
▪ People improve their working methods, and reviews will thus lead to enhanced quality of later products.
Because several persons are involved in a review, a clear and understandable description of the facts is required.
Often the necessity to formulate a clear document lets the author find forgotten issues.
The whole team feels responsible for the quality of the examined object. The group will gain a common understanding of it.
Pitfalls to avoid in reviews
In a badly moderated review session, the author may get into a psychologically difficult situation, feeling that she as a person (ad hominem) and not the document is subject to critical scrutiny
▪ Motivation to subject documents to a review will thus be destroyed
▪ Concretely expressing the review objective, which is improving the document, is helpful
Benefits
The costs caused by reviews are estimated to be 10–15% of the development budget.
Savings are estimated to be about 14–25%
▪ The extra effort for the reviews themselves is included in this calculation
Done properly, more than 70% of the defects in a document can often be found and repaired before they are unknowingly inherited by the next work steps
Since the cost goes up as the project proceeds, it is plausible that defect cost in development is reduced by 75% and more
Agile Case Study: CISCO
Code review is essential to their legendary quality control. Cisco participated in the largest case study ever done on lightweight code review process. Pain:
• Every code review required significant preparation.
• Author had to upload files from Perforce® and select 1-4 reviewers
• Files were reviewed using before/after side-by-side view
• Comments and defects threaded by file, line number
• Review rework required that authors fix defects and upload fixes for verification
• Fixes verified before review is complete
Goals:
• All code must be reviewed before checked into version control
• No in-person meetings
• Process is enforced by tools
• Metrics collected automatically
• Review-level and summary-level reporting
Benefits:
• By implementing lightweight code reviews, Cisco saved, on average, 6.5 staff hours per review.
• Their results drove definitive best practices for code reviews.
Results:
• Cisco proved that lightweight reviews were just as effective as formal inspections at uncovering defects.
Generic characteristics of good review
We will see the different types of review in a few overheads time There are six steps
▪ Planning
▪ Kickoff
▪ Individual preparation ▪ Review Meeting
▪ Rework
▪ Follow-up
Protocol
The protocol contains a list of the issues/findings that were discussed during the meeting.
An additional review summary report should collect all important data about the review itself, i.e.,
▪ the review object,
▪ the people involved,
▪ their roles,
▪ a short summary of the most important issues, and
▪ the result of the review with the recommendation of the reviewers.
Planning
The review leader selects technically competent staff and assembles a review team.
In cooperation with the author of the document to be reviewed, they makes sure that the carefully chosen document is in a reviewable state, i.e., it is complete enough and reasonably finished.
In formal reviews, entry criteria (and the corresponding exit criteria) may be set.
▪ A review should continue only after any available entry criteria has been checked.
A review is more successful when the examined document is read from different viewpoints or when each person checks only particular aspects.
▪ The viewpoints or aspects to be used should be determined during review planning.
A review might not involve the whole document. Parts of the document in which defects constitute a high risk could be selected.
Kick off (overview)
Serves to provide those involved in the review with all of the necessary information. The purpose is sharing information.
▪ a short introduction to the material may be arranged, and a description of how it fits into the application or environment may be provided
Those involved must have access to other documents. These include documents that help to decide if a particular statement is wrong or correct. For example:
▪ requirements specification, ▪ design,
▪ guidelines,
▪ or standards.
Such documents are also called base documents or baselines
Individual Preparation
The reviewers intensively study the review object and check it against the documents given as a basis for it as well as against their own experience.
They note deficiencies (even any potential defects), questions, or comments
Sample Review Meeting Rules [IEEE1028]
1. The review meeting is limited to two hours. If necessary, another meeting is called, but it should not take place on the same day.
2. The moderator has the right to cancel or stop a meeting if one or more experts (reviewers) don’t appear or if they are not
sufficiently prepared.
3. The document (the review object) is subject to discussion, not the author:
• The reviewers have to watch their expressions and their way of expressing themselves.
• The author should not defend themselves or the document. (The author should not be forced into a defensive position.
However, justification or explanation of the author’s decisions is legitimate and helpful.)
4. The moderator should not also be a reviewer at the same time.
5. General style questions (outside the guidelines) shall not be discussed.
6. Solutions and discussing them isn’t a task of the review team.
7. Every reviewer must have the opportunity to adequately present their issues
8. The protocol must describe the consensus of the reviewers.
9. Issues must not be written as commands to the author (concrete suggestions are sometimes considered useful and sensible for quality improvement).
…
The issues should be weighted as follows ▪ Critical Defect
▪ Major defect
▪ Minor Defect
▪ Good
The review team shall make a recommendation about the acceptance of the review object (see follow-up):
▪ Accept (without changes)
▪ Accept (with changes, no further review)
▪ Do not accept (further review or other checking measures are necessary) All the session participants should sign the protocol
Rework
The manager decides whether to follow the recommendation or do something else.
A different decision is, however, the sole responsibility of the manager. Usually, the author will eliminate the defects on the basis of the review
results and rework the document.
More formal reviews additionally require updating the defect status of every single found defect
Follow-up
If the result of the first review was not acceptable, another review should be scheduled (abbreviated manner focussing on changed areas)
review meetings and their results should then be thoroughly evaluated to improve the review process (adapt checklists to the specific conditions, and keep them up-to-date)
Recurring, or frequently occurring, defect types point to deficiencies in the software development process or lack of technical knowledge of the people involved
▪ Necessary improvements of the development process should be planned and implemented
▪ These defect types should be included in the checklists
▪ Training must compensate for lack of technical knowledge.
Roles
Moderator: Lead the review meetings
▪ responsible for executing the review.
▪ Planning, preparation, execution, rework, and follow-up should be done in such a way that the review objectives are achieved
▪ must be able to stop unnecessary discussions without offending,
▪ mediate when there are conflicting points of view, and be able to see “between the lines.”
▪ must be neutral and must not state their own opinion about the review object
Scribe or secretary: Gather information on findings
Author: Describe, explain, answer questions on item
▪ creator of the document that is the subject of a review.
▪ If several people involved in the creation, should be appointed
▪ responsible for the review object meeting its (i.e., that the document is reviewable) and ▪ performing any rework required for meeting the review exit criteria
one person
review entry criteria
…
Reviewer/inspector: Find defects (bugs) in item
▪ several (usually a maximum of five) technical experts participate in the review meeting after necessary individual
preparation
▪ One reviewer might focus on conformance with a specific standard, another on syntax.
▪ The manager should assign these roles when planning the review
▪ should also label the good parts in the document
Manager:
▪ Plan, arrange resources and training, support, analyse process metrics
▪ The manager , assigns the necessary resources, and selects the review team.
▪ Management level because management might evaluate the qualifications of the author and not the document
▪ the manager often lacks the necessary detailed understanding of technical documents.
▪ Management reviews of project plans and the like are a different thing. In this case, knowledge of management principles is necessary.
selects the objects to be reviewed
should not participate in review meetings
…
In some cases, one person may play multiple roles
▪ Authors sometimes act as moderators
One of the reviewers can act as the recorder or scribe or secretary
▪ able to record in a short and precise way, correctly capturing the essence of the discussion.
▪ not easy when contributions are not well expressed.
▪ pragmatic reasons may make it meaningful to let the author be recorder. The author knows exactly how precisely and how detailed the contributions of the reviewers need to be recorded in order to have enough information for rework
Potential Problems in reviews
Skill base not up to scratch
▪ Train or get consultants in to perform
Poor estimation (see over)
Lack of preparation and/or reviewer ‘believing’ in the benefit of the process Missing or insufficient documentation
Lack of management support
(resources, and results not used for process improvement)
On estimation
The basic prerequisites of the test estimation process
1) Insights gathered from working with past experience:
▪ It is always a good practice to spend some time, recalling past projects which posed challenges similar to the current endeavour at hand.
2) The available documents or artefacts:
▪ The test management repository tools come in handy in these types of scenarios as they store the requirement and
clarification documents. These documents can be referred by the testing team to clearly define the scope of the project.
3) Assumptions about the type of work:
▪ Past working experience helps in making assumptions about the project. This is where hiring experienced professionals matters most.
▪ The testing managers can pick up the brains of these people for delivering the desired results.
4) Calculation of potential risks and threats:
▪ The testing team also needs to visualise the potential risks and threats and pitfalls which lie may lie for the team in future. 5) Determining whether the documents have been baselined:
▪ The testing team also needs to determine if the requirements have been baselined or not. If the documents are not baselined then it is important to determine the frequency of the changes.
…
6) All responsibilities and dependencies should be clear:
▪ The organisation should clearly define the roles and responsibilities for all the persons who would be performing the estimation process.
7) Documentation and tracking of the estimation records:
▪ All the relevant information to the estimation process should be documented.
8) Activities which are required to be performed during the test estimation process
▪ Organise the team that would perform estimations
▪ Decompose the project into project phases and subsequent constituent activities
▪ Compute the estimation based upon previous projects and professional experience
▪ Prioritise the possible threats and come up with the approaches to mitigate those risks ▪ Review and document the relevant part of the work
▪ Submit the work to the relevant stakeholders
Types of Reviews
1.Reviews pertaining to products or intermediate products that have been created during the development process
2.Reviews that analyse the project itself or the development process
These are management (or project) reviews (IEEE1028) and not what we
▪
are studying. They are performed at milestones or a postmortem.
Returning to the Types of Reviews
Informal: no real process (hallway chats, buddy tests, pair programming), yet useful, cheap, popular
Walkthrough: author “walks through” review item
Technical: documented and defined defect removal process, involving technical
experts but not managers
Inspection: a trained moderator (other than the author) leads the inspection team (with defined roles) through a formal inspection process (rules, checklists, entry and exit criteria), which includes gathering defect removal metrics
When walkthroughs, technical reviews or inspections are performed by a peer group, the review may be called a peer review
Walk Through
URS = User Required Specification
Both SRS and URS enable the user to relate to the functionality of software, but there is a clear difference between the two. Users of any particular system expect various features to be present in order for the system to function properly and be user friendly.
▪ URS: For example, for an online shopping portal, after a purchase, the user of the system should receive some sort of confirmation, such as a “Thank You” screen that lets him know that the credit card or alternative payment method has been accepted and the order is complete.
▪ SRS: an SRS is what goes on behind the scenes to ensure that the URS is available and functioning properly — the actual design of the system.
The URS is typically written prior to the SRS, because the SRS is largely based on the user’s experience and expectations.
There is also input from the system owner and quality assurance developer to ensure the user’s expectations for the system are feasible for the system.
The URS is often used for validation of software prior to development of a particular system. The URS is also used for retrospective validation when a system has already been created and validated
SRS System Requirement Specification
In order for an SRS to meet the URS, the SRS must contain key information that will help the system function in the manner that the user expects. This information includes
▪ descriptions of the operations performed by each screen,
▪ data that can be entered into the system (eg letters, numbers & special characters),
▪ work-flows performed by the system and system reports or other outputs
▪ specifies who can enter data into the system & how the system meets regulatory requirements applicable to the specific system.
Benefits
▪ SRS and URS, both are connected.
▪ Without the URS, the SRS will suffer in quality and lack in functionality and usability.
▪ A close relationship between URS and SRS generally reduces the amount of time that is needed to develop a system, because essential elements and requirements would have been tested during the URS phase.
Input to Walkthrough
• Statement of objectives
• The software/other element for test
• Standards
• Materials
• Team members pre-read and prepare • Check list for defects
Walkthrough Meeting
• Author presents element to be tested
• Members ask questions and raise issues about deviations etc
• Concerns, perceived omissions or deviations discussed
• Discussions are documented
• Comments and decisions recorded
• Walkthrough leader (usually author) verifies later that the action items assigned, are closed. Outputs:
• List of findings
• List of resolved and unresolved issues after re-verification
Inspections
The inspection is the most formal review.
It follows a formal, prescribed process.
Every person involved, usually people who work directly with the author, has a defined role.
Rules define the process.
this method of reviewing has been called design inspection or code or software inspection.
collected data is analysed in order to find causes for weaknesses in the development process.
After process improvement, comparing the collected data before the change to the current data checks the improvement effect
Technical Review
In a technical review, the focus is
▪ compliance of the document with the specification,
▪ fitness for its intended purpose, and
▪ compliance to standards.
The reviewers must be technically qualified.
Some of them should not be project participants in order to avoid “project blindness.” Management does not participate.
Most of the effort is in preparation.
The author does not normally attend the meeting
The review result must be approved unanimously by all involved reviewers and signed by everyone
Snapshot
Reviews: Review team requests project team leadership or management to act on recommendations.
▪ Volume of material: Moderate to high
▪ Presenter: Software element representative
Walkthroughs: All decisions made by the producer. Change is prerogative of the author
▪ Volume of material: relatively low
▪ Presenter: Author
Inspections: Team declares exit decision: acceptance, rework and verify, rework and inspect. ▪ Volume of material: relatively low
▪ Presenter: Other than author
In summary
Walkthrough is when the author of a document guides. They do most of the preparation. Can have many people, brings lots of views. Good for high level. Scenarios/Dry Runs may be used to validate.
Technical Reviews are aimed at consensus but less rigorous than Inspections. Defects are by experts who focus on the content: architects, chief designers, & key users. They can be formal or informal. Often a peer review without managers. Ideally, let by trained moderator or technical expert
Inspection is the most formal. Checked thoroughly before the meeting. ▪ Efficient.
▪ Egoless Engineering.
▪ formal rules and checklists, noted.
▪ defects documented in a log
▪ possibly process improvement identified ▪ led by trained moderator
Just code?
A document to be analysed must follow a certain formal structure in order to be checked by a tool.
Static analysis makes sense only with the support of tools.
Formal documents can note, for example, the technical requirements, the software
architecture, or the software design.
▪ An example is the modelling of class diagrams in UML. Generated outputs in HTML or XML
can also be subjected to tool-supported static analysis.
Formal models developed during design phases can also be analysed and inconsistencies can
be detected.
Unfortunately, in practice, the program code is often the one and only formal document in software development that can be subjected to static analysis
Static vs Dynamic
Some defects become apparent only when the program is executed.
▪ For example, if the value of the denominator is stored in a variable, that variable can be
assigned zero. This leads to a failure at runtime.
▪ In static analysis, this defect cannot easily be found, except for when the variable is assigned
the value zero by a constant having zero as its value.
▪ All possible paths through the operations may be analysed, and the operation can be flagged as potentially dangerous.
On the other hand, some inconsistencies and defect-prone areas in a program are difficult to find by dynamic testing.
▪ Detecting violation of programming standards or use of forbidden error-prone program constructs is possible only with static analysis (or reviews).
▪ The compiler is a static testing tool of code
… compiler output
Many compilers also generate further information and perform other checks:
• Generating a cross-reference list of the different program elements (e.g., variables, functions)
• Checking correct data type usage by data and variables in programming languages with strict typing
• Detecting undeclared variables
• Detecting code that is not reachable (so-called dead code)
• Detecting overflow or underflow of field boundaries (with static addressing)
• Checking interface consistency
• Detecting the use of all labels as jump start or jump target
Note about developers
In Test Driven Development, it is not correct to state as Brooks did in 1975 that the Developers will invent their way through the gaps.
In TDD, in an agile setting, there is often a stakeholder present and the close coupling together with a test-driven production mindset could imply that this isn’t a problem
I haven’t seen studies which compare TDD and traditional development in this regard. It would be interesting. It’s an open question in Industry.
Ask your tutors for their view!
Successful Reviews
• Provide training
• Review the product, not the producer
• Set and follow agenda and objectives
• Limit debate
• Focusing on finding, not fixing, problems • Take written notes
• Limit and carefully select participants
• Insist on preparation
• Include testers
• Develop a checklist for each type of item that is reviewed
• Review the reviews
• Use the right techniques
• Ensure management support
• Don’t misuse findings
• Learn and get better!
Some common issues
Ambiguities: What exactly does that mean?
▪ E.g.: System shall allow user to read ISP e-mail
▪ What ISPs? What size e-mails? Attachments?
Incompleteness: Okay, and then what?
▪ E.g.: Upon three invalid passwords, system shall lock user’s account… ▪ For how long? How to unlock? Who can unlock?
Untestability: How can I check this item?
▪ E.g.: System shall provide 100% availability
▪ No known test technique to demonstrate perfect availability
Excessive dependencies, coupling and complexity
▪ Look for ugly design diagrams and confusing requirements
Example: Code anomalies (obvious)
• The domain of Help is limited to the function. The first usage is on the right side of an assignment. The variable still has an undefined value, which is referenced.
• Max is used twice consecutively on the left side of an assignment. Either the first assignment can be omitted or the programmer forgot that the first value (before the second assignment) has been used
• In the last assignment of the function, Help is assigned another value that cannot be used anywhere because the variable is valid only inside the function
65
Control Flow Complexity (from Ch 1)
A
• For a control flow graph (G) of a program or a program part, the cyclomatic number v(G) can be computed
v(G) = e – n + 2p
• e = # of edges, n = # of nodes, p = # connected programs
• e = 17, n = 13, p = 1
• v(G) = 17 -13 + 2 = 6
• McCabe considered 6 midrange, and above 10, means code too complex and rework.
➡
B
66
Drawbacks of Cyclomatic Complexity
• Positive: gives us an indication of testability and maintainability. The cyclomatic number specifies the number of independent paths in the program part. If 100% branch coverage is intended, all these independent paths of the control flow graph have to be executed at least once
• The cyclomatic number has been very much discussed since its publication. One of its drawbacks is that the complexity of the conditions, which lead to the selection of the control flow, is not taken into account.
It does not matter for the calculation of the cyclomatic number whether a condition consists of several partial atomic conditions with logical operators or is a single condition.
Many extensions and adaptations have been published concerning this. It’s an ingredient, but not rigorous.
➡
➡ ➡
Can give a “gut feel”
What about code like If (a && b && d || e && f)
67
Summarising
• Several pairs of eyes see more than a single pair of eyes. This is also true in software development. This is the main principle for the reviews that are performed for checking and for improving quality. Several people inspect the documents and discuss them in a meeting and the results are recorded.
• A fundamental review process consists of the following activities: planning, kick-off, preparation, review meeting, rework, and follow-up. The roles of the participants are manager, moderator, author, reviewer, and recorder.
• There are several types of reviews. .
Unfortunately, the
terminology is defined differently in all literature and
standards
68
…
• The walkthrough is an informal procedure where the author presents her document to the reviewers in the meeting. There is little prepara- tion for the meeting. The walkthrough is especially suitable for small development teams, for discussing alternatives, and for educating people in the team.
• The inspection is the most formal review type. Preparation is done using checklists, there are defined entry and exit criteria, and a trained moderator chairs the meeting. The objective of inspections is checking the quality of the document and improvement of development, the development process, and the inspection process itself
69
…
• In the technical review, the individual reviewers’ results must be given to the review leader prior to the meeting. The meeting is then prioritised by assumed importance of the individual issues. The evaluators usually have access to the specifications and other documentation only. The author can remain anonymous.
• The informal review is not based on a formal procedure. The form in which the results have to be presented is not prescribed. This type of review can be performed with minimal effort, so its acceptance is very high, and commonly used.
• Generally, the specific environment, i.e., the organisation and project for which the review is used, determines the type of review to be used. Reviews are tailored to meet specific needs and requirements, which increases their efficiency. It is important to establish a cooperative & collaborative atmosphere among those involved in the development of the software.
• In addition to the reviews, a lot of checks can be done for documents that have a formalised structure. These checks are called static analyses. The test object is not executed during a static analysis.
70
…
• The compiler is the most common analysis tool and reveals syntax errors in the program code. Usually, compilers provide even more checking and information.
• Analysis tools that are dependent on programming language can also show violation of standards and other conventions.
• Tools are available for detecting anomalies in the data and control flows of the program. Useful information about control and data flows is generated, which often points to parts that could contain defects.
• Metrics are used to measure quality. One such metric is the cyclomatic number, which calculates the number of independent paths in the checked program. It is possible to gain information on the structure and the testing effort.
• Generally, static analyses should be performed first, before a document is subjected to review. Static analyses provide a relatively inexpensive means to detect defects and thus make the reviews less expensive.
71
What concepts you need to know?
• Static Analysis vs Static Testing • Dynamic Testing
• Reviews: Formal and Informal
• Walkthrough
• Technical Review
• Inspection
• Ad hoc review
• Check list based review • Scenario based review • Role based review
• Perspective based review
• Check the definitions in the glossary https://rmit.instructure.com/ courses/67079/pages/istqb- glossary
• Read the text book carefully. Questions arise from there
• My lectures are add-ons and not substitutes for the text book