Agile Software Development
Scrum
Christopher Jones Fall, 2020-2021
School of Computing, DePaul University
Principles of Process Re-engineering
Hirotaka Takeuchi and Ikujiro Nonaka [1] proposed that product development should be viewed as the result of a multidisciplinary team working together constantly rather than as a linear progression of a product from one set of specialists to another.
1/61
Companies that were successful at re-engineering their processes to make them more holistic and agile, tended to demonstrate six common traits:
1. Built-in instability.
2. Self-organizing project teams.
3. Overlapping development phases. 4. Multilearning.
5. Subtle control.
6. Organizational transfer of learning.
2/61
Built-in instability is often characterized by setting a strategic direction and establishing aggressive goals. The product teams are often given significant latitude and leeway. The instability is generated because prior knowledge will often not apply. The whole point is to do something radically different.
3/61
Because prior knowledge is often not available, the product team will go through it’s own process of self-organization, almost like a start-up company. A team is self-organized when it exhibits three characteristics:
1. Autonomy.
2. Self-transcendence. 3. Cross-fertilization.
4/61
The product team must have autonomy. Headquarters involvement is limited to guidance, funding and moral support. The intent is that top management act more as venture capitalists than as employers.
5/61
Self-transcendence is achieved when the team sets their own goals based on those established by their charter. The team continuously elevates those goals in an effort to achieve bigger and better things.
6/61
Cross-fertilization is the idea that a diverse, co-located team can share ideas and information just through their day-to-day interchanges.
7/61
The discrete phases of a classic, linear progression serve to manage risk, but doesn’t do much to integrate different points of view. When the phases overlap, it allows the team to absorb the variability generated by the continuous interaction of the team’s individuals. The overlap approach encourages:
• Shared responsibility and cooperation.
• Involvement and commitment.
• Development of diversified skills.
• Heightened sensitivity to market. conditions.
Overcoming gaps in perception is challenging. Every member of the team has to buy into the overall vision and approach.
8/61
The team constantly performs small experiments to find out what works. They also continually acquire new and diverse skills because of cross-fertilization. Multilearning is learning that is:
1. Multilevel.
2. Multifunctional.
9/61
Multilevel learning is about the degree to which learning is provided across an organization.
Individual Continuous learning is encouraged. This doesn’t mean just books, trade magazines or classes, but could include market analysis or other related activities.
Group A group of individuals working toward a shared goal can and often will learn together.
Organization Organizational learning is best done to encourage significant changes in internal philosophy. This kind of
learning is encouraged across the entire organization.
10/61
In multifunctional learning, team members are encouraged to gain knowledg
11/61
Management establishes checkpoints to address instability, ambiguity and tension but does not impose the kind of control that limits creativity. Such subtle control is exercised by:
1. Selecting the right people and keeping the team balanced.
2. Creating an open work environment.
3. Encourage team members to go out and talk to customers.
4. Establish an evaluation and reward system based on group performance.
5. Manage project rhythm during the development process.
6. Anticipating and tolerating mistakes.
7. Encourage suppliers to become self-organizing.
12/61
Team members are encouraged to transfer their knowledge to others outside of the group. This can happen by:
• Move team members to other strategic projects.
• Incorporate lessons learned as organizational standards or best practices.
This needs to be balanced with appropriate unlearning; what worked before was shaped by the prevailing conditions at the time. If the conditions change, then the lessons may no longer be applicable.
13/61
Takeuchi and Nonaka identified some apparent limits:
• Extraordinary effort is required by all team members. This makes sense given that many of these projects were concerned with creating ground-breaking new products for their companies, but it isn’t sustainable for a long time.
• Works better for evolutionary products rather than revolutionary ones. You can never be sure that the research needed for a revolutionary idea will bear fruit on time.
• May not work well when the project scale gets too large and prohibits face-time with all team members.
• Won’t work for organizations where development is spearheaded by a “genius” who hands down complete specifications.
14/61
Management must promote the process. Because the process is fluid, management must be willing to forgo too many top-down decisions, focusing instead on the subtle control we saw earlier.
15/61
The Scrum Framework
Scrum [2] was proposed by Jeff Sutherland in 1993, with help from Ken Schwaber. It was based on the paper by Hirotaka Takeuchi and Ikujiro Nonaka [1] and looked at software development as just another kind of product development.
16/61
1. A product owner creates a prioritized list of user stories called the product backlog.
2. Stories from the product backlog are added to the sprint backlog during sprint planning.
3. The team has a period of time, a sprint, to complete the sprint backlog. The team holds a daily scrum to assess progress.
4. The ScrumMaster keeps the team focused on its goal.
5. The sprint ends with a sprint review and retrospective.
17/61
• Product Owner. Responsible for the business value of the project.
• ScrumMaster. Ensures that the team is functional and productive.
• Project Team. Self-organizes to complete the work.
18/61
19/61
The product backlog is the prioritized list of “requirements” for a product.
• Ordered by the Product Owner based on considerations like risk, business value, dependencies, date needed, etc.
• Items are commonly written as user stories.
• The backlog is editable by anyone, but the Product Owner is responsible for ordering the stories for the Development Team.
• Items contain rough estimates of both business value and development effort.
The Product Backlog, and business value of each item, is the responsibility of the Product Owner. The estimated effort to complete each backlog item is determined by the Development Team.
20/61
The sprint backlog represents the work that the development team has committed to complete for the given sprint.
• Stories are broken down into smaller tasks.
• Stories are not assigned; developers volunteer to complete the
tasks during the daily scrum.
The sprint backlog may only be changed by the development team.
21/61
The basic unit of development time in Scrum is a sprint. Sprints are:
• Timeboxed to a fixed length, typically 2-4 weeks.
• Preceded by a planning meeting to develop the sprint backlog. • Managed only by the development team.
• Concluded with a review and retrospective.
Any items in the sprint backlog not completed are returned to the product backlog where they can be incorporated into later sprints.
22/61
There are several ceremonies commonly performed during Scrum:
• Sprint Planning.
• Daily Scrum.
• Scrum of Scrums.
• Backlog Grooming.
• Sprint Review.
• Sprint Retrospective.
23/61
Sprint planning happens before each sprint. Activities include:
• Define the sprint goal.
• Select items from the product backlog to comprise the sprint backlog.
• Detail the time it will take to complete the items on the sprint backlog.
• Communicate the amount of work that is likely to be done during the sprint.
The meeting is held to at most 8 hours:
• 1st 4 hours: The entire team discusses the prioritization of the product backlog.
• 2nd 4 hours: The development team commits to the sprint backlog.
24/61
Every day during a sprint, the project team meets for the daily scrum or standup. Each member answers 3 questions:
• What have you done since yesterday? • What are you planning to do today?
• Do you have any obstacles?
There are rules for this meeting to be successful:
• Every member comes prepared with their updates.
• The meeting starts on time – even if people are late. • The meeting length is timeboxed to 15 minutes.
• No detailed discussions happen during this meeting.
25/61
If a project involves multiple teams, then a scrum of scrums can be held, typically after the daily Scrum. The intent is to focus on areas of overlap and integration. This meeting has the same basic format as the daily Scrum, but asks slightly different questions:
• What has your team done since we last met? • What will your team do before we next meet? • Is anything slowing your team down?
• Are you about to slow another team down?
26/61
During each sprint, the team should spend some time grooming the product backlog.
• Estimate the existing stories.
• Refine the acceptance criteria.
• Split larger stories into smaller ones.
This meeting should run no longer than 1 hour and should not focus on breaking stories into tasks.
27/61
At the end of each sprint, a sprint review is held in order to: • Review work that was planned but not completed.
• Demonstrate to the stakeholders the work that was completed. This meeting should run no longer than 4 hours.
28/61
At the end of each sprint, a sprint retrospective is facilitated by the Scrum Master. The goal of this meeting is to reflect on the past sprint with an eye toward making incremental process improvements:
• What went well?
• What needs improvement?
• What actions will the team take during the next sprint to
improve things?
This meeting should run no longer than 3 hours.
29/61
Estimation
Estimations are critical to projects. Unfortunately, people are typically bad at estimating, so many schemes have emerged to calculate them:
• Gut feel
• Story points
• Function-point analysis
In general, the more objective the calculation, the more cumbersome it is.
30/61
We’d like estimates that are [3]:
• Useful for planning
• Fast
• Handles imprecision
• Easy to change
• Scalable (works for epics and stories) • Supports progress tracking
The more precise the derivation, the harder it is to derive it.
31/61
We’ll focus on story points. Each team can define a story point however they want. For example:
• Hours
• Comparative complexity • Ideal work day
Story points need to have relative weight. For example, an estimate of 4 story points should be twice as difficult as one with 2 story points and half as difficult as one with 8.
32/61
Story estimates are owned by the team, not an individual.
• Stories aren’t assigned at the time they’re estimated. • Team feedback leads to better estimates.
Need to have enough of the development team present to make the estimates meaningful.
After some estimates have been made, patterns begin to emerge, and it becomes possible to estimate one story based on its relationship to other stories.
33/61
Since estimation is decoupled from sprint planning, we must make sure that the entire team is comfortable with the estimate.
More participants during estimation bring more knowledge to bear on the problem, hopefully yielding more accurate estimates.
This avoids people tweaking estimates based on their own experience. It ensures that the people doing the work are the ones who own the estimate.
34/61
Estimation meetings tend to follow a typical pattern:
1. Gather business and development together.
2. Business selects an unestimated story at random. 3. Developers discuss, ask questions, etc.
4. Developers each select an estimate.
5. Developers show their estimates at the same time. 6. Discuss highest and lowest estimates.
7. Repeat steps 3-6 until consensus is reached. Timebox these estimating meetings.
35/61
Roadmaps, Releases, and Planning
A product roadmap outlines the areas of focus for the next several sprints. Some teams organize their roadmap by:
• The current quarter. • The next quarter.
• 6-12 months out.
Going out further than 12 months is rarely useful; the business landscape simply changes too fast in most cases.
36/61
A release happens when software is deliberately made available to its users.
• Alpha, beta, RC, and GA.
• The scheme depends on the product and users.
In agile methodologies a release will be composed of one or more iterations. Recall that our goal is working, deliverable software at the end of each iteration.
The idea of being constantly able to release is a foundational principle of continuous delivery.
37/61
We’ve ultimately got to be able to schedule releases and allocate user stories to those releases.
Question: How?
Answer: Maximize the value of the software produced by the
team.
There are two major participants in release planning:
Business The people who will collectively make decisions about what functionality must be supported.
Development The people who will collectively be responsible for implementing the functionality.
38/61
Knapsack Problem
Given a set of items, each with a weight and a value, determine which items can be added to a knapsack so that the total weight is less than or equal to the knapsack’s capacity and the total value is as large as possible.
The stories are the items, each with a value, and the story points represent their weight. A release represents the knapsack, with the velocity as its capacity.
The goal is to invest as little as possible to put the most valuable functionality into production as soon as possible while observing programming and design strategies to manage risk.
39/61
The number of story points actually delivered during a sprint is called the sprint’s velocity.
Initial velocity can be derived by:
• Historical data • Trial iteration • Best guess
This is the last piece of the puzzle we need in order to begin planning releases.
40/61
Business and development jointly agree on the work that will be completed within the sprint. They may consider things like:
• Risk that the story cannot be completed as described. • Impact of one story on others.
• Desirability of the story to customers.
• Cohesiveness of a story in relation to other stories.
Business prioritizes based on the developer estimates of the stories.
The business team may opt to introduce more, smaller-value stories rather than fewer, large-value stories. Remember we want to fill the knapsack.
41/61
The budget for one sprint should account for the actual delivered work from prior sprints. Don’t second-guess this number without a compelling reason (e.g. lots of sick time, holidays, training, etc.).
If the sprint duration changes, then the velocity should be adjusted accordingly.
Remember that not all work hours are spent on project work. Use an ideal work day that accurately reflects your environment.
Ideally we could work on a sample iteration to get a better gauge of our velocity. Not always possible.
Don’t forget to allocate time to deal with technical debt.
42/61
Given the stories in the roadmap, their estimates, and the team’s sprint budget, we can derive the time needed to implement all of the stories.
Business and development allocate stories to sprints during sprint planning. Sprints are accumulated until enough value has been delivered to justify a release. This comprises the product’s release plan.
Planning must consider special cases:
• Technical infrastructure • Cross-team coordination • External review
• Architectural spikes
43/61
Recall that sprint planning happens before each sprint. Stories have been assigned to sprints as part of release planning. We now need to go into more detail and assign actual tasks to developers.
1. The team discusses the story.
2. Developers split the story into tasks. 3. Developers accept tasks.
4. Developers estimate the tasks.
44/61
The goal is provide a TODO list of everything that needs to be done in order to complete the story.
May include technical (e.g. coding) and non-technical tasks (e.g. documentation).
Allow different developers to pick up different tasks within the same story.
One goal and benefit of this process is to share knowledg of a how a particular feature works across as many people as possible. This helps to eliminate knowledge silos.
45/61
Developers volunteer to perform each task.
Task acceptance is fluid. If one individual is idle, they can take on tasks currently assigned to another team member. Responsibility cannot be assigned, it must be accepted.
This process helps get developer buy-in.
The task can be re-allocated later and the developer can ask for help.
46/61
It should be possible to estimate tasks with reasonable accuracy.
Each developer should estimate their tasks to make sure that they aren’t over-committed for the iteration.
If a developer is over-committed, then they need to ask someone else on the team to take some of their load. It may also be possible to split the story or defer it to the next iteration.
47/61
Tracking Progress
Agile methods emphasize trust between the various stakeholders:
• Business → development. • Development → business. • Management → everyone.
It is important that we communicate our status simply and clearly.
48/61
We’ve said that velocity is the number of story points actually completed in a sprint.
Question: What about partially-completed stories?
Answer: We don’t include partially complete stories. We’re never sure how much of it is done. We don’t want to introduce false precision. Incomplete stories are generally not valuable to customers. If including partial stories routinely improves velocity, the stories are too big.
Remember, a story is intended to be completed within a sprint.
49/61
Story Points…
At start of iteration
Estimated done during iteration
At end of iteration
S1 S2 S3 S4 140 100 60 20 40 40 40 20 100 60 20 0
Suppose we have determined that we have 140 points worth of stories to complete for a release.
We estimate that we can complete 40 points worth of stories each sprint, so we’ll need 4 sprints to complete the work.
50/61
Ideal Planned
150
100
50
0
We can chart both the ideal, straight-line completion of points and the estimated completion schedule based on our estimated velocity.
This is called a burn-down chart. It shows how the required work “burns down” over the course of the sprint.
01234 Sprint
51/61
Story Points
Story Points…
At start of iteration
Estimated done during iteration Actual done during iteration From changed estimates
From new stories
At end of iteration
S1 S2 S3 S4 140 100 80 20 40 40 40 20
Very little goes according to plan in software, so it would be good to also account for things like the actual points completed during the sprint, points added because of new stories, and point changes because of incorrect estimations.
We can now track changes over the course of the sprint.
100 60 200
52/61
Story Points…
S1 S2 S3 S4 140 123
40 40 40 20
At start of iteration
Estimated done during iteration
Actual done during iteration 45 From changed estimates 10 From new stories 18 At end of iteration 123
Suppose we get the displayed results from the first sprint. What observations can make about the development team? What observations can make about the business team?
53/61
Ideal Planned Actual Scope
150
100
50
0
Notice that the scope of the release has actually changed because of the new stories. It’s not easy to tell that from the prior graphs, so we can add a scope line to make it plain.
01234 Sprint
54/61
Story Points
Story Points…
At start of iteration
Estimated done during iteration Actual done during iteration From changed estimates
From new stories
At end of iteration
S1 S2 140 123 40 40 45 47 10 4 18 8 123 88
S3 S4 88
40 20
Suppose we get the displayed results from the second sprint. What observations can make about the development team? What observations can make about the business team?
55/61
Ideal Planned Actual Scope
150
100
50
0
If you’re the Scrum Master and you saw this burndown, what should you do?
01234 Sprint
56/61
Story Points
Story Points…
At start of iteration
Estimated done during iteration Actual done during iteration From changed estimates
From new stories
At end of iteration
S1 S2 140 123 40 40
S3 S4 88 41 40 20
45 47 48 10 4 -3 18 8 4
123 88 41
Suppose we get the displayed results from the third sprint. What observations can make about the development team? What observations can make about the business team?
Is it possible to complete the release on time?
57/61
Ideal Planned Actual Scope
150
100
50
0
01234 Sprint
58/61
Story Points
Story Points…
At start of iteration
Estimated done during iteration Actual done during iteration From changed estimates
From new stories
At end of iteration
S1 S2 140 123 40 40 45 47
S3 S4 88 41 40 20 48 41
10 4 -3 18 8 4
Suppose we get the displayed results from the fourth What observations can make about the development What observations can make about the business team?
123 88
41 0
sprint. team?
59/61
Ideal Planned Actual Scope
150
100
50
0
01234 Sprint
60/61
Story Points
References i
[1] Hirotaka Takeuchi and Ikujiro Nonaka.
The new new product development game.
The Harvard Business Review, 1986.
[2] Scrum alliance.
http://www.scrumalliance.org, August 2017.
[3] Andreas Schmietendorf, Martin Kunz, and Reiner Dumke.
Effort estimation for agile software development projects.
In Proceedings 5th Software Measurement European Forum, 2008.
61/61