User Research Methods From observation to feedback
1
Interface Hall of Fame or Shame?
Submit these to the Canvas Discussion Board “Interface Hall of Fame or Shame”
and I’ll pick a couple to be discussed in the lecture
RMIT University©2020 CS&IT
Interface Hall of Fame or Shame?
RMIT University©2020 CS&IT
Interface Hall of Fame or Shame?
Contributed by Alexander L. RMIT University©2020 CS&IT
Mid Semester Test A short briefing
5
Important details
• During Week 7 lecture – 8th Sep 2020; 11.35am-12.30pm • Covers weeks 1 to 6 material.
• 15% of course marks.
• Short questions only.
• Week 7 lecture content will be delivered from 10.30-11.30. • Quiz will still run at around 11.30.
RMIT University©2020 CS&IT
6
Usability cycle
7
The Third Part of the Usability Cycle
• Gathering information(i.e. user types, market needs) >
• Design(i.e. applying design principles, using personas and
use cases) >
• Gathering feedback(i.e. removing your ego and make the best application you can for your users)
RMIT University©2020 CS&IT
Iterative Design Process
• Ideo’s human centred design process: http://www.designkit.org/resources/1
RMIT University©2020 CS&IT
Types of feedback
10
Feedback types?
• • • •
•
•
•
(Paper) Prototypes Surveys Interviews Usability tests
Other (see Usability Engineering) Focus groups
Log analysis
RMIT University©2020 CS&IT
11
Setting up for feedback
12
Receiving criticism is not fun
• Nobody enjoys getting feedback.
• However it is essential. In most of the your tutorials you
were constantly asked to review your peers’ work.
• It is humbling to know that you will not be able to hit a home run right off the bat.
RMIT University©2020 CS&IT
13
Embrace feedback
• It is your best tool.
• As programmers we are often enamoured with programming challenges. This is fine, we are not telling you to be enamoured with users.
• However you need to ensure that your application is on track in the design and business sense by involving your users.
RMIT University©2020 CS&IT
14
Knowing when to listen to your user
• Users’ feedback is not the rule.
• The key is to know when feedback is useful or relevant.
• If there is one user out of 100 who is adamant that a scrolling marquee is in vogue right now, you would probably not take that into consideration.
• Unless you REALLY like a scrolling marquee… • Which in that case you should really not…
RMIT University©2020 CS&IT
15
Types of user
• The Control Freak
–These users will try to be very
pedantic about controls.
–They would also try to run the tests or surveys.
–Or create their own questions.
• Information Overloader.
–These will be very eager to tell you
absolutely everything.
–Even if they are not relevant.
–So you will need to sift through the noise.
• The Devil’s Advocate
–These will try to find problems wherever they could. –You will need to see if they are actual problems.
RMIT University©2020 CS&IT
16
Negativity is not productive
• Users can be negative.
• Well, developers can be negative too!
• Sometimes the best thing to do is to look at what works and go from there.
RMIT University©2020 CS&IT
17
Feedback
Leverage on:
• Social Media
• Friends,
• Peers,
• Competitors.
• In fact, talk to ALL of them.
RMIT University©2020
CS&IT
18
User Research Methods
19
User experience research methods
RMIT University©2020
CS&IT
20
What UX methods to use and when?
•
Who are the Users and Customers? Survey
Persona
Contextual Inquiry Stakeholder Interviews Competitive Analysis
Quality Function Deployment
What are users trying to do?
Task analysis
•
•
• • • • •
•
•
Top tasks analysis http://www.measuringu.com/blog/method-when.php
RMIT University©2020 CS&IT
21
3-dimensional framework
• Attitudinal vs. Behavioral
• Qualitative (formative) vs. Quantitative (summative)
• Context of Use https://www.nngroup.com/articles/wh
ich-ux-research-methods/
22
RMIT University©2020
CS&IT
Formative vs summative
RMIT University©2020 CS&IT 23
Formative vs summative
• Do Formative Usability Testing at the start of the design phase, testing with paper-prototypes and similar. Do this to discover insights and shape the design direction.
• Do Summative Usability Testing during latter half of the development phase, testing with actual working prototypes. Do this to determine metrics (time on task, success rates), which you would then compare test results from improved versions against.
• Usability test protocol usually employed
RMIT University©2020 CS&IT 24
Formative vs summative (evaluative)
• Formative usability testing
–beginning stages of the design process
–provides valuable insights of where users have difficulty reaching their user goals with the technology (website, desktop GUI design, hardware product) or service.
• Summative usability testing
– a Quality Assurance (QA) type of test
–usually performed later in the development process.
–formal user acceptance testing before the product is released to the target audience
RMIT University©2020 CS&IT 25
Formative vs summative
Formative
— > Build — > Summative (evaluative)
Ethnography
Interviews
Surveys
Cultural Probes
Focus Groups
Diary Studies
Experience Sampling Studies
Studying Similar Products
Interaction Logging of Past Product / Early Prototype Studying Past Product Documentation
Ethnography
Interviews
Surveys
Focus Groups
Diary Studies
Experience Sampling Studies Interaction Logging
RMIT University©2020 CS&IT
26
User Research Methods
RMIT University©2020 CS&IT
27
Observations
28
Why can’t we just ask users what they want?
• “You can observe a lot by just watching” (YogiBerra, MLB Player / Quote Machine )
“If I had asked my customers what they wanted, they would have said a faster horse.” (Henry Ford, car inventor)
• So, often it’s better to watch what people do than to only design solely for what they say
• Especially when what people say != what they do RMIT University©2020 CS&IT
Ethnography
• In anthropology, ethnography developed as a way to explore the everyday realities of people living in small- scale, non-Western societies and to make understandings of those realities available to others.
• Today, ethnography is applied to the study of people and social groups in specific settings within large industrialized societies, such as workplaces, senior centers, and schools…
(Jeanette Blomberg, Anthropologist, IBM Research, Mark
Burrell, Psychologist, Microsoft Corporation) RMIT University©2020 CS&IT
3 Principles of Ethnography
• Holistic: tiny details into big picture context; attempt to look at things broadly—understand context
• Natural settings: Directly observe in the things that you’re trying to study
• Descriptive: focus on recording behaviour—analysis comes later
RMIT University©2020 CS&IT
Video ethnography
• Trying to discover the obvious, and the non-obvious • Observing product use in context
• Gaining insight into user behaviours
• Video recording provides data for post analysis
• Example:
In 1983, PARC computer scientist Austin Henderson and Lucy Suchman, Ph.D. student in Anthropology at UC Berkeley, placed a brand-new Xerox high-speed copier in a room, and observed people using the machine.
https://www.youtube.com/watch?v=DUwXN01ARYg RMIT University©2020 CS&IT
Video ethnography
• Example:
In 1983, PARC computer scientist Austin Henderson and Lucy Suchman, Ph.D. student in Anthropology at UC Berkeley, placed a brand-new Xerox high-speed copier in a room, and observed people using the machine.
Lucy Suchman,
Ethnographer Xerox PARC
RMIT University©2020 CS&IT
Video ethnography
• Is the problem with the people or with the copier machine? • Who are those people in the copier room?
Allen Newell
Computer Scientist Xerox PARC
Winner of Turing Award
Ron Kaplan
Computer Scientist
Xerox PARC
Natural language technology expert
• To learn more, watch: https://www.youtube.com/watch?v=nV0jY5VgymI RMIT University©2020 CS&IT
Look at what users really do
RMIT University©2020 CS&IT
Look at what users really do
http://www.knowledgeproductivity.com/inspiring-ideo-method-cards RMIT University©2020 CS&IT
Ask users to help
http://www.knowledgeproductivity.com/inspiring-ideo-method-cards RMIT University©2020 CS&IT
A day in the life
• A type of ethnographic research
A user is followed and observed by the researcher throughout a typical day
Observing moments that are so routine to the user as to be unremarkable, or even entirely unnoticed by the user.
This removes the bias towards more interesting (but possibly less relevant or impactful) events.
To demonstrate the various product touchpoints with a user
within the context of their life. RMIT University©2020 CS&IT
•
•
•
•
A day in the life
• Observe this example video. https://www.youtube.com/watch?v=CaNv762hMqE
• A couple of other videos in different occupations
• https://www.youtube.com/watch?v=m0ZTJ7OofIo
• https://www.youtube.com/watch?v=5BSaX_aAcT4
• https://www.youtube.com/watch?v=bs_7jWqSeIM
RMIT University©2020 CS&IT
A day in the life
• From observing those videos:
• What can you learn about the user?
• What sort of problems or issues you observe?
Can you potentially discover user needs just by watching these videos?
• What are the potential solutions you might be designing? RMIT University©2020 CS&IT
•
Diary Studies
41
Diary Study
• Users are requested to record feedback and/or events on a specific part of a product experience.
• Can be on any medium, paper, voice, videos or audio recordings.
• Usually lasts a week to two weeks.
• Daily support from the UX personnel running the study.
RMIT University©2020 CS&IT
Diary Study
• Diary is then analysed by a consultant.
• Debriefing at the end of the study with the user.
• Usually lasts a week to two weeks.
• Daily support from the UX personnel running the study.
RMIT University©2020 CS&IT
Diary Study
• Advantages
• Rich data, covers responses over time.
• Context rich data, user experience may be related to
external and internal aspects.
• Disadvantages
• Resource intensive – time/$$
• Self reporting bias.
• Highly reliant on user participation.
RMIT University©2020 CS&IT
Focus group
45
Focus group
Help to assess user needs and feelings both before interface design and long after implementation.
Bring together 6–9 users to discuss issues and concerns about the features of a user interface.
The group typically lasts about 2 hours and is run by a moderator who maintains the group’s focus.
• Bring out users’ spontaneous reactions and ideas
You can also ask people to discuss how they perform activities that span many days or weeks: something that is expensive to
•
•
•
•
observe directly.
RMIT University©2020 CS&IT
https://www.nngroup.com/articles/focus-groups/
Focus group
• Example questions
• What is your impression of the product? (Use numeric or likert scale)
• What specific features of the product impressed you?
• What features of the product disappointed you?
• How likely are you to purchase this product? (Use a numeric or degree of likelihood scale)
• How likely would you be to recommend this product to a family member or friend? (Use a numeric or degree of likelihood scale)
RMIT University©2020 CS&IT
Simulated focus groups
• Discuss in a group of 3-4 in the lecture
• Amazon Keys
• https://www.youtube.com/watch?v=wn7DBdaUNLA
• https://www.youtube.com/watch?v=9kbFfGrGm_Y
• https://www.youtube.com/watch?v=8bZfZZJ7Q4Q
RMIT University©2020 CS&IT
Surveys
Challenges Recap
49
Surveys
• You fill so many out.
• Tempting to think you know how to write them
• Creating a useful and reliable survey is an art form.
• You not only have to know which questions to ask, but how to frame the questions using concise and impartial words.
50
Survey
• Survey
• Coverage error
• Do you get to everyone using your system • Sampling error
• If you only talk to a sample, some error • Nonresponse error
• Survey the correct people, but what if not all answer?
• Measurement error
• People don’t answer accurately
51
Measurement error
• Ask factual questions people often answer themselves? • Generally no problem
• Ask questions people need to think about?
–The way you ask the question could influence the
answer
• Yes, Prime Minister!
• A clip from a British satire comedy
52
Surveys Recommendations
53
Types of Questions
Open ended questions where you ask a question and leave space for the user to write freeform answers. This is used if you would like to know more about specifics that you were not confident of and to explore other aspects of the problem.
Closed ended questions where you provide the answers and ask the users to select one or more. Use this if you know for sure what are the parameters and variables you are looking for, and also if you want to use a scale like the SUS or other metrics.
54
Start with a screener
Screeners are important questions to help you target the correct audience for your surveys. Start by asking if they used the site/app on that specific platform. Remember that this assignment requires that you survey current users and not potential users.
55
Do not lead the user.
It is not a good idea to project negative or positive sentiments in your questions.
For instance, instead of:
Trying to navigate between courses in blackboard is confusing, what do you think?
you can say:
How often do you switch between courses on blackboard? Did you have any problems switching?
56
Do not waste time.
Every question should provide answers that would play a part in your analysis. Ask yourself this for every question. “what can I do with the data?”
57
You are not studying the features but how easy it is to understand and access those features.
Although you are indeed trying to find out if the users are happy with the app, you will need to be more specific about what you ask in order to produce a useful and actionable report.
For instance, asking if friends would recommend an app to their friends will not help you identify the drawbacks of the interface. You are more interested in the “why”s.
58
Your questions should be succinct, direct and does not contain jargons.
Especially not the guidelines you learn in this course. Nobody Not many know what is affordance or Nielsen’s thingamajig.
Keep to the point, and use natural common language so that your users does not have to do guess work.
Also maintains their confidence.
59
If you want to ask for temporal data (how often do you…), define the timeline (…within the last month).
For instance, instead of:
How often do you visit ABC.xyz? < Not so good
you can say:
In the past 6 months, how often have you visited ABC.xyz? < Better
60
Don’t make the user feel tested.
For instance, instead of:
I was very confused. (Agree/disagree)
you can say:
The app made it very confusing to navigate around. (Agree/disagree)
61
To recap
62
3-dimensional framework
• Attitudinal vs. Behavioral
• Qualitative (formative) vs. Quantitative (summative)
• Context of Use https://www.nngroup.com/articles/wh
ich-ux-research-methods/
RMIT University©2020
CS&IT
63
Formative vs summative (evaluative)
• Formative usability testing
–beginning stages of the design process
–provides valuable insights of where users have difficulty reaching their user goals with the technology (website, desktop GUI design, hardware product) or service.
• Summative usability testing
– a Quality Assurance (QA) type of test
–usually performed later in the development process.
–formal user acceptance testing before the product is released to the target audience
RMIT University©2020 CS&IT 64
Formative vs summative
Formative
--- > Build — > Summative (evaluative)
Ethnography
Interviews
Surveys
Cultural Probes
Focus Groups
Diary Studies
Experience Sampling Studies
Studying Similar Products
Interaction Logging of Past Product / Early Prototype Studying Past Product Documentation
Ethnography
Interviews
Surveys
Focus Groups
Diary Studies
Experience Sampling Studies Interaction Logging
RMIT University©2020 CS&IT
65
Guest Talk
66
Guest talk by Sam Nolan next week Tue 18 Aug 12pm
Talk summary. Sam will give a talk about his recent work on exploring the impact of the lockdown on social roles and emotion, and also his pet project — Dot collector.
Sam Nolan’s brief bio
President of the RMIT Programming Club
Treasurer of the RMIT CSIT Society
Computer Science Honours Student, working under Flora with her PhD students. Currently working with Maria Spichkova on his honours thesis
Nolan SW, Rumi S, Anderson C, David K, Salim F. Exploring the Impact of COVID-19 Lockdown on Social Roles and Emotions while Working from Home. New Future of Work Symposium. 2020. https://arxiv.org/abs/2007.12353; https://www.microsoft.com/en-us/research/publication/exploring-the-impact-of-covid-19-lockdown-on-social-roles-and- emotions-while-working-from-home/
Dot collector is an application inspired by an tool from Ray Dalio’s “Principles” book by the same name. The application is based off the concept of micro feedback, where large quantities of very small pieces of feedback are collected. This application will give you the opportunity to give feedback with a single click many times during the lecture to evaluate Sam’s performance as a guest lecturer. Your feedback will be anonymous. This software is very much in alpha form, but feedback in regards to the application itself is welcome.
RMIT University©2020 CS&IT 67
Now: Example of a formative user study
A guest talk by Nan Gao on her research work “n-gage”.
Nan Gao is currently a PhD Candidate in Computer Science at RMIT University, Melbourne. She finished her Masters and Bachelors in Software Engineering from Beijing Institute of Technology in 2018 and 2016. Her main research interests are human behaviour modelling and applied machine learning for smart buildings and wearable sensing. For more details please visit her homepage: nancygao.com
Gao, N., Shao, W., Rahaman, M.S. and Salim, F.D., 2020. n-Gage: Predicting in-class Emotional, Behavioural and Cognitive Engagement in the Wild. Proc. ACM Interactive, Mobile, Wearable, Ubiquitous Technologies (IMWUT), 4, 3, Article 79 (September 2020). arXiv preprint arXiv:2007.04831.
RMIT University©2020 CS&IT
68