CS代写 Data and Experiments

Data and Experiments

Collecting data –
for evaluations or experiments

Performance – response time, errors Preference – preference in a comparative task Perception – free-form opinions
Process – actions performed in doing a task Product – artefact created by the participant
Purchase, H.C. Experimental Human-Computer Interaction, 2012, CUP

Data Collection Methods
• Questionnaire
• Interview
• Answers on a task sheet • Observational notes
• Automaticcollection – answers
– time taken
– action log (internal log, videos/audio, screen casts)
• Artefactstorage

questionnaire
task sheet
observation
automatic collection
artefact storage
Performance
Preference
Perception
questionnaire
task sheet
observation
automatic collection
artefact storage

questionnaire
easy to administer
difficult to design
typically quick
difficult to get detailed qualitative answers
easy to get large numbers of participants
not always easy to conduct
difficult to design
takes time
can elicit detailed qualitative answers
difficult to get participants
task sheet
easy to administer
based on task
takes as long as the task itself
quantitative responses to direct task-based questions
availability of participants will depend recruitment to do the task
observation
obvious thing to do
difficult to gather systematic data
takes as long as the task itself
can be qualitative and/or quantitative
availability of participants will depend recruitment to do the task
automatic collection
easy to collect
tempting to collect more than is needed
takes as long as the task itself
quantitative only;
time consuming to analyse
availability of participants will depend recruitment to do the task
artefact storage
(maybe) easy to collect
being unambiguous is crucial
takes as long as the task itself
qualitative product data
availability of participants will depend recruitment to do the task

Terminology
subjective/ objective quantitative/ qualitative preference/ performance open/ closed questions formative/ summative
evaluation/ experiment

“Evaluation” vs “Experiment” • qualitative vs quantitative
• formative vs summative
• small numbers vs large numbers
• visualisation tool vs visualisation method/idea • large task vs small task
• lengthy vs brief

Experiments
• Hypothesis-driven, or driven by a Research Question
• In IV, seeking verification that a proposed tool/ method is ‘good’
• The experiment is defined by
– the independent variable (what is being tested)
– conditions (different values of the independent variable) – control variables (stay the same)
– random variables (that you don’t care about)
– dependent variables (the data you collect)
– tasks (what the participants are asked to do)

Experimental factors are things that can change during an experiment, e.g.
the data used in a visualisation
number of menu options available
the colour palette used
the tasks being performed by the participants the participants themselves
temperature of the room the computer used
Some of these you want to change, some you want to keep the same, some you have no control over, some you don’t care about

Variables (or ‘factors’)
Independent variables
factors you want to change, because they are related to your RQ defined by a set of ‘conditions’
Control variables (not really ‘variables’)
factors you need to make sure do not change
Random variables
factors whose values you allow to change randomly
Confounding factors
factors that changed during the experiment (although you wish they hadn’t)
Dependent variables
the data that you collect

Which data chart best shows the trends in club membership, per gender?

We would like to show that:
a change in the values of the independent variable will result in a change in the values of the dependent variable
And we want to be sure that:
any change in the value of the dependent variable is a result of the change in the value of the independent variable (rather than a result of change in the value of any other factors)

RQ: which data chart is best?
• independent: type of data chart
• conditions:
– A: 100% stacked area chart – B: clustered column chart – C: line chart
• control: the data sets presented
• control: the tasks (values to be read off the charts)
• random: size of computer screen
• random: age and gender of participants
• random: club membership; running ability
• dependent: accuracy in reading the values

Which data chart best shows the trends in club membership, per gender?
number of participants getting the answer correct in three questions (out of 32)

Confounding factors
• Factors that changed during the experiment…
• … but only discovered after completion of the experiment
• These variables may have affected the value of the dependent variable…
• … and may have done so in way that means we cannot be sure that it was change in the value of the independent variable that affected the value of the dependent variable…
• … since it may have been the change in the confounding factor instead

Which data chart best shows the trends in club membership, per gender?
number of participants getting the answer correct in three questions (out of 32)

Control variables
• Kept constant throughout the experiment
• If constant, these variables cannot affect the value of the dependent variable…
• … thus ensuring that we can be sure that it is the value of the independent variable that affects the value of the dependent variable

RQ: which data chart is best?
• independent: type of data chart
• conditions:
– A: 100% stacked area chart – B: clustered column chart – C: line chart
• control: the data sets presented
• control: the tasks (values to be read off the charts)
• control: colours used in each chart
• random: size of computer screen
• random: age and gender of participants
• random: club membership; running ability
• dependent: accuracy in reading the values

Data analysis overview
summary for T1
summary for T2 summary for T3
summary for T4 summary for T5
overall summary

Quantitative data analysis overview
5.33 53.33%
44.57 44.57% 7.67 76.67%
14.29 71.43% 110.14 73.43%

Qualitative data analysis overview
summary for T1
summary for T2 summary for T3
summary for T4 summary for T5
overall summary

Video on Qualitative Data Analysis

Data extract:
• Too cluttered
• I’m loving the bright colours
• Things way too close together
• Not enough space
• Too garish
• I like the “busy-ness”
• Very squished up
• Wonderfully active and lively
• Bright and lively
• Colourful (…more data items)
[004] [006] [001]
[005] [006] [002] [005] [006] [002]
Codes (or “themes”):
001: cluttered
002: bright colours (+ve) 003: bright colours (-ve) 004: busy (+ve)
005: lively (+ve)
006 = 004+005: lively (+ve)
Code count:
001 [44] 002 [32] 003 [5] 006 [12]
Most participants (44/70) complained that the screen was too cluttered, while 12 of the other participants liked the “busy-ness” and activity that the display depicted. The brightness and colours were appreciated (32/70), although a minority (5/70) said that the colours were too bright and “made my eyes hurt”.

Number of participants
Formative evaluations:
(2000), “Why You Only Need to Test with 5 Users”
“Elaborate usability tests are a waste of resources. The best results come from testing no more than 5 users and running as many small tests as you can afford.”
Summative evaluations:
Probably more than this, say 10, to be sure of
the validity and generalisability of the ‘proof of worth’ outcome
Experiments:
For statistical analysis, even more; more than 30 is an oft-quoted minimum
(https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/)

Key principles of Evaluations & Experiments
• Participant time is your most precious resource
• Pilot: try out your evaluation/experimental methodology in
advance to make sure it will produce the data needed – phrasing of tasks
– adequacy of training
– expected timing of session
• Don’t start until you are sure that the data you will collect will be useful to you; create ‘dummy’ data to see if it makes any sense
• Participants should never be confused as to what to do
• Get as much as you can out of the participant in each session

Chapters 1-4 (p1-115)

Data and Experiments

程序代写 CS代考加微信: powcoder QQ: 1823890830 Email: powcoder@163.com

Related Posts