Lecture Notes for Human Factors
Chapters 29-34--Evaluation

Richard Halstead-Nussloch, Ph.D.

Evaluation--What is it?

• Gathering, reducing, and analyzing data about the usability of a design or product
  • By a specific group of users
  • For a particular set of tasks or activities
  • Within a specified environment
  • According to needs of a (work) context
• Additionally, it is important to consider features of the item (artifact) evaluated

Questions to Answer in Setting- up Any Usability Evaluation

• What do you want to test? (Item)
• What do you want to know? (Objectives)
• Why do you want to know it? (Purpose, Value)
• Who do you want to evaluate it? (Evaluators)
• When in the development cycle? (Timing)
• Where do you want to test? (Field, Lab, …)
• Which activities/measures are important? (Tasks, Criteria)
• How do you want to test? (Methodology)
• How will the test results be communicated and used? (Reporting)

Methodology versus Reason or Purpose (pg. 611)

Chapter 30 - Usage Data

• Observation
  • Direct
  • Indirect: Video recording
  • Analysis
    • Task Based
    • Performance Based
• Verbal Protocols
  • Thinking Aloud
  • Post-Event

Software Logging / User Opinion

• Software Logging
  • Time stamp keypresses or events (automatic)
  • Log interaction (manual)
• User Opinions
  • Interviews (Structured<---->Flexible)
  • Questionnaires (Closed<---->Open)
  • Surveys

Chapter 31--Experiments/Benchmarking

• Experiments
  • Hypothesis [outcome =f(manipulated)]
  • Manipulated variables
  • Control conditions or variables
  • Measured outcomes
  • Study design
  • Statistical tests
• In practice, HCI experiments are generally iterative paths of product or UI design

Usability Engineering

• Define usability goals in terms of metrics
• Set criteria, though, e.g., benchmarking
• Develop solutions and prototype them
• Measure and/or analyze impacts
• Incorporate user feedback in the re-design
• Iterate design-evaluate-design until the criteria are achieved

Chapter 32 - Interpretive Evaluation

• Premise: All data are subjective, because they are interpreted
• Contextual inquiry (evaluate while in use)
• Cooperative evaluation (designers & users)
• Participative design & evaluation (users have control)
• Ethnography (immersion in the user)

Predictive Evaluation

• Inspections (standards, consistency, etc.)
• Usage simulations by experts
  • Structured reporting (heuristic, discount testing, walkthroughs)
  • Unstructured reporting
  • Predefined list of categories
• Modeling

Chapter 34--Comparing Methods

• Purpose
  • Engineer to target: Is it good enough?
  • Compare designs: Which is the best?
  • Understand the real world: Does it work?
  • Conform to standards? Yes <----> No
• Stage of development
• Involvement of users
• Type of data and measures
• Practical considerations

Research and Scientific Factors

• Validity: Are we measuring what's required to fulfill the purpose of the study?
• Reliability: Are the measurements consistent?
• Biases: Are measures selective or manipulated?
• Scope: Are the results complete with respect to the eventual use of the system?
• Ecological validity: Are the results valid for the environment where the system will be used?

Evaluation Summary

• Usability evaluation provides disciplined and systematic feedback and feedforward to flow between users and developers.
• To be successful, usability evaluations must be built into the product design and development cycle.
• Usability evaluations must meet both business needs and research standards