ECOVAL: A Framework for Increasing the Ecological Validity in Usability Testing

Please download to get full document.

View again

of 46
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Categories
Published
Paper presented at IEEE HICSS'2015 conference. The term ecological validity refers to the study of the measurable correspondence between experimental design and ecological setting (i.e., representative users performing real tasks in their natural environment). Therefore, it is important to anyone intending to generalize the conclusions from the particular circumstances of an experiment to wider ecological situations. However, there is no operational definition that enables to objectively quantify the ecological validity of an experimental design. The ECOVAL framework offers a formal starting point to bridge this gap by helping both define and increase the ecological validity in usability testing. A methodology, a tool and guidelines to increase ecological validity are presented in an industrial case study. The return on investment of increasing ecological validity is examined for both practical and theoretical purposes. Based on these findings, recommendations to increase the ecological validity in usability testing are introduced, contributions are discussed, and outlines for future work are presented.
  • 1. Suzanne Kieffer ECOVAL: A Framework for Increasing the Ecological Validity in Usability Testing Jean VanderdoncktUgo Braga Sangiorgi Université catholique de Louvain Louvain School of Management Louvain-la-Neuve, Belgium
  • 2. Ecological validity Experimental design Natural setting x x x xx x xx x x x x x x x x Environment Tasks Stimuli
  • 3. Ecological validity High Low Experimental design Natural setting ≈ Experimental design Natural setting ≠
  • 4. Internal validity Was the study well done? External validity Can we generalize the findings to particular persons, settings and times? Ecological validity To which extent can we generalize the findings?
  • 5. User interface design Ecological validity Overall validity Conclusions of usability evaluations
  • 6. What’s the problem with ecological validity?  No formal definition that enables  To objectively quantify the ecological validity of experimental designs  To capture the critical attributes of the real-world environment
  • 7. LOGICAL IDITYVALECO ECOVAL A framework for increasing the ecological validity in usability testing
  • 8. Outline  The ECOVAL framework  ECOVAL  Guidelines  GAMBIT  Case study  Operability and usefulness of ECOVAL  ROI of increased ecological validity  Recommendations
  • 9. The ECOVAL framework
  • 10. The guidelines
  • 11. No objects
  • 12. Mock objects
  • 13. Real objects
  • 14. Only verbalized
  • 15. Mimicked/verbalized
  • 16. For real
  • 17. GAMBIT – the supporting tool  Production of prototypes  Easy  Quick  Cost-effective  Real user interface performances  Interaction  Navigation  Responsiveness
  • 18. GAMBIT – how to 1 Produce screens
  • 19. GAMBIT – how to 2 Create behaviour
  • 20. GAMBIT – how to 3 Conduct experiment
  • 21. GAMBIT – how to 4 Download log file
  • 22. • Q1: To which extent is ECOVAL applicable for controlling the ecological validity? • Q2: What are the possible outcomes/benefits of increased ecological validity ?
  • 23. Case study
  • 24. Hot-Dip Galvanizing on Continuous Lines
  • 25. Monitoring report Task Control Result Task Control Result Artefact: paper checklist Section > Subsection Function – Name - Date – Shift Section > Subsection 300 items a day
  • 26. Monitoring report Task Control Result Task Control Result Artefact: paper checklist Section > Subsection Function – Name - Date – Shift Section > Subsection
  • 27. Timeline Task analysis User profile Usability goal setting Screen design Usability evaluation
  • 28. Usability evaluation Paper versus GAMBIT prototype
  • 29. Ecological validity Ratio Paper-to-Gambit for ecological validity of 1:1.17
  • 30. Balsamiq mockups
  • 31. Experimentations Paper GAMBIT 5 key users x 2 10 participants (30’) 18 participants (1 hour) 5 key users x 3 + 3 user representatives V1 6 participants V2 12 participants
  • 32. New version of the prototype
  • 33. Apparatus
  • 34. Results and discussion
  • 35. Early detection of usability problems  Increased user efficiencies  Increased user satisfaction
  • 36. Cost-benefit analysis 130 hours 18 hours
  • 37. Cost-benefit analysis (continued) $74,480 $194,480
  • 38. Paper GAMBIT Ecological validity 1 1.17 Data collection Manual & face-to-face Difficult & tedious Unstructured process Obtrusive Automated & remote Comfortable Structured process Unobtrusive Outcomes Compliant with the literature Early detection of usability problems Increased user productivity Decreased late design changes Increased organizational efficiency
  • 39. Recommendations
  • 40. Take away  {ECOVAL, Guidelines, GAMBIT}  Case study  Operability  Usefulness  ROI  Recommendations
  • 41. Contact details suzanne.kieffer@uclouvain.be ugo.sangiorgi@uclouvain.be jean.vanderdonckt@uclouvain.be Thank you!
  • We Need Your Support
    Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

    Thanks to everyone for your continued support.

    No, Thanks