Terms to Know

Questions

1. What is the different between empirical and non-empirical methods?

2. Which of the methods discussed in the chapter require a working product (or at least a good interactive or Wizard-of-Oz prototype)?

3. Suppose your company is evaluating its spreadsheet product. What differences would you expect to see in the feedback obtained from a co-discovery session, and those from a user workshop?

4. Why do think-aloud protocols generally provide more information that is relevant to upgrading a software product than less structured methods, such as co-discovery?

5. Suppose you are trying to decide which method(s) to use in order to re-design an existing product to make it more usable. The options you are considering are:
   • camera conversations
   • co-discovery
   • focus group
   • user workshop
   • think-aloud protocols
   • usage logs
   • field observation
   For each of them, indicate the "costs" in terms of three criteria:
   1. Amount of investigator time required (should include time to set-up or otherwise prepare for sessions)
   2. Amount of time required per-user
   3. Amount of time to analyze the results and draw conclusions about how the software should be improved
   For example, costs of a user workshop would be set-up and other preparation time for the investigator; for each user, it would include travel to/from the workshop plus the time spent at the workshop (likely to be a few hours long); and for analyzing results, it would probably take a few hours to read over the notes from the workshops and formulate conclusions.

6. Now suppose you are working on a totally new product. Which of the methods you discussed in the previous question are still appropriate? Justify each answer.

7. What is it about the valuation method that distinguishes it from other ways of having users indicate their preferences about planned features (focus groups, feature checklists, questionnaires, and interviews)?

8. Suppose your company is working on a totally new release (i.e., major changes) of its most popular product, a word processor. You have been assigned to figure out if any controlled experiments would be useful. What is your response? If yes, give 1-2 examples of what experiments might be run. If no, justify your answer.

9. Which of the empirical methods could be implemented in some form of electronic survey?

10. Compare and contrast the four non-empirical methods in terms of the following:
    • expertise required to carry out the evaluation
    • types of problems likely to be uncovered
    • likely cost to implement