CS 584: Human Factors of Programming Languages, Winter 2008

Instructor: Dr. Burnett
Office: KEC 3051
E-Mail: burnett at eecs dot oregonstate dot edu
Dr. Burnett's Office Hours are listed on my home page

What's new: changes and announcements (it's wise to check this every couple of days).

Course Objectives

CS 584, Human Factors of Programming Languages, is a four-credit course for graduate students. Programming languages today have been designed primarily with machine efficiency in mind, with little attention given to the efficiency of the human doing the programming. Since labor costs far exceed machine costs in software development, this past way of designing programming languages is no longer appropriate. This course presents basic principles and evaluation methods for designing and evaluating programming languages that emphasize human productivity and problem solving.

The theme for this term is "Consumer Programming Languages/Environments". Students will each pick their own "consumer", whether it be a particular professional programmer, a business owner doing their own web applications, a school teacher setting up educational simulations, or whatever. This consumer will motivate (almost) everything the student does all term. Finally, by the end of the term, the student will present a design of a (portion of) a language or environment for this consumer.

The course will focus on the following topics, drawing from both emerging research and from fundamentals established over the last 20+ years. Therefore, students should come away with a fundamental understanding of:

• What is known about human factors in programming.
• Techniques for evaluating programming languages/environments from a human factors perspective.
• Language design techniques and problems for programming languages for humans
• Case studies of a few languages/environments that have been designed with human factors of particular consumers as the primary design goal
• Dr. Burnett's projects as case studies: Topics from end-user programming and end-user software engineering
• Critical thinking about human factors in programming.
• Critical thinking about doing quality research relating to human factors in programming.

Grading

Your grade will be computed as follows:

• 25% Homework (Late policy: No late homeworks accepted.)
• 25% Presentations and other in-class contributions (Unless otherwise stated, each presentation should cover at least 4 discussion questions, plus 1 application specific to your particular "consumer". There will be a firm time limit, including discussion time, on each presentation.)
• 25% "hands-on test" of ability to evaluate programming languages/environments from a human perspective. Your task will be to turn in a written evaluation of the human aspects of some projects I will assign to you by applying what you have learned in CS 584.
• 25% Final project

Prerequisites

Graduate standing in CS and interest in programming language design issues, or permission of the instructor. Senior undergraduates who are interested in taking this course as an elective should discuss this with Dr. Burnett during her office hours.

Frequency

The course is offered only every second year at most.

Education Methods

This is not a lecture-oriented class. I will do some lectures, but the class is very active-learning based. Thus, it will be quite critique-oriented, with student presentations and analyses regarding research papers and with a strong element of group critiques as work begins to emerge. No textbook -- we will be reading papers instead. No programming is expected, except a little of it might come in handy in the final project. Students should approach the course with the attitude that they get out of it what they put into it.

Tentative Schedule

• Week 1 (week of 1/7):
  • M: Intro to the class, intro to research on people's success/failures in software development, good research vs. bad research in this area. Type: HCI foundation/technique.
  • W: Class contribution due.
  • W: A few theories that can be applied. Introduction to how memory works. Type: HCI foundation/technique.
  • F: Dr. Burnett: One consumer -- the end-user programmer. Type: People/tasks.


• Week 2 (week of 1/14):
  • S, 2:00: HW 2 due.
  • M: Chris C., Tim: Software Design: Theoretical Approaches (ch 3, Detienne 2002). Omit Cognitive Dimensions (we will do them in more depth later). Type: HCI foundation/technique.
  • T, 2:00: HW 3 due.
  • W: Eric W.: Pane et al. 2002, "natural" programming by children Type: Case studies: language (for "create" task), informed by serious attention to the intended audience/consumer
  • W: Dr. Burnett, continues the discussion of formative empirical approaches, the Natural Programming project, Hands.
  • Th, 2:00: HW 4 due.
  • F: Paul: Cherubini et al. 2007, professional programmers diagramming. Type: People/tasks.
  • F: Ethan: Segal 2007, scientific programmers Type: People/tasks.


• Week 3 (week of 1/21):
  • M: No class (MLK Day)
  • T, 2:00: No homework due today after all.
  • W Program committee meeting: 3 submitted papers. Type: Case studies
  • Th 2:00: HW 1 due. This is the one in which you pick your consumer and task.
  • F: Ethan, Chris C.: Green/Petre 1996: Cognitive Dimensions Type: HCI foundation/technique. Note: no discussion questions are required for this paper. Instead, you (and each of the presenters) should try to become proficient at 3-4 CDs of your choice.


• Week 4 (week of 1/28):
  • S, 2:00: HW 5 due.
  • M: Dr. Burnett: Yang et al. 1997, Representation Benchmarks Type: HCI foundation/technique.
  • T, 2:00: HW 6 due.
  • W: Victoria: Lawrance et al. 2008, information foraging in debugging/maintenance Type: People/tasks.
  • W: Dr. Burnett continues the discussion of information foraging. Type: HCI foundation/technique
  • Th, 2:00: HW 7 due.
  • F: Todd: Wilson et al. 2003, Surprise/Explain/Reward Type: Case study: environment for enticing EUPs into "software engineering" behaviors.
  • F: Dr. Burnett: continue discussion of attention investment, minimalist learning, and psychology of curiosity. Type: HCI foundation/techniques.


• Week 5 (week of 2/04):
  • Su, 2:00: HW 8 due.
  • M: Chris B.: Boshernitsan et al. 2007, changing code Type: Case study: environment for "updating" task.
  • M: Heather: Liu and Li 2007, spontaneous design. Type: Case study: environment for UI design by designers.
  • T, 2:00: HW 9 due.
  • W: Jill, Todd: Beckwith 2007 chapter on Gender theory applicable to programming languages/environments Type: HCI foundation/techniques.
  • F: special class. Instead of our usual time/place, class will meet at the colloquium being presented by Andrew Ko starting at 10:45 am.
  Type: "Natural" debugging environment for professional programmers


• Week 6 (week of 2/11):
  • M: no class: (Dr. Burnett at NSF.)
  • Tu, 2:00: HW 10 due.
  • W: Paul: Rashid et al. 2006, encouraging participation Type: People/tasks.
  • W: Chris B.: Leshed et al 2008. How people use CoScripter. Type: People/tasks.
  • Th, 2:00: HW 11 due.
  • F: Eunyoung, Victoria: Coughlin and Johnson 2006, creativity. Type: HCI foundation/technique.


• Week 7 (week of 2/18):
  • Su, 2:00: HW 12 due.
  • M: Randy: Wright et al. 2006, sense-making. Type: Case study: sense-making for analytical problems (like programming).
  • M: Valentina's discussion of sense-making. Type: HCI foundation/technique.
  • T, 2:00: HW 13 due.
  • W: Greg: Stylos and Clarke 2007: APIs. Uses empirical and CD evaluation methods. Type: People/Tasks.
  • W: Chris J.: Abraham and Erwig 2007, type inference in spreadsheets Type: Case study: environment for "finding errors" task.
  • Th, 2:00: HW 14 due.
  • F: Randy: Zimmerman et al. 2007, mixed-initiative programming. Type: Case study.
  • F: Chris J.: Subrahmaniyan et al. 2008, debugging strategies used by end-user programmers. Type: People/tasks.


• Week 8 (week of 2/25):
  • Su, 2:00: Your choice of intellectual tool for the final project is due.
  • Su, 2:00: HW 15 due.
  • M: Jessica: Prabaker et al. 2006, documentation. Type: People/tasks.
  • M: Greg: Tullio et al. 2007, program comprehension (with a program they can't see!) Type: People/tasks.
  • T, 2:00: HW 16 due.
  • W: Eric: Faaborg and Lieberman 2006. Programming by example based on goals. Type: Case study.
  • W: Eunyoung: Nichols et al. 2006. Automatically generating consistent UIs for mobile devices. Type: Case study.
  • (Just to be clear): No HW due for the Friday reading.
  • F: Heather: Shirky essay on Situated Programming. (This presentation has different specs from the other presentations.) Type: People/tasks.


• Week 9 (week of 3/03):
  • M: No class: (Use the extra time to work on your final project presentations)
  • W: Chris J., Jessica: final project presentations
  • F: Paul, Todd: final project presentations


• Week 10 (week of 3/10):
  • M: Randy, Jill: final project presentations
  • W: Chris B., Chris C.: final project presentations
  • F: Heather, Eunyoung: final project presentations


• Finals week (week of 3/17):
  • M 6:00-8:00 pm: Eric, Victoria, Greg, Ethan: final project presentations

An Overview of Papers We Might Cover

We will ultimately cover a subset of a superset of this group, but it gives you an idea of the kinds of papers we're covering. We'll probably read 6-7 of the Foundations papers, and the rest will be a fairly equal division between the remaining two classifications.

How to obtain these papers: Unless otherwise specified, you can obtain them from the ACM Digital Library or the IEEE Digital Library. OSU has subscriptions to both, so all you have to do is (1) be using an OSU IP# and you can just go to the ACM Digital Library or IEEE Xplore directly, or (2) have your OSU ID # ready to type in to access the ACM Digital Library or the IEEE Digital Library (IEEE Xplore) if you're working from elsewhere.

HCI Foundations

• Beckwith dissertation theory chapter (2007) on gender differences
• [Blackwell 2002] Alan Blackwell, First steps in programming: a rationale for attention investment models, Proc. IEEE Human-Centric Computing Languages and Environments, Arlington, VA, Sept. 3-6, 2002, 2-10. This is the best and most up-to-date paper so far on attention investment.
• [Coughlan and Johnson 2006] Tim Coughlan and Peter Johnson, Interaction in creative tasks: ideation, representation, and evaluation in composition, ACM Conference on Human Factors in Computing Systems, Montreal, Canada, Apr 22-27, 2006, pp. 531-540. Good paper! Its about the design of tools to support creative activities. Note: programming is a creative endeavor -- why dont programming environments/languages use these ideas?
• Detienne chapters 3 and 6. (These will be handed out in class as needed.)
• Green/Petre 1996, Cognitive dimensions. Take the 1996 JVLC paper. (Other optional resources on Cognitive dimensions are available too.)
• [Spool et al. 2004] Spool, J., Profetti, C., and Britain, D., Designing for the scent of information, User Interface Eng. (2004). Will be handed out in class as needed.
• Yang et al., Representation benchmarks. Go to Dr. Burnett's home page/Papers link. It's paper #8, 1997.
• TBA, Personas (maybe)

People/Tasks/Situations

• [Cherubini et al. 2007] Mauro Cherubini, Gina Venolia, Rob DeLine, Andrew Ko, Lets go to the whiteboard: How and why software developers use drawing, ACM Conference on Human Factors in Computing Systems, San Jose, CA, Apr 28-May 3, 2007, pp. 557-566.
• [Ellis et al. 2007] Brian Ellis, Jeffrey Stylos, and Brad Myers, The factory pattern in API design: A usability evaluation, International Conference on Software Engineering, Minneapolis, MN, May 20-26, 2007, pp. 302-311. The commonly held belief that the Factory Pattern is better gets debunked empirically here. They then go into a fair amount of CD analysis to see why they got this result.
• Gilmore/Green '84: What/how
• Kissinger et al., AVI'06 on what people want to know
• Ko et al., ICSE'05
• Ko et al., barriers paper
• Lawrance et al., Using Information Scent to Model the Dynamic Foraging Behavior of Programmers in Maintenance Tasks, Joseph Lawrance, Rachel Bellamy, Margaret Burnett, Kyle Rector, ACM CHI 2008, Florence, Italy, April 2008. (CHI'08)
• [Leshed et al. 2008] Gilly Leshed, Eben Haber, Tara Matthews, Tessa Lau, CoScripter: Automating & Sharing How-To Knowledge in the Enterprise, CHI'08 (to appear). CoScripter is an end-user language for web scripting. The paper is about how people use it.
• Petre, What experts want
• [Prabaker et al. 2006] M. Prabaker, L. Bergman, and V. Castelli, An evaluation of using programming by demonstration and guided walkthrough techniques for authoring and utilizing documentation, ACM Conference on Human Factors in Computing Systems, Montreal, Canada, Apr 22-27, 2006, pp. 241-250. This is their work on documenting-by-demonstration. ie, theyre documenting the procedure of how to do something on the computer. Handles multiple examples, if-thens, editing and annotation. There is an evaluation with users with thought-provoking results.
• [Rashid et al. 2006] Al Mamunur Rashid, Kimberly Ling, Regina Tassone, Paul Resnick, Robert Kraut, and John Riedl, Motivating participation by displaying the value of contribution, ACM Conference on Human Factors in Computing Systems, Montreal, Canada, Apr 22-27, 2006, pp. 955-958. So, maybe this could be used for Reward motivating end-user programmers to contribute to a repository of code, a la open source...
• [Scaffidi et al. 2006] Christopher Scaffidi, Andrew Ko, Brad Myers, and Mary Shaw, Dimensions characterizing programming feature usage by information workers, IEEE Symposium on Visual Languages and Human-Centric Computing, Brighton, UK, Sept. 4-8, 2006, pp. 59-62. This is their study of 800+ InfoWeek readers. These were mostly managers and non-programmers, although about 20+% were IT people of some sort.
• [Segal 2007] Judith Segal, Some problems of professional end user developers, IEEE Symposium on Visual Languages and Human-Centric Computing, Couer dAlene, Idaho, Sept. 23-27, 2007, pp. 111-118. Field studies with financial mathematicians, planetary/space scientists, and biologists.
• Shirky: Situated Programming essay
• Soloway/Erlich
• [Stylos and Myers 2007] Jeffrey Stylos and BradMyers, Mapping the space of API design decisions, IEEE Symposium on Visual Languages and Human-Centric Computing, Couer dAlene, Idaho, Sept. 23-27, 2007, pp. 50-57. Design decisions include eg, design pattern choices, naming classes, class design, ...
• [Stylos and Clarke 2007] Jeffrey Stylos and Steven Clarke, Usability implications of requiring parameters in objects constructors, International Conference on Software Engineering, Minneapolis, MN, May 20-26, 2007, pp. 529-538. Big picture theme: testing each API is too expensive; lets instead learn about API design choices that can inform our design of APIs in general.
• [Subrahmaniyan et al. 2008] Neeraja Subrahmaniyan, Laura Beckwith, Valentina Grigoreanu, Margaret Burnett, Susan Wiedenbeck, Vaishnavi Narayanan, Karin Bucht, Russell Drummond, and Xiaoli Fern, Testing vs. Code Inspection vs. ... What Else? Male and Female End Users Debugging Strategies, ACM Conference on Human-Computer Interaction, April 2008 (to appear).
• [Tullio et al. 2007] Joe Tullio, Anind Dey, Jason Chalecki, James Fogarty, How it works: A field study of non-[technical users interacting with an intelligent system, ACM Conference on Human Factors in Computing Systems, San Jose, CA, Apr 28-May 3, 2007, pp. 31-40. End users perceptions of how a machine-learned program works. A field study with 8 workers. Participants early models (which varied a lot between participants!) were remarkably persistent even given counterevidence. Still, their smaller misconceptions did give way to feedback. Participants sometimes did a good job of guessing the machine learning algorithms (in lay terms). Used a series of semi-structured interviews over time, and also surveys.
• A. Marie Vans, Anneliese von Mayrhauser, and Gabriel Somlo. Program understanding behavior during corrective maintenance of large-scale software. International Journal of Human-Computer Studies, Academic Press Ltd., 1999, 51, 31-70.. Type: HCI foundation/technique.
• [Zheng et al. 2006] Q. Zheng, K. Booth, and J. McGrenere, Co-authoring with structured annotations, ACM Conference on Human Factors in Computing Systems, Montreal, Canada, Apr 22-27, 2006, pp. 131-140. I see this as being relevant to EUSE and perhaps allowing the user to explain or supporting collaborative eu development/debugging/maintenance of software.

Case Studies of Languages/Tools for People

• [Abraham and Erwig 2007] Robin Abraham and Martin Erwig, A type system based on end-user vocabulary IEEE Symposium on Visual Languages and Human-Centric Computing, Couer dAlene, Idaho, Sept. 23-27, 2007, pp. 215-222. Did a think-aloud with UCheck on some teachers. The teachers basically were able to use UCheck to find and fix errors, but they had some difficulty tracking down errors that manifested themselves in other cells sometimes. Thus, the authors added an ability to spot irregularities in formula regions (eg, concavities, holes, etc.) as an addition to the fault localization they already do.
• Beckwith et al., VLHCC'05 on software designs to reduce gender gaps
• [Boshernitsan et al. 2007] Marat Boshernitsan, Susan Graham, Marti Hearst, Aligning development tools with the way programmers think about code changes, ACM Conference on Human Factors in Computing Systems, San Jose, CA, Apr 28-May 3, 2007, pp. 567-576. A visual transformation language that allows programmers to update their programs in significant ways.
• Clarke, using personas in API design
• [Faaborg and Lieberman 2006] Alexander Faaborg and Henry Lieberman, A goal-oriented web browser, ACM Conference on Human Factors in Computing Systems, Montreal, Canada, Apr 22-27, 2006, pp. 751-760. Programming by example based on goals.
• [Fernaeus and Tholander 2006] Ykva Fernaeus and Jakob Tholander , Finding design qualities in a tangible programming space, ACM Conference on Human Factors in Computing Systems, Montreal, Canada, Apr 22-27, 2006, pp. 447-456. This is a tangible (ie with physical objects) programming environment for children to collaboratively construct screen-based programs.
• [Kelleher et al. 2007] Caitlin Kelleher, Randy Pausch, Sara Kiesler, Storytelling Alice motivates middle school girls to learn computer programming, ACM Conference on Human Factors in Computing Systems, San Jose, CA, Apr 28-May 3, 2007, pp. 1455-1464.
• [Liu and Li 2007] Alan Liu and Yang Li, BrickRoad: A light-weight tool for spontaneous design of location-enhanced applications, ACM Conference on Human Factors in Computing Systems, San Jose, CA, Apr 28-May 3, 2007, pp. 295-298. I see this as end-user soft. eng. (design stage) for designers at an early stage. Its programming to set up Wizard of Oz experiments, so that they can evaluate their design concepts with human subjects. They do _not_ specify any application logic with this tool -- its strictly for design ideas.
• [Moraveji et al. 2007] Neema Moraveji, Jason Li, Jiarong Ding, Patrick OKelley, Suze Woolf, Comicboarding: Using comics as proxies for participatory design with children, ACM Conference on Human Factors in Computing Systems, San Jose, CA, Apr 28-May 3, 2007, pp. 1371-1374. This technique elicited design ideas very well, even for reluctant children. I think the details of how they did it also made a big difference. Food for thought on how to elicit designs from timid end users (eg, females?) in end-user software engineering design. Particularly interesting from a gender perspective, given Kellehers results about storytelling being effective at motivating females.
• [Myers et al. 2006] Brad Myers, David Weitzman, Andrew Ko, and Duen Horng Chau, Answering why and why not questions in user interfaces, ACM Conference on Human Factors in Computing Systems, Montreal, Canada, Apr 22-27, 2006, pp. 397-406. Very cool. Why didnt it fix my spelling error this time? Or, why did it? Etc.
• [Nichols et al. 2006] Jeffrey Nichols, Brad Myers, and Brandon Rothrock, UNIFORM: Automatically generating consistent remote control user interfaces, ACM Conference on Human Factors in Computing Systems, Montreal, Canada, Apr 22-27, 2006, pp. 611-620. Good paper! Its about automatically creating personally uniform remote controls on your PDA.
• [Pane et al. 2002] J.F. Pane, B.A. Myers, and L.B. Miller, Using HCI techniques to design a more usable programming system. Proc. IEEE Human-Centric Computing Languages and Environments, 3-6 Sept. 2002, 198 - 206.
• [Repenning and Ioannidou 2006] Alexander Repenning and Andri Ioannidou, AgentCubes: Raising the ceiling of end-user development in education through incremental 3D, IEEE Symposium on Visual Languages and Human-Centric Computing, Brighton, UK, Sept. 4-8, 2006, pp. 27-31. AgentSheets goes 3D in terms of what its trying to support (rich media environments, etc.).
• Rothermel et al., WYSIWYT (ICSE'98)
• (Not reading this one this term.) Subrahmaniyan et al., VLHCC'07 on filling information gaps
• [Wilson et al. 2003] A. Wilson, M. Burnett, L. Beckwith, O. Granatir, L. Casburn, C. Cook, M. Durham, and G. Rothermel, Harnessing Curiosity to Increase Correctness in End-User Programming, ACM Conference on Human Factors in Computing Systems, Ft. Lauderdale, FL, Apr. 3-10, 2003, 305-312. Surprise-Explain-Reward
• [Wong and Hong 2007] Jeffrey Wong and Jason Hong, Making mashups with Marmite: Towards end-user programming for the web, ACM Conference on Human Factors in Computing Systems, San Jose, CA, Apr 28-May 3, 2007, pp. 1435-1444.
• [Wright et al. 2006] William Wright, David Schroh, Pascale Proulx, Alex Skaburskis, and Brian Cort, The sandbox for analysis - Concepts and methods, ACM Conference on Human Factors in Computing Systems, Montreal, Canada, Apr 22-27, 2006, pp. 801-810. Its to support people doing analytical sense-making.
• [Zimmerman et al. 2007] John Zimmerman, Anthony Tomasic, Isaac Simmons, Ian Hargraves, Ken Mohnkern, Jason Cornwell, Robert Martin McGuire, VIO: A mixed-initiative approach to learning and automating procedural update tasks, ACM Conference on Human Factors in Computing Systems, San Jose, CA, Apr 28-May 3, 2007, pp. 1445-1454. This is in the Koala class of work, but also might be a case of the Surprise-Reward parts of Surprise-Explain-Reward. Theres also an interesting empirical strategy in which they succeed in teasing apart AIs effects vs. the interfaces effects, perhaps useful for our future efforts.
• TBA, three papers under review . Handed out in class as needed.

Margaret M. Burnett
Date of last update: Feb 28, 2008