CS 569/589 Winter 2014
Empirical Lab Studies (Quantitative) of Programming

Instructor: Dr. Burnett
Office: KEC 3051
E-Mail: burnett@eecs.oregonstate.edu

Dr. Burnett's Office Hours are listed on my home page

Course Description

The course is about empirical methods in understanding humans' use of languages, environments, etc., in software development. There are many possibilities about empirical methods, and we can't cover them all. Thus, this course will focus on a method that is very useful for certain kinds of questions about the HCI (human aspects) of software development, but is not well understood in computer science: how to scientifically conduct and analyze (statistically oriented) laboratory studies with human participants.

This course will cover how you go about designing, preparing for, running, analyzing, and writing-for-publication quantitative (statistical) lab experiments of programming situations involving human participants. This is an end-to-end coverage of the entire process, and will put you in a position to conduct lab studies of your own with human participants.

Note that this is not a statistics course, although we will cover a couple of basic stats. If what you really want is a statistics-for-experiments course, I recommend the Stats 511 and/or Stats 515 courses, which are excellent, and are specifically targeted to non-stats grad students.

Course objectives

1. Choose when a (statistically oriented) lab study is the right choice of empirical work.
2. Design, conduct, and gather data in such lab studies...
3. ... according to accepted ethical principles of dealing with human subjects.
4. Analyze data in lab studies using quantitative (statistical) methods.
5. Report quantitative lab study empirical work in research publications.

How to sign up

This course is cross-listed under both CS 569 (Special Topics in Software Engineering) and as 589 (Special Topics in Programming Languages). Register for 4 hours of either one (but not both). CS 589 counts toward the Languages category of requirements, and CS 569 counts toward the Software Engineering category, so you should choose whichever one helps you the most to fulfill your degree requirements.

Prerequisites

Contrary to what the catalog or scheduling system might say, the only prerequisite for this course is grad standing in Computer Science.

How the course will be conducted, method of instruction

You'll actually do a lab study of some programming language, tool, or practice, with a team member. You and your teammate will choose the language/tool/practice you want to study:

• Any idea within the scope of this course will be fine; it does not have to relate to your research or thesis activity. However, it is allowed to relate to your research if you want it to (read on).
• Optionally, this can be a study you need for your research, with the IRB approval already in place very early in the course (so it is probably something your advisor has been planning to do for awhile).
• Optionally, what you do in this class could turn out to be a pilot in preparation for a "real" study related to your thesis and/or research.

I'm anticipating little or no programming in this course. There will be some lectures by me; about half of the classes will be like this. The other half of the class will be more studio/discussion style, with teams discussing and critiquing each others' work, and based on team presentations in which the class jointly provides feedback on some aspect of a team's case study. In short, it will be highly interactive.

Textbooks

• Required, ISBN/SKU 9781848000438, Shull et al., Guide to Advanced Empirical Software Engineering, Springer, 2008.

We will also have selected readings from other sources, but you don't have to buy those.

Components of your grade

There will be one midterm, one final exam, assignments (most of which will contribute toward your team project), and a final presentation of your study. In addition, every student will need to participate as human subjects in 3 (tentative number) of the studies your classmates are conducting.

Note: This class does not have a TA, and so not all assignments can be graded formally. You will not know in advance which assignments will be graded formally and which will not.

I have high expectations, and expect performance worthy of graduate students in computer science. Thus, in this class, "A" does not mean "adequate" or "nothing wrong" -- it means "excellent". For an A, you should expect to dig deep and get the most you can out of the class.

Weights:

• Midterm = 20%
• Final exam = 25%
• Interim assignments and presentations = 30%
• Final project = 25%

Tentative schedule

Subject to change! Please check back every week for updates.

• Week 1 (Jan 6-...): Introduction to empirical studies.
  • Types of empirical studies, when to use what.
  • Ethical issues when working with human subjects.
  • ==Read chapter 11: sections 1-5 and section 8.
  • ==Skim chapter 9.
  • ==HW#1: Take the "CITI" or "NIH" IRB ethics of human subjects tutorial. Hardcopy certificate due in class Wed Jan. 8.
    • If you have previously obtained this certificate, turning in a copy of your old one is fine -- you do not have to take the tutorial again.
    • It is wise to keep a pdf of this certificate for reuse later. The IRB sometimes loses their copy of these, so saving a copy is wise.
  • ==HW#2 (see below for due date): Start thinking about who your team member should be and what you would like to do for your study project. Here is a resource page.


• Week 2 (Jan 13-...): Designing statistical studies.
  • ==Read from the Ko et al. 2013 paper: Skim section 1, read sections 2 and 4. Note: you can access this paper free of charge via any OSU computer, or by going thru the OSU Library's e-journals page.
  • Win-win research questions.
  • What types of data you might collect to answer the research questions.
  • The concept of statistical significance, statistical dangers (noise and where it can come from, type I errors, type II errors).
  • The humans: population sources (real people, mechanical turks, ...), between-subject vs. within-subject, how many groups, random assignment, counterbalancing, Latin Squares.
  • Designing the tutorial.
  • ==HW#2: Choose your platform, initial research questions, teammate. Here is a resource page. Turn in electronically by sending email to Dr. Burnett (my lastname at eecs.oregonstate.edu) by Mon. Jan. 13, 9:00 am
  • ==HW#3a: Design your tutorial. Turn in electronically by Fri. Jan. 17, 9:00 am.
  • ==Presentation: On Fri. Jan. 17, one team will present their tutorial, and the rest of us will critique it. That team is: Michael/Will/Sergii


• Week 3 (Jan 20-...): Designing statistical studies (cont.)
  • Monday Jan. 20: no class (university holiday)
  • ==HW #3b: Critique the team's tutorial you have been given. (The presenting team is exempt from this assignment.) Turn in electronically by Wed. Jan. 22, 9:00 am. Your team to critique is:
    • Mihai/Caius: critique Iftekar/Arpit/Soroush
    • Iftekar/Arpit/Soroush: critique Beatrice/Archana/Chad
    • Beatrice/Archana/Chad: critique Anahita/Jianqing
    • Anahita/Jianqing: critique Mihai/Caius
    • Rithika/Isneet/Subarna: critique Catharina/Pratik
    • Catharina/Pratik: critique Rithika/Isneet/Subarna
    • Michael/Sergii/Will: N/A
  • Surveys/questionnaires and log data: how to do them, problems to watch out for.
  • ==Read chapter 3 (more readings may be added too).
  • ==HW #4a: Design the data collection part of your experiment plan. Turn in electronically by sending email to Dr. Burnett (my lastname at eecs.oregonstate.edu) by midnight Mon, Jan. 27, to fit into the structure described on the projects page (you can add roles and other things that don't quite fit into their structure at the end). Your homework should include the following information:
    • Data: What data you'll collect: questionnaires, log data files, any other data you intend to collect. For questionnaires, include the questionnaire itself. For log data files, include the list of data items you'll have in the log file. For anything else, what you'll measure from it and how.
    • Include the research questions (you turned these in before, but you can update them now if you want to) -- and with each, point to the data item that will enable you to answer that question.
  • ==Read from the Ko et al. 2013 paper: Sections 3 and 5.
  • ==Read the Kulesza et al. EUD'11 paper (sample experiment paper).


• Week 4 (Jan 27-...): Designing (cont.)
  • The following team will present their HW 4a on Mon, Jan. 27: Anahita & Jianqing.
  • Designing the task, materials, time limits.
  • ==HW #4b: Design the rest of your experiment plan. Turn in electronically by sending email to Dr. Burnett (my lastname at eecs.oregonstate.edu) by midnight Mon, Feb. 3, structured as described on the projects page (you can add roles and other things that don't quite fit into their structure at the end). Your homework should include the following information:
    • All human subject decisions: How many subjects, Mechanical Turks or "live", within/between, how you will recruit, whether you'll use Latin square, randomization plan, etc. (This is for your real experiment -- the CS569/589 version will have only 4-5 subjects, but we'll find ways to pretend like you got more.)
    • Experiment roles: who on your team will fulfill what roles.
    • Procedures, task, and related materials: what are your subjects going to do and in what order. If there are materials they'll use during the task (eg, source code, prototype, handouts), include pictures of them in the packet you turn in.
  • ==Read about Mechanical Turk: Amazon Mechanical Turk. Requester Best Practices Guide.


• Week 5 (Feb 3-...): Formative user studies and pilots (studies to perform before the "real" study, aka "studying for the test")
  • The following team will present their solution to HW 4b on Mon, Feb. 3: Pratik & Catharina.
  • ==Read from the Ko et al. 2013 paper: Section 8.
  • Wed is a "lab day": Bring your remaining experiment design questions/issues (HW 4a-4b) to discuss in class.
  • ==HW #5a: Run 1 to 3 sandbox pilots. Due midnight Wed, Feb. 12. Turn in the following electronically by midnight. (Any of ppt, doc, pdf, or txt will be fine):
    • Any problems found (with fixes) with your tutorial(s).
    • Any problems found (with fixes) with the design/use of your prototype for this experiment.
    • Any problems found (with fixes) with your questionnaire(s).
    • Any problems found (with fixes) regarding answering your RQs with the data actually ollected.
  • ==HW #5b: Send a brief blurb about your project to the class mailing list to recruit participants from the class. (Just for us (your friends), nothing fancy: a brief description, qualifications you seek, and any time constraints.) As with HW #5a, it is due midnight Wed, Feb. 12.
  • Friday's lecture (which was to be on formative/pilots) was canceled due to OSU's weather closure
  
  
• Week 6 (Feb. 10-...): Theory. Also Formative user studies and pilots (studies to perform before the "real" study, aka "studying for the test")
  • Monday: formative/pilots. And start theory (whew!)
  • The following team will present their solution to HW 5a on Wed, Feb. 12: Beatrice, Archana, Chad.
  • ==Midterm exam will be Friday. (Covers lecture/readings through Week #5.)
  • Empirical studies and theory
  • ==Read chapter 12.
  • ==Project: Sometime between Feb. 12 and Feb. 23, actually conduct your study, using 5 people from the class as your participants. (If you need the EUSES lab for this, coordinate with Dr. Burnett.)


• Week 7 (Feb. 17-...): Statistically analyzing the data
  • ==Project: When you've finalized your design, you should update your write-up of your experiment's design, procedures, etc. (See my Papers page for examples. Good choices for statistical examples include 2010/#4, 2008/#1, 2008/#6. Section 5 of 2010/#3 provides a different kind of statistical example, but had to be very short due to space.)
  • ==HW #6: Be a participant ("subject") in two of the empirical studies your classmates are running
  • ==Read chapter 6.
  • Here is a directory of R-related examples.


• Week 8 (Feb 24-...): Statistically analyzing (cont.)
  • ==Project: No later than Feb. 24, you should start statistically analyzing your data.
  • Wednesday: Team presentation of their statistical analyses, as well as problems, issues in their statistical analysis. The presenting team will be: Subarna, Ishneet, Rithika.
  • Friday is a "lab day". Bring your statistical issues in some kind of sharable form, and we'll brainstorm about them together.


• Week 9 (Mar 3-...): Validity, start Project presentations:
  • ==Read chapter 11's section 7, revisit chap 6's section 3.2, handout from Wohlin et al.
  • last half of Monday: Team presentation of their "Sensemaking/story telling". The presenting team will be: Iftekhar, Arpit, Soroush. (This presentation will probably be shorter than the early ones, so only one half class is scheduled for it.)
  • Wednesday: Team presentation of their "Threats to Validity" section. The presenting team will be: Mihai/Caius. (This presentation will probably garner less discussion than the early ones, so we will probably resume discussion sensemaking/why's partway thru the class.)
  • Friday: final project presentation by: Mihai/Caius.


• Week 10 (Mar 10-...): Project presentations.
  • Monday: presentations by: Will/Sergii/Michael, Anahita/Jianqing.
  • Wednesday: presentations by: Pratik/Catharina, Ishneet/Subarna/Rithika.
  • Friday: presentations by: Iftekar/Arpit/Soroush, Archana/Beatrice/Chad.
  • ==Turn in Final project (email) by midnight Friday, March 14. What to turn in:
    • Your powerpoint (can be updated).
    • Your written report. You should combine your write-up of your design and research questions that you did earlier (updated if needed) with your new "Results section" that describes the statistical results in the usual way (as per acceptable standards). It should look like the Results section of an empirical research paper.


• Finals week: Final exam is on Tues., March 18, 2014, from 12:00 noon to 1:50 pm. (Note that this date is fixed, so make your vacation plans accordingly.)

Things I may work in if time permits:

• Replication (Chapter 14)

Handy link to internal page.

Margaret M. Burnett
Date of last update: Feb. 25, 2014