CS 589 Fall 2011
(To understand humans' use of languages, environments, and practices of software development)
- Study some software you and/or your advisor care about, that somehow relates to supporting some kind of software development.
- Compare programming language A against language B. (See my papers page, 1993 #3, for an example of this kind.)
- Compare debugging tool A against debugging tool B.
- Study effects of some kind of testing tool. (There are several examples like this on my papers page).
- Study effects of some code-change management tool. (An example like this can be found in IEEE VL/HCC 2011, co-authored by Duc Le and Martin Erwig.)
- Compare people's ability to productively use library A vs. library B.
- Study effects of some software visualization tool.
- Compare people's ability to productively use API X vs. writing their own code instead to achieve the desired functionality.
Standard reporting of empirical studies
Derived from APA guidelines.
Usually these are included in the paper's Introduction section. If they aren't
there (or if only an overview question is there), put them in full detail here.
This section must be very detailed and clear. It tells the
reader that someone else can repeat the experiment just by
reading your method section. The method section generally
consists of three subsections: participants, apparatus (or
materials), and procedure. A fourth, optional, subsection is
The age, sex, and any other relevant demographic data are
presented here. State how many subjects participated, how they
were selected, and how they were assigned to groups.
Apparatus or Materials
A description of the apparatus used is given here. In the case
of standard laboratory equipment, rather than describing the
entire apparatus, the company name and model and/or serial number
is sufficient. If this is not possible, the equipment should be
described in detail.
If materials (such as a questionnaire) were used, either cite
your source (if published materials were used) or provide a copy
in the appendix of your paper if you devised the instrument
yourself. You should describe the instrument in your materials
section. For example,
A 50-item six-point Likert-type questionnaire was devised by
the experimenter to measure attitudes toward authority
figures. Half of the questions were worded such that....The
highest (positive) score that could be attained on the
measure was 300; the lowest (negative) score was 50. Thus,
higher scores reflected more positive attitudes toward
This section describes what the experimenter did and how it was
done. It is a detailed description of the events that the
experimenter went through from the beginning until the end of the
study. Such things as experimental and control group assignment
to conditions, order or manner of experimental treatment
presentation, and a summary of the instructions to the
participants are presented here. Include a statement about your
research design and the operational definitions of your
variables. (If your design is complex, a separate section can be
designated for this information.)
This section is often very short and combined with Procedure, but for this class
we want it in detail, and separate. This will enable us to think through your
experiment plan in advance.
This section is where you present your data and analyses. The
experimenter gives a description and not an explanation of the
findings of the experiment. In order to fulfill this
requirement, the results section should include descriptive
statistics (rather than the raw data) and statistical tests if
used. Include degrees of freedom used, obtained values of
inferential statistics performed, probability level, and
direction of effect. Underline letters used as statistical
symbols, such as "N", "F", "t", "SD", and
"p." (Use underlining, not quotation marks. Since
many Web browsers using underlining to indicate a link, avoid
underlining within web pages.) Make reference to any figures and tables used,
for example, "(see Table 1)."
The reference to the table or figure should be close to the
relevant material in the text. Never use a figure or table
without referring to it in the text.
Tables are often used when presenting descriptive statistics such
as means, standard deviations and correlations. Pictures,
graphs, and drawings are referred to as figures. You should use
as few tables and figures as possible. They should be used as
supplements, not to do the entire job of communication. (See
the APA manual for detailed guidelines for Tables and for Figures.)
Generally, one reports descriptive statistics, then inferential
statistics, then states in words what was found.
In this section, you state your conclusions on the basis of your
analyses. The conclusions should be related to the questions
raised in your introduction section. How is this study, and
these results, relevant to the field? You should open the
discussion section with a statement of support or nonsupport for
your original hypothesis. You may want to point out differences
or similarities between other points of view and your own. You
may remark on certain shortcomings of the study, but avoid
dwelling on flaws.
In general, this section allows you
relatively free rein to examine, interpret, and qualify your
Other resources for your project
Margaret M. Burnett
Date of last update: Oct. 20, 2011