--On Wednesday, November 14, 2001, 1:22 PM -0800 Smets Philippe
<psmets@ulb.ac.be> wrote:
> At Queen Victoria time, the British Parliament passed a law making
> illegal the homosexuality between male partners. Why only male, and
> not female? Because making illegal the homosexuality between females
> partners requires first to acknowledge that such a thing could exist,
> an idea totally unacceptable at that time.
>
> To establish a double blind review system requires first to
> acknowledge that reviewers might be dishonest. Is that acceptable?
Actually, I believe that we are facing a task that is easier than that
faced by the British Parliament at Queen Victoria's time.
There is a substantial body of literature in the field of
experimental design that testifies to the existence of so-called "subject
experimenter artifacts." These refer to (1) subjects' behavior that leads
to biased results, (2) biases in experimenter's perception of results of
empirical studies, and (3) interactions between the subjects and the
experimenters. These effects are so strong that it is a common knowledge
that results of empirical studies should be trusted unless they have been
based on some kind of blinded design. We know of subjects being affected
by the treatment (placebo effect). An acceptable remedy for that is
blinding the subjects, so that a subject does not know whether he/she/it is
being treated or not. A similar thing happens at the experimenter side,
when the experimenter makes unconscious errors in reading and
interpretation of data (so, no malicious intent!) when he/she knows what
he/she is looking for. The remedy for this is double-blinding so that
neither the subject nor the experimenter know which treatment is being
applied or which data is being collected or looked at. In a hospital
setting, a nurse is asked to administer a drug and he/she does not know
which of the pills that he/she is giving to a patient is medicine and which
is placebo. A consciencious physicist may purposefully decalibrate an
instrument, take readings that are meanigless to him/her, and then, at the
very end of the experiment, when all the data have been collected and only
need to be plugged into a formula and analyzed, transform the collected
data into true, meaningful measurements.
A spectacular example that involves the second and the third effect is due
to, I believe, Rosenthal (I don't have the reference handy). It describes
a study that tests subject-experimenter effects in empirical studies. Two
groups of students (sections of the same class?) were asked to teach rats
to find food in a maze and measure the learning times. The first group was
told that the rats belonged to a special breed of "maze-smart" rats. The
second group was told that the rats they received were just ordinary stupid
laboratory rats. There was an amazingly large difference in the results --
the "maze smart" rats learned much faster than the "ordinary rats" and, as
you have probably guessed, both groups of rats came from the same breed.
How does this relate to double-blinding in the UAI reviewing process? If
merely framing a problem can induce a different behavior in humans and if
even rats can respond differently to treatment, what can we expect from a
problem as complex as reviewing a scientific paper? While, as some of my
predecessors noted, it is a matter of balance between the amount of work on
the part of the authors (hiding their identity) and the program chairs
(organizing the double-blinding reviews) and the resulting improvement in
the quality of the reviews and the conference program, there is no doubt to
me that double-blinding should improve the process. There is no need to
refer to dishonesty. Even when we try to be honest, we may be
"dishonest" enough :-).
Marek
--------------------------------------------------------------------------
Marek J. Druzdzel http://www.pitt.edu/~druzdzel
This archive was generated by hypermail 2b29 : Mon Nov 19 2001 - 10:29:51 PST