Adversarial collaboration

I had a good idea (see the “when two or more scholars” paragraph), but turns out someone else had it first. (That’s how it usually goes.)

Do Frequency Representations Eliminate Conjunction Effects? An Exercise in Adversarial Collaboration.

Barbara Mellers, Ralph Hertwig, and Daniel Kahneman.

The present article offers an approach to scientific debate called ad- versarial collaboration. The approach requires both parties to agree on empirical tests for resolving a dispute and to conduct these tests with the help of an arbiter. In dispute were Hertwig’s claims that fre- quency formats eliminate conjunction effects and that the conjunction effects previously reported by Kahneman and Tversky occurred be- cause some participants interpreted the word “and” in “bank tellers and feminists” as a union operator. Hertwig proposed two new con- junction phrases, “and are” and “who are,” that would eliminate the ambiguity. Kahneman disagreed with Hertwig’s predictions for “and are,” but agreed with his predictions for “who are.” Mellers served as arbiter. Frequency formats by themselves did not eliminate conjunction effects with any of the phrases, but when filler items were removed, conjunction effects disappeared with Hertwig’s phrases. Kahneman and Hertwig offer different interpretations of the findings. We discuss the benefits of adversarial collaboration over replies and rejoinders, and present a suggested protocol for adversarial collaboration.

Psychological Science, 2001 vol. 12 (4) pp. 269-275

From the article, a guide to adversarial collaboration:

1. When tempted to write a critique or to run an experimental refutation of a recent publication, consider the possibility of proposing joint research under an agreed protocol. We call the scholars engaged in such an effort participants. If theoretical differences are deep or if there are large differences in experimental routines between the laboratories, consider the possibility of asking a trusted colleague to coordinate the effort, referee disagreements, and collect the data. We call that person an arbiter.

2. Agree on the details of an initial study, designed to subject the opposing claims to an informative empirical test. The participants should seek to identify results that would change their mind, at least to some extent, and should explicitly anticipate their interpretations of outcomes that would be inconsistent with their theoretical expectations. These predictions should be recorded by the arbiter to prevent future disagreements about remembered interpretations.

3. If there are disagreements about unpublished data, a replication that is agreed to by both participants should be included in the initial study.

4. Accept in advance that the initial study will be inconclusive. Allow each side to propose an additional experiment to exploit the fount of hindsight wisdom that commonly becomes available when disliked results are obtained. Additional studies should be planned jointly, with the arbiter resolving disagreements as they occur.

5. Agree in advance to produce an article with all participants as authors. The arbiter can take responsibility for several parts of the article: an introduction to the debate, the report of experimental results, and a statement of agreed-upon conclusions. If significant disagree- ments remain, the participants should write individual discussions. The length of these discussions should be determined in advance and monitored by the arbiter. An author who has more to say than the arbiter allows should indicate this fact in a footnote and provide readers with a way to obtain the added material.

6. The data should be under the control of the arbiter, who should be free to publish with only one of the original participants if the other refuses to cooperate. Naturally, the circumstances of such an event should be part of the report.

7. All experimentation and writing should be done quickly, within deadlines agreed to in advance. Delay is likely to breed discord. 8. The arbiter should have the casting vote in selecting a venue for publication, and editors should be informed that requests for major revisions are likely to create impossible problems for the participants in the exercise.