Adversarial collaboration

I had a good idea (see the “when two or more scholars” paragraph), but turns out someone else had it first. (That’s how it usually goes.)

Do Frequency Representations Eliminate Conjunction Effects? An Exercise in Adversarial Collaboration.

Barbara Mellers, Ralph Hertwig, and Daniel Kahneman.

The present article offers an approach to scientific debate called ad- versarial collaboration. The approach requires both parties to agree on empirical tests for resolving a dispute and to conduct these tests with the help of an arbiter. In dispute were Hertwig’s claims that fre- quency formats eliminate conjunction effects and that the conjunction effects previously reported by Kahneman and Tversky occurred be- cause some participants interpreted the word “and” in “bank tellers and feminists” as a union operator. Hertwig proposed two new con- junction phrases, “and are” and “who are,” that would eliminate the ambiguity. Kahneman disagreed with Hertwig’s predictions for “and are,” but agreed with his predictions for “who are.” Mellers served as arbiter. Frequency formats by themselves did not eliminate conjunction effects with any of the phrases, but when filler items were removed, conjunction effects disappeared with Hertwig’s phrases. Kahneman and Hertwig offer different interpretations of the findings. We discuss the benefits of adversarial collaboration over replies and rejoinders, and present a suggested protocol for adversarial collaboration.

Psychological Science, 2001 vol. 12 (4) pp. 269-275

From the article, a guide to adversarial collaboration:

1. When tempted to write a critique or to run an experimental refutation of a recent publication, consider the possibility of proposing joint research under an agreed protocol. We call the scholars engaged in such an effort participants. If theoretical differences are deep or if there are large differences in experimental routines between the laboratories, consider the possibility of asking a trusted colleague to coordinate the effort, referee disagreements, and collect the data. We call that person an arbiter.

2. Agree on the details of an initial study, designed to subject the opposing claims to an informative empirical test. The participants should seek to identify results that would change their mind, at least to some extent, and should explicitly anticipate their interpretations of outcomes that would be inconsistent with their theoretical expectations. These predictions should be recorded by the arbiter to prevent future disagreements about remembered interpretations.

3. If there are disagreements about unpublished data, a replication that is agreed to by both participants should be included in the initial study.

4. Accept in advance that the initial study will be inconclusive. Allow each side to propose an additional experiment to exploit the fount of hindsight wisdom that commonly becomes available when disliked results are obtained. Additional studies should be planned jointly, with the arbiter resolving disagreements as they occur.

5. Agree in advance to produce an article with all participants as authors. The arbiter can take responsibility for several parts of the article: an introduction to the debate, the report of experimental results, and a statement of agreed-upon conclusions. If significant disagree- ments remain, the participants should write individual discussions. The length of these discussions should be determined in advance and monitored by the arbiter. An author who has more to say than the arbiter allows should indicate this fact in a footnote and provide readers with a way to obtain the added material.

6. The data should be under the control of the arbiter, who should be free to publish with only one of the original participants if the other refuses to cooperate. Naturally, the circumstances of such an event should be part of the report.

7. All experimentation and writing should be done quickly, within deadlines agreed to in advance. Delay is likely to breed discord. 8. The arbiter should have the casting vote in selecting a venue for publication, and editors should be informed that requests for major revisions are likely to create impossible problems for the participants in the exercise.

Sentences that blew my mind

From Tooby and Cosmides’ chapter and coalitions and the psychology of morality. Does this sound like politics?

The winning outcome in social negotiation is to get everyone to adopt your position as their own, so that they conform to it, effectively enforce it, and carry the costs of enforcement. Mental coordination is defeated to the extent that it is publicly recognized that there are differences of position. This is usually recast not as moral relativism, but as individual mistakes in perceiving what the moral position “really” or “truly” is. It is almost definitional of morality that people intuitively represent morality to be intrinsically good, and support its being seen as real and objective. We argue that this is an evolved circuit.

Good intentions…

are not enough. Felix Salmon has more on why the outpouring of charity that occurs after major disasters does less good than it could. Mailing socks to a country with a per capita GDP near $40,000 is to please (or worse, sate) our moral intuitions, but is not a very efficient channel for generosity.

Makes for good marketing, though. Red Cross actually allows you to earmark your donation for the Japan disaster. Although in fairness, they do have a disclaimer that says “when donations exceed American Red Cross expenses for a specific disaster, contributions are used to prepare for and serve victims of other disasters.” This might be coded language for, “If you are donating on a whim because this stuff has been in the news lately, we’ll oblige you, but we reserve the right to shift this money to where it might actually do some good.”

Everything I read about Doctors Without Borders is that it does aid the right way.

The Science Factory

In college, a friend of mine who studied philosophy used to joke that he wanted to work at the Philosophy Factory after he graduated. Maybe the idea is not as silly as it seems. I’m currently soliciting venture capital for a Science Factory. Consider some of the benefits.

In academia, social scientists are sort of like traveling salespeople. They are largely autonomous, each working on their own projects, sometimes with only passing notice of what their colleagues are up to.

At the Science Factory, everyone works as a team focused on one goal: the production of knowledge.

In academia, an individual scholar (or perhaps a small team) prosecutes a task from start to finish. She seizes an idea, secures the grant, designs the study, collects the data, conducts the analysis, writes the results, and presents everything to an audience. Oh, and she teaches.

At the Science Factory, we benefit from the division of labor. Up in the Ideas Department (which is on the top floor; it’s important) whole teams are dedicated to synthesizing vast literatures and noting inconsistencies — but that’s all. The most compelling ones, after some culling, are sent over to the Methodology Department, which generates a set of pluralistic approaches to get at each of them. Then over to the grant writing department, and so on. The people who do each of the tasks become very good at them, and they end up saving a lot of wasted transition time.

In academia, scholars arriving at different conclusions get attached to their ideas and defend them to the death. Progress becomes mired in deep personal animosities that endure for years and do the community as a whole little good.

At the Science Factory, when two or more of our scholars find it impossible to settle on something, we stick them in a room together and don’t let them out until they have jointly agreed on a test that, ex ante, will discriminate between their two perspectives. We then run these studies and let the chips fall where they may. Both scholars become authors on the ensuing report, with a hearty thanks and slap on the back for pushing the ball forward.

In academia, science is an art, and an ad hoc one at that. Nobody can quite say exactly where they get their ideas or what makes some and not others successful. Moreover, this intangible trait, more than breadth of individual knowledge or methodological know-how, is often what makes the difference between a successful career and a less successful one.

At the Science Factory, science is 10% art and 90% process. The set epistemological rationales for conducting a study is small and well-defined: an inconsistency between the predictions of existing theories, a new area in need of descriptive attention, a puzzling observation in need of explanation… and that might be it. We weight these by their normative importance* and feasibility, and the agenda thus becomes deterministic. Where academia is ad hoc, we are routinized.

In academia, individuals market themselves. Sometimes they market the quality of their product — the thought and rigor that went into producing it. Unfortunately, because some of the products’ buyers know what they want to hear, the conclusions and implications of research can also become pertinent criteria.

At the Science Factory, we market our brand, and we want our brand to stand for good, systematic research, not for advancing a pre-formed agenda. (Admittedly, such places exist.) Choosing our projects in a systematic way makes us relatively agnostic to outcomes. The failing of one employee would hurt us all, which is why we work to instill a healthy ethos throughout the operation. Truth is our passion, and transparency our mantra.

A pipe dream? A solution to problems that don’t exist (or that only exist in the social, as opposed to the hard sciences)? Something we already see in the form of think tanks? Think about it. And while you think about it, polish the ol’ CV, because we’ll be accepting applications soon.

*We often outsource this task to the Philosophy Factory across the street.

Short term vs. long term

I can get on board with Norm’s thoughts on the current political posturing over what, in the grand scheme of things, are trivial cuts to discretionary spending:

The debate we should be having, over the larger long-term debt problem and over how to make smart savings that preserve both a safety net and critical investments in the future, is both obscured by the current one and made more difficult to have if we end up with vicious disputes and potential shutdowns draining energy and the tiny amount of trust and goodwill that exists in the larger political process. Not to mention that the focus on cuts this year, when there is a real danger of weakening further a job-depleted struggling economy, could add immensely to our debt burden by pushing us back into recession.

Against Christmas (sort of)

A friend of mine links to this article — a bit out of season — and says, “I know you’re mostly interested in seeing deadweight losses in the economy disappear, as we are all made better off then. Better start campaigning against Christmas.”

I think he overestimates how averse I am to such a campaign. I’ll go on record right now. Here it is: I’m against (the wasteful shallow materialistic overritualized overcommericalized overdramatized overhyped side of) Christmas!

I’ll put it another way. If the question is whether I would prefer to live in a world where my second Aunt Bernie*, whom I see precisely once a year, didn’t feel obliged to spend $20 on tube socks that I value at -$3 (negative because they take up space) and where I didn’t feel obliged to spend $20 on salad shooter accessories that she values at -$5 (because I got the cheap ones and they break), my answer is yes. I think that just might lessen the stress and help us focus on the part of Christmas that really matters.

But I don’t want to be a total Scrooge and we’re not going to get rid of gift giving any time soon, so let me take the economic framework my friend points out and try to pull a helpful thread out of it. Given that most gifts represent a deadweight loss (because if the recipient valued it at more than the price, he/she would already have bought it), what makes for a good, efficient gift? Three thoughts:

1. Things that a person can’t buy because they are not for sale — the sentimental, nostalgic, hand-made.
2. Things that can be bought, but the recipient doesn’t know exist. (E.g. a new CD that he/she hasn’t heard of.)
3. Things whose value the recipient underestimates. (E.g., I could afford a crock pot, but never would think to buy one. Now that my mom gave me one, I use it all the time. and, if it disappeared tomorrow, I’d probably buy another one.)

One quip with my friend’s statement. Eliminating the deadweight loss doesn’t make *everyone* better off. In this case, it would make the peddlers of tube socks, salad shooter accessories, and other crap we give each other around the holidays worse off.

More recent coverage of the deadweight idea is here.

*A stylized example, not a real person or pseudonym for a real person. I love my family and they give me totally sweet stuff.