The Reality-Based Community

 Lott, Donohue, and Levitt

If blogging is a form of journalism -- which, it seems to me, it more or less is -- then I'm clearly a "columnist" rather than a "reporter." But the latest episode of the soap opera called "John Lott" has roused me from my usual opinionated torpor to actually make some telephone calls and read some documents.

The latest round involves two different issues. First, Lott's new book brings up once again the charge that Steve Levitt of the University of Chicago, a member of the National Research Council panel on guns, is "rabidly anti-gun." There's the primary question of whether that's true, and the secondary question of who said so anonymously and was quoted in an NRO column by David Kopel and Glenn. Second, Lott has removed his name from a paper about to appear in the Stanford Law Review, part of his ongoing dispute with John Donohue and Ian Ayres about Lott's famous "more guns, less crime" hypothesis, and it is being suggested that by doing so he has in effect conceded that they're right and he's wrong.

[If you're coming at this fresh, I'd suggest starting with Tim Lambert's accounts here and here -- Tim has been harshly critical of Lott in the past -- and then taking a look at Glenn's post which links to most of the key documents. (Glenn has been among Lott's defenders, but trusts Ayres and Donohue.)]

I've talked at some length with Kopel and Lott and exchanged lengthy emails with Donohue. As a result, I'm coming in to the conversation late, and had it not been for insistent public demand (well, okay, one email from Glenn Reynolds) I would be inclined to hold my peace. But here's what I think I know:

The Stanford Law Review affair:

Ayres and Donohue have an article forthcoming in the Stanford Law Review reporting on some new work, using post-1993 data, whose findings are largely inconsistent with the "more guns, less crime" hypothesis. The editors invited Lott to submit a response, which he did along with two co-authors. The understanding was that Ayres and Donohue would be permitted a rejoinder, but not allowed to change their original paper in reaction to the response. Lott claims that the Ayres and Donohue went back on that understanding, and that the editor of the law review sided with them under pressure. (Donohue teaches at Stanford Law School.) Donohue and the editor deny this, and assert that the changes Ayres and Donohue wanted to make were part of the normal editing process. See the link below for a set of emails back and forth, provided by John Lott.

Email to MK from John Lott.htm

It's hard for an outsider to say whether Lott was or was not treated unfairly in this process. Still, it's clear that his withdrawal of his name was not a concession of error, and he continues to say that he stands by the assertions in the paper that no longer bears his name.

But the procedural question is less important here than the substantive one. It seems clear that Ayres and Donohue detected a significant coding error in the response, just as Donohue had earlier detected significant coding errors in other work by Lott and Mustard. Moreover, in addition to putting a hole in Lott's earlier work, Ayres and Donohue's analyses of subsequent data suggest that the "more guns, less crime" hypothesis is, at best, true in some places and not in others, and even then to only a slight extent. There is no justification for continuing to claim that Lott has proven that liberalizing concealed-carry reduces crime.

At the end of the day, though, it's pretty clear that if "shall-issue" increases gun violence at all, it doesn't do so by very much. To that limited extent, Lott was right and the gun controllers were wrong.

There remains the question of character. In this latest instance and the earlier instance involving the Lott and Mustard paper, Lott made significant errors in a direction favorable to a view he wanted to be the case, and responded to the discovery of those errors by continuing to insist that he'd been right all along. His attitude seems to be that of an advocate rather than that of a scientist.

As Ayres and Donohue note, coding errors are easy to make. That why it's so hard to take the "con" out of econometrics: there are always sample definition, coding, and model-specification decisions to be made, and more often than not they're big enough to matter. That's why judging econometric work, except for the occasional finding of an intra-ocular correlation (i.e., one that hits you right between the eyes), depends so much on knowledge of the character of the person doing it.

Five characteristics go into the making of a first-rate econometrician: meticulousness, to minimize the number of frank errors; impartiality in practice, so all those difficult calls get made even-handedly; technical competence to make the right calls on the tricky specification questions; ingenuity in finding and exploiting natural variations in the independent variables to explore their impact on the dependent variables; and appropriate modesty in making claims about the meaning of the results.

It's in this context that "Mary Rosh" and other apparently trivial matters loom so important; doing good econometrics is much more morally demanding than doing good history or law or policy analysis, because it's enormously easier to get away with sleaze.

The contrast between Lott's attitude toward his results and Donohue's attitude toward his own couldn't be sharper. Donohue says, roughly, "We found a trend toward more crime as a result of shall-issue; if we're right it's a small effect; the sensitivity of the results to the specifications means that the econometrics probably isn't getting at the real issues here; it could well be that shall-issue increases crime in some places and decreases it in others, but in neither case by a substantial amount."

[Editorial note: That leaves me favoring shall-issue, especially as part of a package including a crackdown on sales to ineligibles; if giving a bunch of people something they intensely desire does no demonstrable (net) harm, then why not?]

That brings us to the Levitt affair.

There's a long, tangled history here; start with this post and follow the links, and you'll get most of it; here's Glenn Reynolds's update. A quick summary of the relevant facts:

1. Kopel and Reynolds had a column in NRO about the NAS study panel on guns, which quoted an unnamed source as describing Steve Levitt as "rabidly anti-gun."

2. Levitt protested.

3. Reynolds semi-retracted, but the NRO piece had not been updated as of today, though Kopel submitted an update a couple of weeks ago.

4. Lott's new book says, as a matter of Lott's own knowledge, that Levitt is biased against guns, and cites the NRO piece as evidence that others think so too.

5. Levitt has charged that the source of the "rabidly anti-gun" quote was Lott himself.

6. Reynolds and Kopel have refused to confirm or deny that, as a matter of protecting journalistic sources.

7. Lott, in a telephone conversation with me, also refused to confirm or deny, but did assert again that he knows Levitt to be strongly anti-gun and thus considers him an inappropriate choice for the panel.

With no one talking, it's impossible to know what the truth is. But the fact pattern surely points squarely at Lott as the anonymous accuser:

First, Lott is clearly at risk of having the NAS panel trash his work, and Levitt is the panel member most competent to judge its technical merits and tell the rest of us how significant the coding errors were and whether Lott's conduct met professional standards.

Second, the Mary Rosh affair shows that Lott is not unwilling to hide behind anonymity to denigrate his critics. Thus it's not hard to imagine Lott deciding to "poison the well" by trying to discredit the panel in advance.

Third, Kopel told me that he wouldn't have published the assertion about Levitt's bias unless it came from someone in a position to know Levitt's views; Lott was for some time at the University of Chicago.

Fourth, under the circumstances it would have been well within the bounds of journalistic ethics for Reynolds and Kopel to have cleared Lott of the accusation, had someone else been their source. Surely, journalists can't let other people play "Twenty Questions" to identify their sources. But when a particular innocent person has been convincingly fingered as a source, and the universe of possible sources isn't so small that clearing one person clearly implicates another, a simple "No, he wasn't the one" is perfectly appropriate; Woodward and Bernstein did that after a book fingered someone as "Deep Throat." An abundance of ethical caution might lead a journalist to check back with the real source in that situation: "Lott has been accused of saying what you said, and will be damaged unless he is cleared of the imputation. May we do so?" It's hard to imagine why someone on the pro-gun side of the controversy, as the source presumably was, would deny such permission, or even why he wouldn't proactively call the authors and say, "Keep my anonymity but go ahead and clear Lott."

Fifth, there's absolutely no reason why, if Lott wasn't the source, he shouldn't simply say so. He surely has no obligation to protect Kopel's and Reynolds's sources, and a simple denial from him wouldn't leave anyone else obviously on the hook.

Therefore, I am convinced to a "clear and convincing evidence" standard, though not to a "beyond reasonable doubt" standard, that Lott was indeed the source of that quote. If so, his reference back to it in his book as confirmation of his opinion was a thoroughly reprehensible act of deception practiced on his readers. [That would be a problem for any reputable publisher; since the book in question was published by Regnery, that problem doesn't arise.]

Lott now has a very long string of serious ethical charges to deal with:

-- "Mary Rosh" (confirmed, but arguably not truly scholarly misconduct);

-- The Levitt quote (fairly solid, though not quite conclusive, evidence of what I would call a serious ethical lapse in the course of a scholarly dispute);

-- The possibly fictitious 1997 survey (an open question factually, but again with a fairly damning fact-pattern: Lott's change in story about where the number came from and the fairly far-fetched account of why no evidence exists that the survey was conducted are a bad start, and every month that passes without finding any of the workers on the purported survey, or any credible person who remembers being surveyed, makes the case look that much darker. Obviously, the fabrication of results, if that's what happened, would obviously constitute the most serious possible variety of misconduct);

-- The slippery use of what would have been, at best, a dubious point-estimate to summarize a very large error band, both in the original book and in many op-eds and interviews. That the data, even if they existed, couldn't support an assertion that 98% of defensive gun uses involve brandishing only would be obvious to any student in a first-year statistics course; it's inconceivable that a professional statistician could make this sort of mistake innocently. The defense that the number cited in the book was about a side-issue, and that the repetition of that number didn't take place in scholarly publications, strikes me as extremely weak; a scholar engaged in public controversy shouldn't deliberately misrepresent the state of his knowledge, and doing so is academic misconduct because it weakens the capacity of academics as a community to bring their knowledge to bear on matters of public concern;

-- The coding errors themselves (venial) and Lott's seeming unwillingness to concede their significance (possibly serious, depending on the econometric merits of the case);

My conclusions from all of this:

It's time -- past time, some of us would say -- for defenders of gun rights to stop citing Lott as an authority. It should be enough for them that the current state of scientific opinion does not support the assertion that "shall-issue" has big public safety costs.

The University of Chicago Press, which published More Guns, Less Crime, should start a formal inquiry about into the existence vel non of the 1997 survey and the extent to which the original results rested on the now-evident coding errors. If the survey took place, it ought to be possible to unearth evidence to that effect in the form of the students who purportedly did the work. If it did not, the press ought to say so. Since the book now has the status of a classic, withdrawing it from the backlist would be a disservice; but there's no reason not to include an appropriate erratum slip. If in fact the coding errors were important to the book's main finding, that too should be made clear by the publisher if the author refuses to do so. These are extraordinary steps for a publisher to take, but we seem to be facing an extraordinary situation.

-- If Lott were at a university, he would certainly be facing an inquiry into his professional ethics. The American Enterprise Institute needs to decide whether it is a scholarly institution or a propaganda mill, and act accordingly, and the rest of us need to adjust our attitude toward AEI activities and publications accordingly.

Update here: Donohue withdraws the suggestion that Lott's removal of his name from the paper represented a confession of error, but continues to assert that such a confession would have been in order.