September 24, 2003
Michael Maltz on John Lott
Michael Maltz of the University of Illinois is a statistician with an impressive record as a student of crime and crime control. In an email to me, reprinted in full below with his permission, he offers some reflections on the John Lott affair. Right now, Lott seems to have few defenders left, which isn't surprising. But he does still have his job at the American Enterprise Institute, and apparently is going on a speaking tour financed by the Federalist Society [*]. Friends of those organizations, and of intellectual integrity, should consider speaking out.
Here's Michael Maltz:
To me, the measure of a person's integrity is not whether the person makes mistakes (and I've made some beauts), but how s/he deals with the mistakes s/he made. Based on this standard, I have lost any respect that I originally had for John Lott.
It seems that most of Lott's critics and supporters forgot about what I feel is the most damaging lie he told while hiding behind the skirts of his fictitious Internet persona Mary Rosh: s/he described himself as "a chaired professor" at the Wharton School at the University of Pennsylvania. Had s/he forgotten that he was never even awarded as much as a stool? One can be disgusted by his unfairly lashing out at his critics while in drag, calling them liars who hide behind fictitious personas (I wonder where he got that idea from?), but this lie within a lie takes the cake. How can any of his supporters defend this?
[An aside: I usually don't like to use the "s/he" construction, but in this case it fits!]
He also misrepresented himself in front of the Nebraska Legislature, calling himself a professor when he was the John M. Olin Fellow in Law and Economics; see
his testimony here.
But what about the substance of his analysis? The data set he analyzed, county-level crime data, is really crappy. If there ever is a reason to use the expression Garbage In, Garbage Out, this is it; there's a hell of a lot of missing data in the county-level file. That's the reason that the FBI doesn't publish county-level crime data: the percent of the population not included in the county crime reports is huge. And what he came up with is, naturally, garbage out: no matter how sophisticated the techniques used by Lott (and, I might add, his critics), no matter how you slice it -- it's still baloney.
Well over a year ago I sent a data file showing the extent of this missing data to Lott, and he countered by saying that I didn't take into account that most of the counties with high missing data have low populations. He was right: but taking population into account just reduced the number of problematic states from 21 to 15 (just to use some benchmark, I called a state problematic if more than 10 percent of its observations -- county-years -- had more than 30 percent missing data; weighting by population, 21 drops to 15).
Lott then reanalyzed the data after eliminating the 16 worst states from his analysis. He doesn't explain why he dropped 16 states (why not 15, or 17? forget sensitivity analysis!), nor does he state why he dropped whole states instead of problematic counties, nor does he make any provision for the remaining errors (which are still plentiful) in the remaining states' counties.
How did he handle the errors in the remaining states? By ignoring them. At the very least, these errors should increase the uncertainty (standard error) attached to the observations. And the errors are not normally distributed, either. These are errors of omission and are always negative, so they can't be handled by the usual statistical techniques.
On top of this, he lied about recoding and remodeling his data, and tried to slither away from other charges as well. It's no wonder that few believe that he actually completed the survey that supposedly got lost.
