Mark Kleiman and I have written extensively about intervention programs for substance-involved criminal offenders which combine regular drug/alcohol testing with swift, certain and mild consequences for use and non-use (see for example here and here).
One of the prototypes is 24/7 Sobriety, a South Dakota program that sentences repeat drink drivers to twice a day breathalyzation. The chart below uses the most recent National Highway Traffic Safety Administration data to show how the safety of the state’s roads has improved since the program began.
I used to display these sort of data in the conventional metric of alcohol-involved auto-fatalities per million or hundred million vehicle miles traveled. That didn’t work well because neither policymakers nor the general public had a sense of what it really meant. Few people have a firm concept of how much driving a million vehicle miles is, or how many million miles Americans collectively drive per year.
The below chart adopts a different approach, namely expressing how dangerous it is to drive in South Dakota relative to the US as a whole. In the two years prior to the founding of 24/7, the risk of dying in an alcohol-involved auto accident per mile driven was almost twice as high in South Dakota as in the US as a whole. In the next two years, during which 24/7 began steadily expanding across the state, the risk dropped to about 1.7 times as great as the US as whole. Since the program went statewide in 2007, South Dakota is approaching being as safe as the national average.
Beyond ease of comprehension, another advantage of this way of breaking down the data is that it adjusts for the fact that the US as a whole has seen a drop in drink driving fatalities. Because the ratio between the state and national risk displayed in the chart is dropping, we can conclude that the welcome change in South Dakota is beating the secular national trend substantially rather than just being a local case of a broader trend.
Population-level relationships like this may not be replicated in individual data. Dr. Beau Kilmer of RAND Corporation is now doing more rigorous work on connecting individuals’ outcomes in 24/7 to traffic safety and crime impacts, and he will have a study of those issues out in a journal next week.
Note also that 24/7 sobriety is not the only effort South Dakota mounted against drink driving, so it could be that other policies (e.g. sobriety checkpoints) are causing the change evident in the chart. However, Montana, a similar nearby state with a serious drink driving problem tried all those policies as well and stayed extremely unsafe until 2010. In that year they began their own 24/7 sobriety program, and experienced a 40% decrease in the rate of alcohol-involved auto fatalities per mile driven. Most encouraging.
James Wimberley says
You nicely anticipate the objection one might call “post hic, propter hic”.
Other suggestions for intuitively accessible metrics:
- absolute annual numbers of deaths caused and maybe saved in the state
- ten-year normalised risks per driver.
The first is the more immediate, but doesn’t compare across states with different demographics. The second does, because it adjusts for the fact that Dakotans will drive many mote miles per year than Rhode Islanders.
Dennis says
James,
The annual counts are of some interest, but not of much use. The number of deaths due to cause X is highly correlated with the number of persons at risk. So, if the at-risk population is growing at a rate faster than rate of decrease in the risk it will appear that the program isn’t working because the number keeps going up.
More insidious is the fact that people tend to think they understand simple things like totals, and won’t (or can’t) work back to see why the counts are increasing. Slimy politicians (especially those of the Law’n'order ilk) tend to use this fallacious argument when they want to show that their opponent is a Bad Person because the murder counts in her jurisdiction are 2x what they were ten years ago. Never mind the fact that the population has tripled over that period and the murder rate has gone down.
The ratio of the regional normalized risk to the national normalized risk is not without its problems, though. Simple ratio estimators are biased in the statistical sense, tending to somewhat over estimate the true ratio. In this case, that is probably okay — if it looks like you’re decreasing you really are decreasing somewhat faster than it initially appears.
James Wimberley says
Is the population of North Dakota (or for that matter its driving habits) changing fast enough for a simplé annual death total to be seriously misleading, over the timescale of a few years? This wouldn’t do as a test of the modest preventive virtues of aspirin, but it does serve here to show Keith’s large and rapid effects.
Dennis says
In the case of either of the Dakotas, the amount of change over a couple of years is almost certainly small enough to be negligible. My objection is that misapplying statistics in this fashion encourages the same misapplication in settings where it matters.
Keith Humphreys says
FWIW, the 2003 number was 90 alcohol involved deaths and the 2010 number was 37.
Dennis says
@ Keith:
Wow. Just wow.
Would you care to come to New Mexico and talk to our Legislature?