Metrics vs. metrics

Over at Ezra Klein’s blog, Senator Kent Conrad is quoted to say the following about job training:

There were over 40 different job training programs, with very little coordination between them, and different definitions of who was eligible. And there were almost no metrics on any of them.

This attracted one reader’s ire, who responds, in part,

I am a grant-writer and program manager at a nonprofit that receives many federal grants. WE LIVE AND DIE BY METRICS! Almost to the point, in my opinion, of spending too much money for compliance and performance-measurement systems, and not enough money on the actual training. In rough terms, we hire one full-time compliance and performance staff person for every 10 staff who actually work with people. All of the federal job training programs have performance metrics. In fact, many have started using the so-called Common Measures, a simplified, standardized system for measuring outcomes across different agencies…..

Different agencies use additional measures, and yes, those get complicated. And I’m not saying these systems are perfect. But the federal government has become increasingly better, ever since the passage of GPRA, at using metrics. It drives me a little nuts to spend so much time out here in the field collecting, analyzing, and reporting performance metrics, and then to have a politician claim that “there were almost no metrics on any of them.” No, simply not true. They’re just ignorant of the data that is available….

One last note, in case you really want to dive into this issue: In some programs, future funding gets tied to performance, creating an unhealthy obsession with the metrics. For example, programs start “creaming,” or accepting only clients that are somewhat more likely to succeed. … Grantees like my organization are trying to serve people who are most in need, and we’re caught between these daunting skill deficiencies in our clients and the rigorous performance standards we must meet in order to keep our jobs. For someone like Conrad to claim that there are no metrics — he misses the point entirely.

Anyone who has participated in program evaluation finds much to embrace in both perspectives. Indeed, there is really no contradiction between Conrad’s and the anonymous letter writer’s perspectives. They are using the word “metric” in two different ways.

The federal government and other funders indeed require an incredible profusion of program activity data. These data are useful to document that you’ve actually spent their money to deliver services, and to characterize the people you have served. Such performance data-metrics, if you will-are collected in nice binders and are placed on the shelf, where they generally reside, blissfully undisturbed. I’ve produced a few of these binders myself.

These binders don’t get much use because they can’t really tell policymakers such as Senator Conrad what’s actually working and how we should target (say) job training resources to do the most good. (Indeed there’s probably more good evaluation data regarding job training than there are in other important areas such as crime control.) Funders still spend impressive amounts of blood and treasure collecting these data, sometimes imposing perverse incentives or excessive burdens on service providers along the way.

It’s frustrating that these same funders rarely support or finance the kind of rigorous program evaluation that might actually answer these questions, and thereby do some good.

Postscript: Yeah, I just outed myself. I was initially reluctant to admit that I am a producer of said binders….