In today’s WSJ, Bill Gates writes about the power of data collection and analysis. Â He writes; “In the past year, I have been struck by how important measurement is to improving the human condition. You can achieve incredible progress if you set a clear goal and find a measure that will drive progress toward that goal …”
I have been told that before Gates dropped out of Harvard that he audited first year Ph.D. econ classes there. Â If he takes his goal seriously, he will need to grapple with some frontier research questions. Â All micro econ students are taught the production function. In the typical boring example, a firm hires labor and capital to produce pizza. Â The professor stands at the board and writes; pizza = f(L,K) and talks about the demand for inputs where f(L,K) is the production function. Â What makes this boring is that the firm knows f(L,K) and inputs are homogeneous. Â All workers (L) and all robots (K) are identical.
But, diversity and uncertainty are the spice of life and of modern economics! Â For a taste of this, read Sherwin Rosen’s AEA Presidential Address. Â What Gates’ article is really about is how from observing some noisy measures of output, and inputs do we infer what the production function f() is? Â Gates doesn’t care about “pizza production”. Â He cares about healthy kid formation and human capital formation but on some level it is the same question. Â Consider the following example:
Suppose that we want to rank doctors with respect to their “value added” of saving patients’ lives. Â The research nerd observes whether a given patient survives, and observes some coarse observables such as age, zip code, ethnicity. Â The researcher also observes which doctor was assigned to the patient. Â Suppose the researcher assumes that doctors are randomly assigned to patients while the truth is that the best doctors are assigned to the sickest patients. Â Note the asymmetry of information here. Â The hospital recognizes the diversity of patient types and doctor types but the naive statistician does not. Â Once the data nerd crunches the data, he will falsely conclude that the best doctors are the worst doctors because on the performance criteria (dead patients), they will have a high share. Â To explicitly address this nasty self selection on unobserved attributes challenge requires an economic model of how doctors and patients differ and how doctors are assigned to patients. Â What is the data generating process? Â Â If this interests you, you should follow the work of Jim Heckman. Â You can see that he is well cited.
This is a crucial insight as it relates to value added evaluations of teachers, which not surprisingly has been a focus of the Gates Foundation
What Curious said!
And to reinforce it, the underlying principle here is that the world is vastly complex, so “post hoc ergo propter hoc” is usually an oversimplification.
Like this? http://flowingdata.com/2013/01/25/internet-explorer-causation/
Well, I stopped using Internet Explorer because it was making me homicidal.
All MicroSoft products make me homicidal. Or suicidal.
Bill Gates & Public Education. Where to begin? I can’t say what his motives are but I can say what his results are, except this is a family blog.
For details, see “The Billionaires Boys’ Club” chapter in Diane Ravitch’s last book, “The Life and Death of the American School,” or go to her blog and click on the “Bill Gates” tag: http://dianeravitch.net/
Crucial insight into the deficits of the education “reform” movement.
What I don’t get, is why we’re still wasting so much time “looking at the data” in trying to improve test scores. We’ve lost the forest for the trees. Test scores correlate most strongly with SES, which itself is hugely multivaried but at its core can almost be defined by an inability to leverage societal capital. What is education but an ability to leverage that capital into human development? When looked at it this way, teachers are such a tiny part of the problem so as to be almost insignificant in comparison to larger factors in child development.
In any other area of life we recognize the structural problems, i.e. the quality of input versus output. Doctors with especially sick patients have a different job. Soldiers in battle have a different job than soldiers in peacetime. Yet with education we pretend that kids are all the same - disrespectfully minimizing the very real disadvantages they face - and put almost 100% of our focus (accountability, standards, tenure, unions, etc.) on the teachers.
It’s insane. The common response is this is an excuse for bad teaching. But the original problem has never been addressed. Would we say it was “making excuses” to have different expectations for the mission of doctors and soldiers depending on where they work?
Value-added has a lot of problems, but maybe chief among them is the way in which it bends over backwards to try and measure performance in what would be considered a totally understaffed, under-resourced ICU or a hellishly SNAFU warzone. Radically lower class sizes, quadruple social workers, institute parent accountability and support programs, etc. THEN start talking about which teachers are better. In the meantime stop looking for Rambo Escalante to save the nation’s children on the cheap (while continuing to rely on a vast underclass living in crushing poverty).
This comment says it all.
I have my doubts that the real goal is to save the nation’s children - whether or not on the cheap - or even to educate them at all. After all do we really have an economy that can support a nation where everyone is a college grad but there are no takers for all the jobs that instead require a skill a trade or simply an unpleasant task (e.g. garbage hauler). Back before college for all became the mantra of the school reform movement, we actually had various shop, typing, sewing, and other non-college prep classes available in high school. No more - at least in most comprehensive high schools in LAUSD. instead we have a system that requires passing Algebra to get a high school diploma, but no ability to actually earn a living with that education, and no college level education available freely available without financial sacrifice or maybe for-profit trade schools. I think the private sector is hugely invested in the perceived “failure” of schools and that the goal of school reform is anything but saving children.
You are also correct and your comment says it all too. The goal is indeed to make it look like all public schools are failing, the better to privatize as much of the education sector as possible. Big bucks to be had there, and this effort is well underway. The fact that there are “all kinds of minds” and that a good number of children would benefit from vocational approaches that are no longer offered in many places is, as they say, a feature, not a bug.
Eli is correct when he takes the edu-deformers at their word that helping children is the goal and then lays out what real reform would look like. The difference between his version and their version is huge, and stark, and helps to show how dishonest Bill Gates and his ilk are.
Just two different sides of the same coin.
I agree with much of this critique, and I would add in that I think there’s a huge dose of sexism inherent in the ed reform movement. When I’m feeling particularly snippy, I like to make fun of the names, like the Nubile Teacher Project. Which is horribly unfair of me, no doubt.
Otoh … I also wonder if behind all this isn’t a certain amount of guilt. I wonder if certain billionaires aren’t a tiny bit aware that their rewards in this life are out of balance with their actual supposedly greater virtue or value. And one way to try to alleviate guilt would be to pretend to oneself that we actually live in a fair society. And if you’re a numbers guy, then you’d naturally be more sympathetic to certain approaches more than others. Bingo, an obsession with test scores.
Plus, once you’re a billionaire, probably no one ever tells you the truth anymore.
To explicitly address this nasty self selection on unobserved attributes challenge requires an economic model…
No it doesn’t. Just what do you think economics is, anyway?
Nice talk by Rosen. But he seems to be unable to escape an obsession with exchange. Exemplifies, in a way, the confusion of measurement with understanding that plagues Gates endeavours as well.
Goodhart’s law: if you manage by the numbers, your underlings will assure that you get the numbers you want.
It’s funny. We all laugh at management-by-metric when we are talking about the mythical Stalinist show factory, which made all shoes the same size to raise its productivity numbers, and then decided that it could raise productivity even more if it specialized in left shoes. But all of a sudden, it becomes Mister Science when propounded by some square-jawed capitalist.
Can’t we just call this Rhee’s Law: when you set goals with a very clear measurement metric, you will indeed find rapid improvements in the metric. By whatever means necessary.