… it’s worth cheating for. Turns out that some of the “miracles” Michelle Rhee created in the DC schools were artifacts of test-tampering. This is no surprise. It’s also not, by itself, a reason to abandon the process of measurement. But it does mean that the higher the stakes on the process the more you have to invest in cheat-proofing it. It’s annoying that the educational reform movement, which puts so much stress on being able to fire bad teachers, puts so little effort into punishing cheaters. It appears that, in D.C. the Rhee administration preferred discretion to anything that might have rocked the boat of test-score improvements. That was certainly true of Rod Paige in Houston, who parlayed a faked miracle on dropout rates into a cabinet position.
Update It was, barely, possible that Rhee was culpably negligent in the cheating and the cover-up, but no worse. However, her slime-and-defend reaction to the exposure of the cheating eliminates that possibility. She was, and is, complicit in the cover-up, if not the cheating itself. This should be a complete disqualification for her ever having any active role in educational reform. (I say that as someone sympathetic to the goal of improving public schools, even if that requires breaking some eggs.)
Alas, it won’t be.
You are understating Rhee’s involvement. Normal practice would have been to investigate the flagged schools following the initial report. Instead, on Rhee’s watch, the report was immediately hushed up and things went back to business as usual. It is simply implausible that Rhee and her people would not have known about likely cheating (identified through massive erasures on test forms). The new administration is doing the right thing and has fired up an investigation. Of course, delaying the investigation results in information lost, so the chances of finding the culprits now are much lower than they would have been had Rhee allowed the investigation earlier.
Once the NYC test scores were adjusted on audit - all *that* great improvement vanished. After $25 billion, hundreds of “failing” schools closed, disrupting neighborhoods, sending teachers and families scurrying, currying anti-union sentiment - you could have done as well doing nothing. Is there a jurisdiction anywhere where validated test scores point to some intervention everyone else should be doing?
“some”? Anytime some metrics are being corrupted, I don’t think you give credence to any miricles
Um, it’s *Rod* Paige. Clarence Page is the Chicago-based pundit.
Thanks. Corrected.
Had the “privilege” of observing the pressure placed on teachers to cheat in a school where high stakes test results were the metric of continued operation. My idealistic daughter, there as a Teach for America corps member, blew the whistle on her school’s principal and was greeted with…silence and no action. (She had already developed extreme skepticism of TFA.)
Thinking about this more, the more I realize testing needs to be treated as a double-blind experiment rather than a high stakes gate that controls money.
Teachers should not know the answers to the test, and students should have randomized test-books with re-ordered answers.
There should be incentives to get accurate results rather than good results.
And so on…
Do the individual students have an incentive?
In other test-taking environments, there can be varying degrees of pressure on the student, taking the test, to perform, qua test-taker. With standardized tests, I wonder, why the student should care. Does this widen the measurement error across students?
So Bruce, in 11th grade, I decided to keep records of the stupid standardized testing answers because they used the same test every year.
Why? Because it’s boring taking the same tests again & again. In 12th grade, I ended up not using those same results because why bother.
(it’s not cheating to copy your own answers, which is how I rationalized it.) But incentives are everywhere.
Bruce, our high school - a continuation school with a highly dysfunctional and transient population, routinely blows off the tests. Heck, many of them go weeks - months even - without completing any assignments at all. We get new students monthly who have been expelled from prior schools, and who whose test scores who supposedly be reflective of learning/teaching.
Our “data guru”, when asked whether the tests reflected much of anything about student learning, much less our teaching, responded that we ought to be doing everything possible to incentivize test participation.
Which raises the question: how much are we measuring students scores and how much are we measuring our ability to get students to participate?
Current reform efforts are frighteningly misconceived.
The whole thing (at least in our school) is pathetic from top to bottom.
It’s discouraging, and hard to know what to do. Here in the DC area, we have had a long period of excitement about Rhee in the WaPo, and my friends from the District (I’m in the burbs) have been hopeful. The schools have been really dreadful in poorer areas, and it seemed they were getting better. Still, parents for whom those schools were local signed up in droves for the chance to send their kids private/parochial with vouchers, and there were snouchy editorials in the usual places (WSJ, Washington Examiner) when the Dems set out to kill the voucher program.
Current state of play in a lot of schools is: we are sending a lot of happy and eager 5-year-olds into the system, people of varying competence are getting middle-class wages to handle and control them for twelve years, and the kids are coming out semi-literate and unready to arrive a work on time and put in a day of productive activity at the end. It is clear that the results of the Catholic schools are better, with kids from the same neighborhoods, though you can’t know how much of that comes from having made parents who made the effort to get them out of the public schools in the first place. Klitgaard published some stuff suggesting that low-SES kids need highly controlling structure in their education more than do high-SES kids (long time ago, before he moved into studying public corruption) and the KIPP schools seem to be doing well with that - but again, how do you know, when they way you are measuring is with tests.
I’m kind of inclined to beef up voucher programs and charter schools. I have reasonable faith that parents want the best for their kids and will be able so see what options, if there are options, are doing best for them. And it seems like a huge tragedy that the schools which are succeeding are being closed by the dioceses because they haven’t got enough kids coming through the door with tuition, while we continue to pour huge amounts into maintaining the failure factories.
Goodhart’s Law! Though I think the phenomenon was first noticed by acute observers, like Janos Kornai, of Soviet central planning, which gave massive incentives to managers to fulfil plans - or, the simpler option, to create the illusion of doing so.
@ Dave Schutz:
Actually evaluating whether or not a school is adding value to a child’s education is tough, much the same way that evaluating whether or not a doctor’s practice is adding health value. I am more familiar with the medical side of things, but quite a few doctors who receive good patient satisfaction scores are winning those scores on the fact that they have a nice receptionist, a clean waiting room with a quiet alcove for the 2 year to have their tantrum without disturbing everyone else, phone calls either answered quickly or returned quickly and a more than three minute interaction with the patient. All of that stuff is “customer-service” orientated actions that do not indicate quality. Those actions have minimal correlation with actually following best practices and producing good outcomes.
Schools are the same way. Are the teachers accessible? Are phone calls returned? Is lunch good enough? Those are nice things to find in a school, but they do not correlate well with actually teaching kids. But those are the heuristics parents use to evaluate schools right now.
Dave wrote:
“Current state of play in a lot of schools is: we are sending a lot of happy and eager 5-year-olds into the system, people of varying competence are getting middle-class wages to handle and control them for twelve years, and the kids are coming out semi-literate and unready to arrive a work on time and put in a day of productive activity at the end. It is clear that the results of the Catholic schools are better, with kids from the same neighborhoods, though you can’t know how much of that comes from having made parents who made the effort to get them out of the public schools in the first place.”
It seems here that you’re underemphasizing the degree to which poor students are a difficult population to teach, for a variety of reasons. While its hard to know the degree to which selection plays a role in the success of Catholic schools with these kids, we know that it does play a role. I guarantee you that the most difficult children in any any class will be those with parents that are “not in the picture”. These are the kids that are frequently absent, don’t return homework, don’t respond to consequences, and - no surprise - have little respect for authority.
One of the things that gets glossed over in education debates (which are really about poor schools), is the meaning of SES. What this is often measured by is a simple formula: free/reduced lunches = low-SES. The problem with this formula is that while it is a pretty good at measuring financial capital, it is a poor measure of things like parent education, intact family, substance abuse, parenting skills, and a variety of other factors that are profoundly deterministic in a child’s development.
@dave schutz: KIPP is gleamingly funded but doesn’t seem to make a damn bit of difference compared to well-run, relatively broke “progressive” institutions like NYC’s Central Park East/Debra Meier model.
“When it comes to resources to open new schools, Central Park East is badly overmatched. According to its most recent tax forms, KIPP had a $1.7 million school expansion budget for New York in 2008. KIPP also has many well-paid executives working on new-school development, including David Levin, the KIPP New York superintendent, who makes $296,751 a year; eight other New York staff members earn $104,299 to $150,950.” (http://www.nytimes.com/2011/03/28/nyregion/28winerip.html?src=recg&pagewanted=print)
One thing is certain: charter school executives with over-amped greed glands are making a bundle.
Thanks, James, for the term and the link. I have written before that the State cannot pay for a good or service (e.g., education, health care) without a definition of that good or service. The definition then binds students, teachers, parents, and taxpayers.
A wise lady from the ETS once said to the assembled Campbell High School Math faculty: “We can’t measure what’s important so we measure what we can”.
Mark’s basic point applies as well to anything on which the State spends a lot of money. We measure the performance of politicians in votes. The larger the State’s share of the economy, the greater the incentive to commit election fraud.
James and Malcolm - when I was small, my grandmother subscribed to Reader’s Digest, and one of their anecdotes was of a Soviet knit gloves and hats factory. Well, they had a quota of 600,000 hats and pairs-of-gloves. So, they made 595,000 hats and 5,000 pairs-of-gloves. I hadn’t heard of it as Goodhart’s Law, though.
In The Gulag Archepelago, Solzhenitsyn wrote of “tutka” (sp=?. It’s been a while), where people would inflate figures for trees harvested (and get paid), then sawmills would inflate figures of board feet cut (and get paid), then finally, the underage would get scored as shrinkage (rot in storage, etc.).
China trawler catch numbers comes to mind as well.
The scope of Goodhart’a Law is not limited to the public sector. If managers are paid huge bonuses for raising the stock price, they will do so by methods that undermine the reasons for creating the incentive in the first place. Trust as social capital, anyone?
James,
100% agreement. I wish advocates for State-monopoly enterprise could see how aggregation of resources enhances the incentives for (electoral and academic) fraud and so undermines trust. Competition spurs the evolution of verification mechanisms. True, it does not always work: consider the signal that wide hips send about reproductive potential-”your mate won’t die in labor”-, and how this leads to fat storage around the hips, which looks like a wide pelvic girdle but which makes no difference to the diamater of the birth canal. It’s not just about government.
There’s a fly order - the Empidae. So, courtship - and trust and verification mechanisms. In primitive empids, the gent catches a tender little aphid, and presents it to his intended. And while she’s eating it, he mates with her. In somewhat more advanced empids, he wraps the aphid in silk, and it takes her longer to get it unwrapped so she can eat it, giving him more time for mating. In the fully advanced empids, he gift wraps some air in silk, and she works on unwrapping it, and he has meanwhile mated her without having taken all the time and effort necessary to get her an aphid. Nature red of tooth and claw.