I am frequently asked by journalists to comment on unpublished studies that make one odd claim or another. When I was younger and more patient, I used to dig up the press release about the study and try to determine from the limited bit of information available whether there was any basis at all to the study’s conclusions. But I am older and less patient now with colleagues who blast unreviewed findings into the media. The most recent example was the press released “finding” that oreos are as addictive as cocaine, about which I cut to the chase with journalists:
The cornerstone of scientific quality is peer-review,” Humphreys said in an email to HuffPost. “This study hasn’t been published in a peer-reviewed journal. It has not even been presented at a conference. All I know is that they put a press release making a series of claims based on a study, the details of which they have not shared with their colleagues. On principle, that makes me doubt their conclusions.
The Oreos press release, which was based on an undergraduate research project at Connecticut College, is part of a broader trend of sensationalized non-peer reviewed scientific claims. I decry this trend and assess the damage today at Stanford Medical School’s SCOPE Blog.
If you were an amateur astronomer and you spotted a new asteroid on track to hit the Earth in a fortnight, you should get the news out and have it reviewed later. Barring such extreme scenarios, you are right.
Some university and corporate press officers do still take the trouble to craft enlightening and accurate material for the reasonably educated public. An example I came across recently. It’s an important part of the public accountability of science to do this. Perhaps the press release should, in the academic world, be co-authored by the researcher and the press officer, and treated as work product for the careers of both.
Even in your asteroid-spotting hypothetical, “getting the news out” would be ineffective: your diligent amateur would be just another nutcase with an internet connection. They’d have to get their data out, so that it might be confirmed, in order to be taken seriously. Neil Degrasse Tyson might be granted credibility to simply announce “Death Rock From Space!” but his third cousin’s neighbor must show their work.
Also: when peer review is sought, it is standard practice for the story to be embargoed from the Press until the paper appears in an issue. Obviously, these entrepreneurs of scientific publicity are doing precisely the opposite. A better system might be one like my secondhand received impression of the ArXiv: manuscripts complete with data are made public, and are only then submitted to peer review and consideration for formal publication, and also mined for the Press.
I haven’t had an Oreo cookie in years and just you mentioning those little, round, dark, crunchy, cream filled, yummies gives me a jones. Now I ask you, if that ain’t addiction, what is?
But sumpthin’ tells me that if I went out and bought a pack and sunk my teeth in one, they wouldn’t be the same ’cause American corporations can’t keep themselves from F***ing up a good thing. Just look what they are doing to democracy. If they think they can make a dollar more on a truck load they will screw it up and lose millions in the process.
But as to scientific studies: Yeah your’e probably right.
You know what I love about studies? The ones that say: ‘A study of 163 student volunteers found that american men consume an average of six cases of beer a week.’ Actually they only mention the part about student volunteers in the foot notes. And those are the studies that get published.
I once saw a study that found that wearing a neck tie cut off blood to the brain and caused a marked decrease in IQ. Explains a lot, doesn’t it.
I have no disagreement with your larger point (in fact, I couldn’t agree more with the substance of your statement), but I have a nitpick. The following doesn’t apply to all disciplines: “his study hasn’t been published in a peer-reviewed journal. It has not even been presented at a conference.”
Specifically, in computer science, conferences are peer-reviewed and tier 1 conferences are generally the most prestigious publication venues, with journals often providing a “tidied up” version of existing research (as my advisor once said, with dramatization for effect: “Writing a [CS] journal paper is easy; you just staple a few conference papers together and edit the transitions”). While this model has its problems (e.g. that research gets pushed out for publication when the annual conference deadline rolls around rather than when it has fully matured), this is the way it is, for better or worse. And unfortunately, that does cause the occasional problem with tenure reviews and other forms of evaluation that happen outside of CS departments.
I.e.: Yes, peer review is a cornerstone of the scientific method (despite its occasional implementation problems), but the exact procedures can vary considerably by field.
@Katja: Some conferences in other fields peer-review things too, I go to a health services research meeting that rejects most of what is submitted. Some physics conferences also have high standards. That said, I would still want the paper to have been presented at the conference rather than have it released as something that will be presented in the future, peer-reviewed or not, because even at conferences with peer review it is not unusual as you know for someone in the audience to point out a problem and have the author respond “Good point, I need to fix that”.
If the “study†did not compare cocaine and cookies head-to-head, there had better be a very good reason for not doing so. They had the maze and the other equipment needed for making direct comparisons between different exposures.
The problem with conference posters that never get published is that you never have more than a bare bones description of the methods and you never know enough detail about a study to make sense of it. In addition, you need reviewers who have some content expertise who can tell you whether the outcome you are measuring (time spent in a maze) is even a good surrogate for the outcome (craving for cocaine leading to compulsive and criminal behavior) that you need to know about.
We have a clash of cultures between two ways in which evidence is used to find truth. The advocacy approach involves someone arguing one side of a case and an opposing side arguing another side; there are rules of evidence which often have nothing to do with the truth of a proposition (such as admissibility in court). It is the job of one team to argue one side, and of another side to argue the other side. Each side has a duty to argue its own side regardless of what other evidence is brought to light. The scientific approach makes it the job of every participant to look for evidence on both sides and to apply rules of evidence (how variables are measured and how they are analyzed) which are principally concerned with protecting truth. Each side has a duty to switch sides if enough evidence is brought to light on one side or the other.
The advocacy approach is vigorous and thriving in our society. The scientific side is, by contrast, precarious and in need of strengthening. It needs all the support it can get. Bad science is one more threat to its well-being which it does not need. This stuff is worth getting bothered by.
The advocacy approach is vigorous and thriving in our society. The scientific side is, by contrast, precarious and in need of strengthening. It needs all the support it can get. Bad science is one more threat to its well-being which it does not need. This stuff is worth getting bothered by.
@Ed Whitney: Very well said. I am particularly worried about advocacy arguers abusing the scientific mantle, e.g., gin up some fake organization, bribe a third-rate scientist, pay off a beltway bandit to release “new scientific findings” that further an advocacy agenda. There are organizations that do this now, styling themselves as researchers but working for an advocacy (usually but not always, corporate) interest.
How do we create incentives both for scientists and the press to not engage in this kind of behavior?
How do we create incentives? Well, back in the Cold War era, we had a Sputnik moment in which our global strategic rival gave us a good swift kick in the pants and did something we had not been able to do involving rocket science. Science had a surge of respect for a while. We were motivated to learn our times tables so that we could beat the Russians. Al Qaeda is not likely suddenly to blindside us in any great scientific-technical achievements, so we can rule that out as an incentive for taking science and its methods seriously.
Sputnik was a motivator because it was seen as a threat in a global power struggle, not because it was an advance in finding out true things about the universe. I suspect that the most effective incentive will have to do with convincing enough people that our world hegemony requires us to pursue it with intellectual rigor. Something highly dramatic will do the job most efficiently. Maybe someone can think of something.
I agree, Ed Whitney.
I thought of something……let’s look at what happened to WTC7 again. I hear a bunch of architects and engineers smell a rat.
Lets elevate science to its rightful place, once and for all.
Ed Whitney wrote: Maybe someone can think of something.
This reminds me of what Aldous Huxley put in one of his novels…
Most of the consolations of philosophy can be summarized in these six words: You kinda get used to it.
And then there’s the behaviour of addicts… So far as I know, people who run out of Oreos late in the evening don’t go out and rob gas stations at gunpoint to get the money to buy another package. So there may be a strength-of-craving measurement somewhere.
Deviant behavior, which figures into some definitions of “addiction.” comes from craving for a substance which cannot be purchased legally. If Oreos without a prescription were banned tomorrow, there might be a few people who would go doctor-shopping to find physicians who would prescribe Oreos; others might steal things in order to purchase Oreos from a black market. If cigarettes were strictly banned, there would be an army of addicts who would seek them through similar deviant behaviors.
Back in the late 1980s, “A Prairie Home Companion” had a skit involving the “Diet Squad,” which patrols the streets of Chicago in search of people suspected of selling and distributing things like onion rings; one of the people they stop and frisk said that she had less than an ounce and for her own personal use. Other people were awaiting a shipment of Chocolate from Colombia. IIRC Studs Terkel may have been in the skit. Maybe it is online somewhere.
“Deviant behavior, which figures into some definitions of “addiction.†comes from craving for a substance which cannot be purchased legally.”
Any expat or immigrant will be able to name *scores* of foods he eats regularly while in one country but stops eating in the other because of lack of availability. Or even just consider how you behave re some food you enjoy in excess, but is only available seasonally. Nor does one encounter very many people who are incapable of thriving in the absence of favorite foods to the extent of needing withdrawal/deaddiction therapy. I don’t think GP’s claim about strength-of-craving can be brushed away *this* easily.
Sorry, I meant to quote the sentence after the one I copied. As in “If Oreos without a prescription were banned tomorrow, there might be a few people who would go doctor-shopping to find physicians who would prescribe Oreos; others might steal things in order to purchase Oreos from a black market.”