Evidence-based catfighting

Want a change from watching the turds circle the drain in the Kavanaugh confirmation circus? Let me bring you a nasty academic spat between high-minded medical researchers. This is how learned gentlemen stab each other in the back! With a couple of serious morals. Everybody named below is a highly credentialed professional; I leave the titles out to avoid repetition.

The milieu is the Cochrane collaboration. Inspired by and named after the  epidemiologist Archibald Cochrane  (d. 1988), the Cochrane people promote evidence-based medicine through meta-analyses of randomised clinical research trials using methods as rigorous and objective as they can make them. (Our own Keith Humphries has been a Cochrane reviewer.) [Update] The very solid proposition is that if you can analyse correctly a handful of properly conducted trials, you are in effect adding the sample sizes, so you can draw much more statistically reliable conclusions than by cherry-picking one. There is of course a lot of art here behind “correctly”, “properly”, and “in effect’. [/update] They are not the only researchers carrying out meta-analyses, but a Cochrane review is widely regarded as the gold standard. Depressingly often, the answer is “we don’t know”.

One recent Cochrane review (lead author Marc Arbyn) was on vaccines against human papilloma virus (HPV), which causes much cervical cancer among women and lesser numbers of anal and penile cancers in men. This is not a trivial health issue. Fortunately there are vaccines sold by Merck (Gardasil) and GSK (Cervarix). Do these work? Short answer: yes. Are they dangerous? Short answer: no. (Please DO NOT quote me, read and cite the report, they do provide a summary for dummies.)

So far so routine. But then an article was published in the journal BMJ - Evidence-Based Medicine by Lars Jørgensen, Peter Gøtzsche, and Tom Jefferson, alleging that the vaccine review was sloppy on several counts and hinting that it was influenced by pressures from the Big Pharma vaccine vendors. (Note that while they argue that the side-effects are greater than the review says, the critique does not recommend stopping or curtailing vaccination programmes.) This naturally provoked a rebuttal from the Cochrane management (David Tovey and Karla Soares-Weiser), saying the criticism is wrong on all counts.

It did not stop there. Gøtzsche is, or rather was, a member of the Cochrane board, indeed a founder member of the organisation. He could presumably have raised his concerns there first rather than publicly. After a presumably furious board meeting, Gøtzsche was expelled and four other board members quit. The great collaboration is now in existential danger. Will donors, including the Gates Foundation, keep the funding flowing? Will Gøtzsche set up a breakaway fitzCochrane, applying his own higher standards? Will anti-vaxxers and misogynists exploit the row to attack the vaccination campaign? Only 27 % of American men under 26 are vaccinated.

It’s important that the crisis be resolved quickly and the collaboration continue. There’s not much outsiders can do to help this in the short term, and I am quite unqualified to take sides. I have though one reflection and one suggestion for the future.

The reflection (scarcely original) is about irreducible human nature. Cochrane reviewers are scientists utterly committed to objectivity in the scientific process. Rigorous and uniform statistical methods, they say, will surely bring us closer to the truth. Closer, yes. All the way, no. Science cannot be reduced to an algorithm. There are judgements to be made all along the process: in this case, what studies make the cut methodologically, and what weight to give to observational evidence within or outside a clinical trial. The general public and policymakers have at some point to trust the experts: and it’s a sight better to trust any of the players in this incident than amateurs. But what if the experts fall out, and matters of judgement turn into disputes and name-calling and resignations? It’s possible that the Cochrane commitment to extreme rigour and objectivity has made it harder for the participants in the conflict to recognise the human limits to this commitment, and their own essential frailty.

That’s my armchair moralising for today. Let me try to be constructive, even if constructively wrong. I was struck by this passage in Tovey’s rebuttal (my italics):

In recent years, evidence synthesis researchers in Cochrane and elsewhere have recognized that reliance on the published reports in scientific journals may introduce bias due to incomplete and selective reporting. In addition, the generally poor reporting of harms in reports from randomized controlled trials has led to the reporting of harms in many systematic reviews being sub-optimal. This has led to an increased interest in searching for and identifying studies, reports and data from different and more diverse sources, including clinical study reports and individual participant data from trials, data from trials registries, and non-randomized studies. This has consequences that reach well beyond Cochrane, as shown by a report by Page et al in 2016 comparing the quality of reporting in Cochrane and non-Cochrane systematic reviews. This study found that 62% of Cochrane Reviews searched trials registers, compared with 20% for non-Cochrane reviews. These additional or expanded searches may add value in selected circumstances, but they all also add substantially to the resources needed to complete the review and are a challenge to Cochrane’s traditional model of reliance on unfunded ‘volunteer’ authors, who have been the engine of the organization for 25 years.

And this in the Gøtzsche criticism:

More importantly, harms cannot be assessed reliably in published trial documents—especially in journal publications of industry-funded trials where even serious harms often are missing. One reason may be the space restrictions that most medical journals have. As an example, the journal publication for the PATRICIA trial is 14 pages long while its publicly available corresponding clinical study report is over 7000 pages long, although it is an interim report that has been shortened.

In other words: Cochrane does not have enough money to do the job as well as they would like. This is ridiculous. Merck’s sales of the Gardasil vaccine alone are $2.7 bn a year, mostly to national health care systems. Skimping on a few million to find out whether the vaccine works or is unsafe is crazy public policy. Going through the clinical study reports to find evidence of harms requires research assistants, plenty of them. Given the enormous commercial interests at stake, it also makes sense to insulate meta-analysts from any possible conflicts of interest. British High Court judges had their pay raised to £1,000 a year in 1645, for that reason, and are are still very well paid.

National health administrations and the European Medicines Agency could just throw money at Cochrane the way it is. But the breakdown suggests the institutional crisis is broader. Cochrane may be stuck in the difficult transition from an informal band of brothers to an efficient rule-governed bureaucracy, see Acts 5. I worked for an international organisation that failed in this. When I joined the Council of Europe in 1973, it didn’t have a filing system. When I left in 2005, it still did not have a filing system.

The radical solution is for Cochrane to become a formal international organisation. There are several precedents for this in other fields of science, and they look very positive.

CERN : Annual budget $1.3 bn. (Really.) Responsible for numerous discoveries in its field of particle physics, and the invention of the World-Wide Web (to solve their multimedia filing system problem). Built the $9bn LHC using untried technology and SFIK within budget.

ECMWF : European organisation for computer-based weather forecasting. Core budget only £78m, but there are large additional resources from contracts with the EU and others and sales of data. Predicted the correct landfall for Sandy, unlike US forecasters – and was listened to.

ESO : originally European Southern Observatory. Builds and runs large telescopes as joint facilities for European astronomers in Chile, where conditions are much better than anywhere in Europe. Annual budget €165m.

These three are full international organizations, with a statute laid down in a multilateral convention, a governing council of representatives of member states (I expect often scientifically qualified non-specialists like government scientific advisers), and a professional director appointed by the council to run the activities day to day. The directors are all real scientists in their fields – a crucial point. It is much easier to find managerial talent among a large group of scientists than for a generalist manager to learn enough science to be a credible arbiter between conflicting claimants on resources.

On a different and less transparent model, we have Eurofusion, a consortium of national research councils and universities that runs the JET lab in Oxfordshire and the European physics contribution to ITER. It’s also under the EURATOM umbrella, so it can be seen as an EU agency.

These examples are enough to show that the formal bureaucratic structure of an international organisation is compatible with high scientific standards, professional autonomy, and financial stability. They also show that European governments at least are prepared to entrust large sums of money to scientists who know what they are doing and offer value for public policy – sometimes value far less direct than in evidence-based medicine. However they do all work in fields with few commercial pressures, against which an intergovernmental structure only offers partial protection. The formula does look to me like a safer home for Cochrane than where it is today.

But they should keep the name rather than join the alphabet soup of instantly forgettable acronyms. Branding and name recognition matters, and there is every reason to keep honouring Archie Cochrane.