Research Manipulation Part IX: The Replication Crisis, and Where Placebos Fit In

Research Manipulation Part IX: The Replication Crisis, and Where Placebos Fit In

(Placebos Part D)

By Malcolm Kendrick

Anyone interested in science will know that we are in the midst of a ‘replication crisis.’ In 2016 Nature published an article which began

“More than 70% of researchers have tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce their own experiments.” [1]

A more recent article, from 2024, in the Institute for Policy Research (IPR) journal stated:

“In its most basic sense, the replication crisis refers to a pattern of scientists being unable to obtain the same results previous investigators found. At its most expansive, the crisis threatens the scientific enterprise itself, leading to questions not just about research practices and methods, but the very reliability of scientific results. It is what IPR education researcher and statistician Larry Hedges has called “an existential crisis” for science.” [2]

An ‘existential’ crisis for science. This is no exaggeration. Nothing could be worse. If we cannot rely on the results of studies, or experiments, however they are done, then we are standing on quicksand.

Yes, there have always been mistakes made in the history of science. Some honest, others less so. But is seems that the direction of travel is ever downwards. With medical research leading the charge.

Where do placebos fit into this?

If you did find yourself concerned about the results of a clinical trial, you may want to try and reproduce it. However, in the world of medicine, and medical research, there are two giant obstacles. One is cost, the other is the ethics of placebo controlled trials.

Large scale medical trials carried out on drug efficacy and safety are often massive. They can involve tens of thousands of participants over several years. The costs are eye watering:

“The estimated median cost for a pivotal clinical trial came out to $48 million, with an IQR (Interquartile Range) of $20 million to $102 million. In the 2014 Tufts study, researchers estimated the average cost of a Phase III trial to be $255 million.” [3]

You want to prove a clinical trial was ‘non-replicable’ nonsense? All you need to do is scrape together a quarter of a billion dollars and off you go. Knock yourself out. I am just guessing here, but I sense you may struggle to gather the funding you need. The pharmaceutical company that did the original trial may not prove an eager supporter of your cause either.

You want to prove a clinical trial was ‘non-replicable’ nonsense? All you need to do is scrape together a quarter of a billion dollars and off you go.

However, even if you did succeed in your miraculous money gathering effort, you would never be allowed to replicate the trial. How so? Because any ethics committee you approached, to sign off your study, will refuse to do so…instantly.

They will do this for the simple reason that, if a medication has already been ‘proven’ to work, in a large-scale study, it would be considered unethical to withhold it. Here, from the article The ethics of Placebo-controlled Trails: Methodological justifications:

“The use of placebo controls in clinical trials remains controversial. Ethical analysis and international ethical guidance permit the use of placebo controls in randomized trials when scientifically indicated in four cases:

(1) when there is no proven effective treatment for the condition under study;

(2) when withholding treatment poses negligible risks to participants;

(3) when there are compelling methodological reasons for using placebo, and withholding treatment does not pose a risk of serious harm to participants; and, more controversially

(4) when there are compelling methodological reasons for using placebo, and the research is intended to develop interventions that can be implemented in the population from which trial participants are drawn, and the trial does not require participants to forgo treatment they would otherwise receive.” [4]

In truth, these four cases are all pretty much the same case, just phrased in different ways. What they all say, in essence, is you cannot run a trial where you give people a placebo, when there is known to be an effective treatment available. To do so would be, potentially, harming them, deliberately.

Practical Case: Statins

The 4S study (Scandinavian Simvastatin Survival Study) [5] was published in 1994. It was considered the pivotal trial which ‘proved’ that simvastatin/statins reduced overall mortality: in men between the ages of thirty-five to seventy – with previously diagnosed cardiovascular disease – and a total cholesterol > 5.5mmol/l (213mg/dl).

(There were a series of exclusion criteria such as: Atrial fibrillation, history of stroke, heart failure and suchlike.) Because this study was done on people with known CV disease it is referred to as a ‘secondary’ prevention study. You are trying to prevent a second CV episode from happening. Primarily another heart attack, or stroke.

Which means that, after the results of the 4S study were published, a placebo controlled secondary prevention study – using the same, or very similar population – could never be done again. You are no longer allowed to have a placebo group as it would be considered unethical.

Yes, you can still do studies on cholesterol lowering in primary prevention – people not known to have cardiovascular disease. And several of these have been done over the years. You can also do studies in people with ever lower cholesterol levels. For example, the JUPITER study, which was done on participants with normal to low cholesterol levels – but higher levels of inflammatory markers e.g. C-reactive protein.:

“…we hypothesized that people with elevated high-sensitivity C-reactive protein levels but without hyperlipidemia (normal/low cholesterol levels) might benefit from statin treatment.” [6]

They were still allowed to do this as a placebo-controlled study, because prescribing statins to people with average to low cholesterol levels was not part of any guideline at the time, and there was no evidence that giving a placebo here could cause any harm.

Outside of JUPITER, it has also been possible to do studies on people with diabetes, who have normal cholesterol levels. And on you can go, finding new study populations.

Today, virtually every variation of cholesterol levels, and sub-divisions of cholesterol, in every population you can think of has been done. But with each new study, on each different population, another door closes. Until, eventually, there is no placebo-controlled study left to do, on any population.

Catch 22

The 4S study was considered ground-breaking, with significant benefits in reducing cardiovascular and overall mortality. It was the pivotal statin study, and it paved the way for statins (and cholesterol lowering in general) to become the most widely prescribed class of drugs in history.

This study has rarely, if ever, been questioned. It pretty much represents the solid foundations upon which the great cholesterol lowering citadel was built. I consider it to be the single most influential clinical trial in the history of medicine. Had it ‘failed’ I suspect the world would have heard nothing of cholesterol lowering, or statins, ever again.

...it is difficult to think of any study that outranks 4S in terms of the impact on medical treatment. It has driven statin prescribing for hundreds of millions of people over the years. Possibly billions. With around a trillion dollars in profit for the pharmaceutical industry.

I may be guilty of slight hyperbole in making those statements, but it is difficult to think of any study that outranks 4S in terms of the impact on medical treatment. It has driven statin prescribing for hundreds of millions of people over the years. Possibly billions. With around a trillion dollars in profit for the pharmaceutical industry.

What if it could not be replicated? Afterall:

“More than 70% of researchers have tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce their own experiments.” [1]

As explained, 4S can never be replicated, because a placebo arm in any such trial would be refused on the grounds that it is unethical. So, we must trust that the results were true. We have no other choice.

Here the Catch 22 with placebo-controlled trials. Once you have ‘successfully’ done one, you can never try to replicate it, ever again, to prove that it was actually true. And where does that leave us, exactly?