“I Learnt Very Early On In My Career That You Cannot Slice Data The Way They Did.” Says John Mandrola, A Cardiac Electrophysiologist.​

There has been a lot of buzz around a new cholesterol-lowering drug called bempedoic acid. In the US, the drug is approved for people who are intolerant to statin therapy.

In a major trial published in New England Journal of Medicine of over 14,000 statin intolerant people, bempedoic acid reduced cardiovascular events (composite of cardiovascular death, nonfatal MI, nonfatal stroke, or coronary revascularisation) by 13% (relative risk) or 1.6% (absolute risk).

The trial had a mix of primary prevention (30%) and secondary prevention (70%) subjects, and the authors were interested to know if either subgroup benefited more from bempedoic acid.

So, they carried out a subgroup analysis and published the findings in the Journal of the American Medical Association (JAMA) two months after their full report in NEJM.

The findings

THE SIZE OF THE EFFECT WHICH RIVALS EVEN SOME OF THE MOST EFFECTIVE MEDICATIONS, IMMEDIATELY RAISED SUSPICIONS SINCE THE MAIN TRIAL SHOWED NO SUCH BENEFIT IN THESE ENDPOINTS. NONETHELESS, OVER 80 NEWS OUTLETS COVERED THE STORY, INCLUDING INTERVIEWS WITH THE LEAD AUTHOR OF THE STUDY STEVE NISSEN, A CARDIOLOGIST FROM CLEVELAND CLINIC.

The findings only disclosed data on the primary prevention group – the secondary prevention group was excluded from the publication.

In the primary prevention group, bempedoic acid reduced cardiovascular events (i.e. cardiovascular death, nonfatal MI, nonfatal stroke, or coronary revascularisation) by 30% (2.3% absolute reduction) compared to placebo.

The authors then carved up the data further to show bempedoic acid reduced cardiovascular death by 39% (1.3% absolute reduction) and all-cause death by 27% (1.6% absolute reduction) compared to placebo.

The size of the effect which rivals even some of the most effective medications, immediately raised suspicions since the main trial showed no such benefit in these endpoints.

Nonetheless, over 80 news outlets covered the story, including interviews with the lead author of the study Steve Nissen, a cardiologist from Cleveland Clinic.

Nissen told reporters that the new results were “frankly striking” and a “wake-up call for the medical community that we need to pay far more attention to high-risk primary prevention patients.”

Critics push back 

Despite the enthusiastic press coverage, notable researchers were quick to point out flaws in the analysis. 

Subgroup analyses

John Mandrola, a cardiac electrophysiologist at Baptist Health in Kentucky, was particularly critical of the methods used in the study.

“I’m going to try and stay calm through this,” said Mandrola. “I learnt very early on in my career that you cannot slice data the way they did.”

“You could excuse this if they were junior researchers who didn’t know better,” he said, “but the authors clearly know it’s a dubious analysis. The JAMA editors know it’s dubious, and yet there it is, published and promoted in media.”

Mandrola said normally trials are designed with a certain number of participants in order to sufficiently power the study to show statistical significance.

“But when you take a subgroup, you’re just taking a small slice of the pie, and if you’re cutting it up, you will have a smaller number of events, and it’s going to be more way more susceptible to [statistical] noise,” said Mandrola.

“It’s hard not to be cynical about the research profession, when you see stuff like this in the highest-level journals,” he added. “I’m really worried about the state of adjudication of science.”

Vinay Prasad, a haematologist oncologist at San Francisco General Hospital agreed with Mandrola saying, “It’s very rare for a top journal to publish a subgroup analysis of a paper because it’s considered low credibility. I think it’s shocking.”

Prasad added, “Another way to think about it is if you get a Jackson Pollock painting where he splattered paint all over the canvas. And if you allow me – after the fact – to cut out the part that’s green, I can cut out pieces that look whatever colour you want, it doesn’t mean it’s a real signal.”

Rita Redberg, cardiologist, and former editor of JAMA Internal Medicine, agreed.

“If you want to show the benefit of a drug in healthy people in primary prevention, then you should do a trial in that population. I don’t think doing a subgroup analysis is reliable,” said Redberg.

“If you have enough subgroups, you’re going to find some subgroups where it will look like the drug works,” added Redberg.

In fact, Redberg pointed to the ISIS-2 study which famously demonstrated the fallibility of subgroup analyses.

ISIS-2 was a large trial that reported the benefits of aspirin on heart attacks. But when the authors analysed subgroups within the trial, as a protest after the journal editor demanded they do subgroup analyses, the researchers demonstrated that the drug could also benefit people based on their astrological star sign – only Geminis and Librans did not benefit.

The study, to this day, serves as a stinging reminder of why subgroup analyses are scientifically dubious and should require confirmation with a properly designed RCT of that particular subgroup.

Missing data

The alarm was raised when the authors did not publish the secondary prevention results in the same paper.

James McCormack, a professor in the Faculty of Pharmaceutical Sciences at the University of British Columbia said, “There was not a single mention of the secondary prevention group in the JAMA subgroup analysis.”

“If you decided to do a subgroup analysis on a large trial which included different sexes, why would you only publish data on the male subgroup and exclude the female subgroup? It doesn’t make sense,” said McCormack.

Nissen responded to the criticism saying that “including both subgroups would have generated a manuscript much too large for a single paper” and that “additional presentations and manuscripts describing the secondary prevention findings are nearing submission.”

McCormack did not accept Nissen’s explanation, saying it was a “ridiculous” excuse.

“It would’ve been easy to present both subgroups in the same paper.” said McCormack. “It should have been picked up in peer-review, but then again, peer-review is not perfect”.

I asked Nissen if the peer-reviewers had requested the secondary prevention group data before publication, but said he could not answer my query “due to the strict ethical rules about the confidentiality of peer review.”

In the absence of this data, McCormack did his own analysis by subtracting the primary prevention group from the overall data to get results for the secondary prevention group.

By his calculations, McCormack estimated that the drugs benefit in reducing cardiovascular events was lost in the secondary prevention group, which he says, doesn’t add up.

“It’s really strange a drug would work in primary prevention, and not work in secondary prevention because in secondary prevention people are at a higher risk of heart attacks,” said McCormack.

Even Gregory Curfman, executive editor of JAMA raised doubts about the findings of Nissen’s analysis during a public discussion with Dhruv Kazi, a cardiologist at Beth Israel Deaconess Medical Center in Boston.

Curfman said:

“It’s puzzling at least to me, that there would be a reduction in cardiovascular mortality and overall mortality in the primary prevention subgroup, but not in the overall trial of all the patients, suggesting that there might have even been an increase in these mortality endpoints in the secondary prevention patients.”

Kazi replied:

“I agree with you. This is somewhat conflicting to see the signal of pretty substantial benefit in mortality in the primary prevention arm versus no evidence of benefit in the overall trial and possibility even of harm in the secondary prevention arm. I think that this temptation to think that bempedoic acid is somehow more effective in primary prevention than in secondary prevention and may even be more effective than statins is one that we should forego. I think that conclusion would be incorrect.”

Benefits not mediated via cholesterol-lowering

Finally, McCormack said bempedoic acid reduced LDL-cholesterol to an equivalent degree in both the secondary and primary prevention groups yet, the cardiovascular outcomes for each group were not equivalent. This suggests if there is an effect from bempedoic acid it’s not because of its ability to lower cholesterol.

When I asked Nissen to explain the paradox he simply replied, “We were careful not to make any claim about the mediation of the results”.

McCormack said, “I think this puts Nissen in a corner. He does not address this in the paper because it requires an explanation that goes against the whole LDL hypothesis which is difficult for some researchers to accept.”

“I’m not interested in whether a drug lowers cholesterol, I’m only interested if it lowers cardiovascular outcomes. There are a number of medications that lower cholesterol, but they don’t change cardiovascular outcomes,” added McCormack.

Researchers now await Nissen’s upcoming paper which they hope will give more context to the latest controversial analysis.