How COVID broke the evidence pipeline

It wasn’t long into the pandemic before Simon Carley realized we had an evidence problem. It was early 2020, and COVID-19 infections were starting to lap at the shores of the United Kingdom, where Carley is an emergency-medicine doctor at hospitals in Manchester. Carley is also a specialist in evidence-based medicine — the transformative idea that physicians should decide how to treat people by referring to rigorous evidence, such as clinical trials.

As cases of COVID-19 climbed in February, Carley thought that clinicians were suddenly abandoning evidence and reaching for drugs just because they sounded biologically plausible. Early studies Carley saw being published often lacked control groups or enrolled too few people to draw firm conclusions. “We were starting to treat patients with these drugs initially just on what seemed like a good idea,” he says. He understood the desire to do whatever is possible for someone gravely ill, but he also knew how dangerous it is to assume a drug works when so many promising treatments prove to be ineffective — or even harmful — in trials.

“The COVID-19 pandemic has arguably been one of the greatest challenges to evidence-based medicine since the term was coined in the last century,” Carley and his colleagues wrote of the problems they were seeing¹.

Other medical experts echo these concerns. With the pandemic now deep into its second year, it’s clear the crisis has exposed major weaknesses in the production and use of research-based evidence — failures that have inevitably cost lives. Researchers have registered more than 2,900 clinical trials related to COVID-19, but the majority are too small or poorly designed to be of much use (see ‘Small samples’).

Organizations worldwide have scrambled to synthesize the available evidence on drugs, masks and other key issues, but can’t keep up with the outpouring of new research, and often repeat others’ work. There’s been “research waste at an unprecedented scale”, says Huseyin Naci, who studies health policy at the London School of Economics.

At the same time, shining examples of good practice have emerged: medical advances based on rigorous evidence have helped to chart a route out of the pandemic. The rapid trials of vaccines were spectacular successes, and well-run trials of possible treatments have shown, for instance, that some steroids help to fight COVID-19, but the drug hydroxychloroquine doesn’t. Many physicians point to the United Kingdom’s RECOVERY trial as exemplary in showing how quick action and simple protocols make it possible to conduct a large clinical trial in a crisis.

And researchers have launched ‘living’ systematic reviews that are constantly updated as research emerges — essential in a fast-moving disease outbreak.

As the COVID-19 response turns from a sprint to a marathon, researchers are taking stock and looking ahead. In October, global-health leaders will meet for three days to discuss what’s been learnt from COVID-19 about supplying evidence in health emergencies. COVID-19 is a stress test that revealed the flaws in systems that produce evidence, says Elie Akl, an internal-medicine specialist and clinical epidemiologist at the American University of Beirut. “It would be shameful if we come out of this experience and not make the necessary change for the next crisis.” ...