Evidence based decisions in global health

Inspiring and informing policy and practice on global health issues to enhance evidence-based decision making

7th European Conference on Tropical Medicine and International Health, Copenhagen 10.-12. September 2013.

Morten Sodemann

Evidence doesn’t solve problems – people do.

Evidence is just a tool to convince other people that you are right and has nothing to do with innovation, originality or giant scientific steps. A new computer software is currently being developed that can produce Cochrane reviews automatically – there is evidence for you!

The problem is that the way evidence is produced today any robot can do it: its all about routines methodologies, RCTs and guidelines for good clinical practice (GCP). We have forgotten that the real research, the food for the evidence engine, is created outside the lab, far away from GCP. Research is about stumbling over interesting  un-expected observations, we observe strange patterns, we get the wrong result but ask the right question: why do I get the wrong result. So maybe we should actively protect and nourish the curiosity driven research environment that feeds the evidence environment. Without curiosity no basis for evidence.

One basic problem in the discussion of evidence is that there is a basic misconception of what creates good science and how this becomes evidence. A frequently cited quote of uncertain origin says: “The plural of anecdote is not evidence”, but in fact it turns out that who ever said it first actually said the complete opposite: “The plural of anecdote is fact”. Every scientific discovery starts with an anecdote and another anecdote and a researcher suddenly seeing a strange pattern while discussing the anecdotes in a coffee shop with the friend that just told the third anecdote. Unfortunately GCP and RCT studies seem to be winning the funding battle over the anecdotes. We allow scientific journals and funding agencies to choose what research is and what is not: if you find something you didn’t plan it is not research. If you find something that lacks a biological explanation it is not research.

Researchers tend to think their job is finished when they have published their evidence in a paper. Evidence is the easiest part of research – selling your evidence is your real task - researchers have to realize that.

It is a problem that we produce too much media friendly routine evidence and too little unexpected evidence. In a systematic review asking “Is everything associated with cancer?”,  looking at all studies relating food ingredients to cancer risk it was found that over 75 % of ingredients had been related to either increased or decreased risk of cancer. The majority of the studies were in fact weak and inconclusive and therefore useless. We have cornered ourselves with increasingly narrow minded and echo like research questions with taking the time to realize that we are extremely bad at communicating and selling evidence to the public.

Key influencers are considered in a recent paper by Dubois and colleagues published this year in a paper where they study when evidence is sufficient for decision-making and come up with a framework for understanding the pace of evidence adoption.

Five factors stood out: 1) validity, reliability, and maturity of the science available before widespread adoption; 2) communication of the science; 3) economic drivers; 4) patients’ and physicians’ ability to apply published scientific findings to their specific clinical needs; and 5) incorporation into practice guidelines. Now, unfortunately, this research was based on fairly straight forward biomedical interventions like statins and cardiac stents that do not represent potential moral, religious or delicate political undertones. The problem in global health is that it is meant to be global and that many interventions at the global level face country specific and regional barriers that follow invisible and unpredictable paths. To understand how we can enhance evidence based decision making, we need a research field that focuses on these invisible global counter productive factors that slow down the pace of evidence adoption.

Though the world demands evidence, in fact it is often evidence itself that stops innovation and slows the adoption of new theories. Evidence can help us understand the past but gets in our way when we are trying new things. Just because we understand important evidence doesn’t mean we know how to solve important problems. Just because there are 117 clear RCT results it doesn’t mean that decision makers agree, understand or are motivated to carry the evidence into the room where decision makers do their decision making. Science is one thing, evidence another, but decision makers belong to a different environment with an unfamiliar mindset. Decision makers are not particularly ambitious or well informed. Politicians do what they are best at: They like ideas that are easy to understand and sell to the public and they like ideas that can position them and define them distinctly in the political environment. A short scientific executive research brief in the right hands, in the right room, at the right time can far more impact than a paper in Nature.

A comment in a US newspaper titled “Fact is dead” started a global series of newspaper articles and blogs on the way our precious and only asset: research results, had disappeared completely from the face of the earth. Everybody seems to have the right to their own data and their own explanations.  The media, the web, research spin and a failure of the entire research community to realize that we have been shunted out of the decision circles has created this situation. There is no need for evidence any more – we have enough evidence already, seems to be the political opinion - and with modern communication technology decision makers can combine existing evidence and present it as well tailored new evidence that supports their political arguments. Fifty per cent of media spin can be traced back to the abstract of the scientific paper, so scientist have been very instrumental in creating a media environment where facts are in effect dead.

We need researchers that start behaving like scientists by analyzing their situation in relation to decision makers a little better. We, the scientists, are stuck in a scientific echo chamber where we all agree that evidence is available and we agree that we feel paralyzed and ignored by decision makers. But in fact we don’t really agree to all evidence and we even disagree on the presumptions, sometimes  even the basic theories: “that is not research”, “that is not biologically plausible”, “that was not a planned study”, are frequent comments from reviewers of research. We need to take full responsibility of our evidence – this means agreeing on theory, results, interpretations and it means bringing the evidence into the rooms where decisions are made.

Researchers are often told by politicians: If you want to discuss politics – become a politician. So, understanding the process by which some evidence is used in decision making while other evidence is not, could help us in communicating with decision makers. As researchers, we will have to realize that politicians will never come and meet us in the research department. If we want to meet them and interact with them it will be on their terms in their playfield and in their language.

One of the things we have learnt about global epidemics over the past three decades is that it is absolutely crucial to know your epidemic – what are the forces that fuel the epidemic. We learned that the hard way through HIV, bird flu and SARS. We are currently facing a new strange epidemic of absence – namely absence of evidence from public discussion and especially from political decisions.

When politicians say polio – we say polio. When politicians say user fees – we say user fees, and when they say more evidence we say more evidence. How come diarrhea, traffic accidents  and handwashing are boring and low politics while HIV and polio vaccine get all the attention?

As scientists, we have become a minority in public health discussions and decisions. Sometimes decision makers just don’t want our research: it is noisy, uncontrolled and have a lot of assumptions and precautions built into the evidence. And even politicians have become a minority in terms of global health decisions: the non-governmental players like Bill and Melinda Gates and The World bank have taken over that scene.

We missed a perfect opportunity to produce the kind of evidence the world is screaming for by not asking for a data collection system to monitor interventions and effects. Child mortality has fallen drastically since the MDGs were launched, now even in countries that we had given up on, but we forgot to do our research home work: We forgot to monitor was was going on because we were too far away from the decision making and when we finally got closer the party was over.

Likewise global health is basically retrospective and rarely real time. Malaria prevalence has fallen dramatically in tropical countries and everybody, from WHO to The Gates Foundation and the drug industry, is queuing up to take the credit. But some studies have indicated that it could be explained entirely by the massive urbanization that has happened over the past 15 years. Maybe we have created a global health problem by uncontrolled urbanization while we UNPLANNED at the same time got rid of malaria. We will never be able to tell the truth, or learn from it, because scientist never anticipate or want unexpected observations.

In 1996, John Brockman published his book The Third Culture, a term describing the missing link between science and humanities. The third culture is formed around the group of scientists who are also writers and communicators that can talk to everybody and thereby transform knowledge into words and arguments satisfying to a broader audience. Researchers need to adopt some of this third culture if we want to be taken seriously. Traditionally there has been a sort of struggle between science and the humanities. For some reason we are opposed and mutually exclusive arguments, that when we choose one side, the other automatically sacrifices. And there are certain stereotypes, while scientists are, according to the stereotype, unable to speak in a language for ordinary people (sometimes calling them unsociable), humanists write books, novels, to the normal world, stories typically made for the world to understand them. In short, the general stereotype is that a scientist can only communicate successfully with other scientists, while the humanist has everyone in the "normal" range of his pen.

In my view researchers should stop weeping and behave like adult scientist: go back to your labs, stop whatever you are doing there for 6 months  and be scientific about it: analyze successful use of evidence within your research field in decision making. Like Tanzania declared a 6 months break from Foreign aid missions we shall declare a 6 months break in producing time consuming routine evidence and instead engage ourselves in constructive collaborative research seminars where we discuss why it took 10 years to persuade WHO to adopt impregnated bed nets, why hand washing is still so boring and why water pumps are more difficult to maintain than a car on Mars. Some health problems and a few health interventions have a very high political X-factor, but most never make it beyond round one.

I suggest we start behaving like scientists: we should analyze successes and failures in selling our evidence to decision makers. Is it the evidence or the wrapping that is our problem?

We should start communicating like the third culture and learn the language and dynamics of politics through a scientific process:

What were the determinants, the risk factors and confounders for convincing a politician?

Where and by whom are decisions made?

Some innovations spread fast, almost too fast. But how do we speed up the ones that don’t spread so fast.

As Atul Gawande, a public-health researcher, observes in the New Yorker Annals of Medicine earlier this year  "We are chasing frictionless, technological solutions. But people talking to people is still the way that norms and standards change."

That is what global health is about: evidence doesn’t solve problems – people do.