Sunday, February 8, 2015

Let's Play Outcome Mash-up - A Clinical Trial Shortcut Classic!

Deciphering trial outcomes can be a tricky business. As if many measures aren't hard enough to make sense of on their own, they are often combined in a complex maneuver called a composite endpoint (CEP) or composite outcome. The composite is treated as a single outcome. And journalists often phrase these outcomes in ways that give the impression that each of the separate components has improved.

Here's an example from the New York Times, reporting on the results of a major trial from the last American Heart Association conference:
"There were 6.4% fewer cardiac events - heart disease deaths, heart attacks, strokes, bypass surgeries, stent insertions and hospitalization for severe chest pain..."
That individual statement sounds like the drug reduced deaths, bypasses, stents, and hospitalization for unstable angina, doesn't it? But it didn't. The modest effect was on non-fatal heart attacks and stroke only.*

CEPs are increasingly common: by 2007, well over a third of cardiovascular trials were using them. CEPs are a clinical trial shortcut because you need fewer people and less time to hit a jackpot. A trial's main pile of chips is riding on its pre-specified primary outcome: the one that answers the trial's central, most important question.

The primary outcome determines the size and length of the trial, too. For example, if the most important outcome for a chronic disease treatment is to increase the length of people's lives, you would need a lot of people to get enough events to count (the event in this case would be death). And it would take years to get enough of those events to see if there's anything other than a dramatic, sudden difference.

But if you combine it with one or more other outcomes - like non-fatal heart attacks and strokes - you'll get enough events much more quickly. Put in lots, and you're really hedging your bets.

It's a very valuable statistical technique - but it can go haywire. Say you have 3 very serious outcomes that happen about as often as each other - but then you add another component that is less serious and much more common. The number of less serious events can swamp the others. Everything could even be riding on only one less serious component. But the CEP has a very impressive name - like "serious cardiac events." Appearances can be deceptive.

Enough data on the nature of the events in a CEP should be clearly reported so that this is obvious, but it often isn't. And even if the component events are reported deep in the study's detail, don't be surprised if it's not pointed out in the abstract, press release, and publicity!

There are several different ways a composite can be constructed, including use of techniques like weighting that need to be transparent. Because it's combining events, there has to be a way of dealing with what happens when more than one event happens to one person - and that's not always done the same way. The definitions might make it obvious, the most serious event might count first according to a hierarchy, or the one that happened to a person first might be counted. But exactly what's happening often won't be clear - maybe even most of the time.

There's agreement on some things you should look out for (see for example Montori, Hilden, and Rauch). Are each of the components as serious as each other and/or likely to increase (or decrease) together in much the same way? If one's getting worse and one's getting better, this isn't really measuring one impact.

The biggest worry, though, is when researchers play the slot machine in my cartoon (what we call the pokies, "Downunder"). I've stressed the dangers of hunting over and over for a statistical association (here and here). The analysis by Lim and colleagues found some suggestion that component outcomes are sometimes selected to rig the outcome. If it wasn't the pre-specified primary outcome, and it wasn't specified in the original entry for it in a trials register, that's a worry. Then it wasn't really a tested hypothesis - it's a new hypothesis.

Composite endpoints, properly constructed, reported, and interpreted are essential to getting us decent answers to many questions about treatments. Combining death with serious non-fatal events makes it clear when there's a drop in an outcome largely because people died before that could happen, for example. But you have to be very careful once so much is compacted into one little data blob.

 (Check out slide 14 to see the forest plot of results for the individual components the journalist was reporting on. Forest plots are explained here at Statistically Funny.)

More on understanding clinical trial outcomes:

New this week: I'm delighted to now have a third blog, one for physicians with the wonderful team at MedPage Today. It's called Third Opinion.

Sunday, November 30, 2014

Biomarkers Unlimited: Accept Only OUR Substitutes!

Sounds great, doesn't it? Getting clinical trial results quickly has so much going for it. Information sooner! More affordable trials!

Substituting outcomes that can take years, or even decades, to emerge, with ones you can measure much earlier, makes clinical research much simpler. This kind of substitute outcome is called a surrogate (or intermediate) endpoint or outcome.

Surrogates are often biomarkers - biological signs of disease or a risk factor of disease, like cholesterol in the blood. They are used in clinical care to test for, or keep track of, signs of emerging or progressing disease. Sometimes, like cholesterol, they're the target of treatment. 

The problem is, these kinds of substitute measures aren't always reliable. And sometimes we find that out in the hardest possible way.

The risk was recognized as soon as the current methodology of clinical trials was being developed in the 1950s. A famous statistician who was key to that process, Austin Bradford-Hill, put it bluntly: if the "rate falls, the pulse is steady, and the blood pressure impeccable, we are still not much better off if unfortunately the patient dies."

That famously happened with some drugs that controlled cardiac arrhythmia - irregular heartbeat that increases the chances of having a heart attack. On the basis of ECG tests that showed the heartbeat was regular, these drugs were prescribed for years before a trial showed that they were causing tens of thousands of premature deaths, not preventing them. That kind of problem has happened too often for comfort.

It happened again this week - although at least before the drug was ever approved. A drug company canceled all its trials for advanced gastric (stomach) cancer of a new drug. The drug is called Rilotumumab. Back in January, it was a "promising" treatment, billed as bringing "new hope in gastric cancer." It got through the early testing phases and was in Phase III trials - the kind needed to get FDA approval.

But one phase III trial, RILOMET-1, quickly showed an increase in the number of deaths in people using the drug. We don't know how many yet - but it was enough for the company to decide to end all trials of the substance.

This drug targets a biomarker associated with worse disease outcomes, an area seen by some as transforming gastric cancer research and treatment. Others see considerable challenges, though - and what happened to the participants in the RILOMET-1 trial underscores why.

There is a lot of controversy about surrogate outcomes - and debates about what's needed to show that an outcome or measure is a valid surrogate we can rely on. They can lead us to think that a treatment is more effective than it really is.

Yet a recent investigative report found that cancer drugs are being increasingly approved based only on surrogate outcomes, like "progression-free survival." That measures biomarker activity rather than overall survival (when people died).

It can be hard to recognize at first, what's a surrogate and what's an actual health outcome. One rule of thumb is, if you need a laboratory test of some kind, it's more likely to be a surrogate. Whereas symptoms of the disease you're concerned, or harm caused by the disease, are the direct outcomes of interest. Sometimes those are specified as"patient-relevant outcomes."

Many surrogate outcomes are incredibly important, of course - viral load for HIV treatment and trials for example. But in general, when clinical research results are based only on surrogates, the evidence just isn't as strong and reliable as it is for the outcomes we are really concerned about.


See also, Statistically Funny on "promising" treatments.

Sunday, October 12, 2014

Sheesh - what are those humans thinking?

I can neither confirm nor deny that Cecil is now a participant in one of the there-is-no-limit-to-the-human-lifespan resveratrol studies at Harvard's "strictly guarded mouse lab." If he is, I'm sure he's even more baffled by the humans' hype over there.

Resveratrol is the antioxidant in grapes that many believe makes drinking red wine healthy. And it's a good example of how research on animals is often terribly misleading and misinterpreted. I've written about it over at my Scientific American blog if you're interested in more detail about resveratrol.

But this week, it's media hype about a study using human stem cells in mice in another lab at Harvard that's made me ratty. You could get the idea that a human trial of a "cure" for type 1 diabetes is just a matter of time now - and not a lot of time at that. According to the leader of the team, Doug Melton, "We are now just one preclinical step away from the finish line."

An effective treatment that ends the need for insulin injections would be incredibly exciting. But we see this kind of claim from laboratory research all the time, don't we? How often does it work out - even for the studies that are at "the finishing line" for animal studies?

Not all that often: maybe about a third of the time.

Bart van der Worp and colleagues wrote an excellent paper explaining why. It's not just that other animals are so different from humans. We're far less likely to hear of the failed animal results than we are of human trials that don't work out as hoped. That bias towards positive published results draws an over-optimistic picture.

As well as fundamental differences between species, van der Worp points to other common issues that reduce the applicability for humans of typical studies in other animals:

  • The animals tend to be younger and healthier than the humans who have the health problem;
  • They tend to be a small group of animals that are very similar to each other, while the humans with the problem are a large very varied group;
  • Only male or only female animals are often used; and
  • Doses higher than humans will be able to tolerate are generally used.
Limited genetic diversity could be an issue, too.

So how does the Harvard study fare on that score? They used stem cells to develop insulin-producing cells that appeared to function normally when transplanted into mice. But this was the very early stages. When it came to the test they reported on the ones with diabetes, there were only 6 (young) mice who got the transplants (and 1 died) (plus a comparison group). Gender was not reported - and as is common in laboratory animal studies, there wasn't lengthy follow-up. This was an important milestone, but there's a very long way to go here. Transplants in humans face a lot of obstacles.

Van der Worp points to another set of problems: inadequacies in research methods that we've learned over time in human research bias the proceedings too much - including problems with statistical analyses. Jennifer Hirst and colleagues have studied this too. They concluded that so many studies were bedeviled by issues such as lack of randomization and blinding by those assessing outcomes, that they should never have been regarded as being "the finishing line" before human experimentation at all.

There's good news though! CAMARADES is working to improve this - with the same approach for chipping away at these problems as in human trials: by slogging away at biased methodologies and publication bias. And pushing for good quality systematic reviews of animal studies before human trials are undertaken.

Laboratory animal research may be called "preclinical," but even that jargon is a bit of over-optimistic marketing. Most of what's tried in the lab will never get near human trials. And when it does, it will mostly be disappointing. Laboratory research is needed, and encouraging progress is great. But people should definitely not be getting our hopes up too much about it.


The National Institutes of Health (NIH) addressed the issue of gender in animal experiments earlier in 2014. After I wrote this post, the NIH also released proposed guidelines for reporting preclinical research.

Thanks to Jonathan Eisen for adding a link for the full text of the paper to PubMed Commons, as well as to a blog post by Paul Knoepfler discussing the context of the stem cell work by Felicia Pagliuca, Doug Melton and colleagues. NHS Behind the Headlines have also analyzed and explained this study.

Thanks to Jim Johnson for pointing an oversight: that animal studies - this one included - can also suffer from having too little follow-up.

Interest declaration: I'm an academic editor at one of the journals whose papers on animal research I commended (PLOS Medicine) and on the human ethics advisory group of another (PLOS One), but I had no involvement in either paper.

Sunday, March 16, 2014

If at first you don't succeed...

If only post hoc analyses always brought out the inner skeptic in us all! Or came with red flashing lights instead of just a little token "caution" sentence buried somewhere. 

Post hoc analysis is when researchers go looking for patterns in data. (Post hoc is Latin for "after this.") Testing for statistically significant associations is not by itself a way to sort out the true from the false. (More about that here.) Still, many treat it as though it is - especially when they haven't been able to find a "significant" association, and turn to the bathwater to look for unexpected babies.

Even when researchers know the scientific rules and limitations, funny things happen along the way to a final research report. It's the problem of researchers' degrees of freedomthere's a lot of opportunity for picking and choosing, and changing horses mid-race. Researchers can succumb to the temptation of over-interpreting the value of what they're analyzing, with "convincing self-justification." (See the moving goalposts over time here, for example, as trialists are faced with results that didn't quite match their original expectations.)

And even if the researchers don't read too much into their own data, someone else will. That interpretation can quickly turn a statistical artifact into a "fact" for many people.

Let's look more closely at Significus' pet hate: post hoc analyses. There are dangers inherent in multiple testing when you don't have solid reasons for looking for a specific association. The more often you randomly dip into data without a well-founded target, the higher your chances of pulling out a result that will later prove to be a dud.

It's a little like fishing in a pond where there are random old shoes among the fish. The more often you throw your fishing line into the water, the greater your chances of snagging a shoe instead of a fish.

Here's a study designed to show this risk. The data tossed up significant associations such as: women were more likely to have a cesarean section if they preferred butter over margarine, or blue over black ink.

The problem is huge in areas where there's a lot of data to fish around in. For published genome-wide association studies, for example, over 90% of the "associations" with a disease couldn't consistently be found again. Often, researchers don't report how many tests were run before they found their "significant" results, which makes it impossible for others to know how big a problem multiple testing might be in their work.

The problem extends to subgroup analyses where there is not an established foundation for an association. The credibility of claims made on subgroups in trials is low. And it has serious consequences. For example, an early trial suggested only men with stroke-like symptoms benefit from aspirin - which stopped many doctors from prescribing aspirin to women.

How should you interpret post hoc and subgroup analyses then? If analyses were not pre-specified and based on established, plausible reasons for an association, then one study isn't enough to be sure.

With subgroups that weren't randomized as different arms of a trial, it's not enough that the average for one subgroup is higher than the average for another subgroup. There could be other factors influencing the outcome other than their membership of that subgroup. An interaction test is done to try to account for that.

It's more complicated when it's a meta-analysis, because there are so many differences between one study and another. The exception here is an individual patient data meta-analysis, which can study differences between patients directly.

In the end, it comes down to being careful not to see a new hypothesis generated by research as a "fact" already proven by the study from which it came.

Post hoc, ergo propter hoc. This description of basic faulty logic - "after this, therefore because of this" - is as ancient as the language that made it famous. We've had millennia to snap out of the dangerous mental shortcut of seeing a cause where there's only coincidence. Yet we still hurtle like lemmings over cliffs into its alluring clutches.

Sunday, December 29, 2013

What's so good about "early," anyway?

"Early." It's one of those words like "new" and "fast," isn't it? As though they are inherently good, and their opposites - "late," "old" and "slow" - are somehow bad.

Believing in the value and virtue of being an early bird has deep roots in our cultural consciousness. It goes back at least as far as ancient Athens. Aristotle's treatise on household economics said that early rising was both virtuous and beneficial: "It is likewise well to rise before daybreak; for this contributes to health, wealth and wisdom."

But just as Gertrud came to suspect the benefits for her of being early weren't all they were cracked up to be, earliness isn't always better in other areas either. The "get in early!" assumption has an in-built tendency to lead us astray when it comes to detection of diseases and conditions. And even most physicians - just the people we often rely on to inform us - don't understand enough about the pitfalls that lead us to jump to conclusions about early detection too, well…early.

Pitfall number 1: Those who need it least get the most early detection

This one is a double-edged sword. Firstly, whether it's a screening program or research studying early detection, there tends to be a "worried well" or "healthy volunteer" effect (selection bias). It's easy to have higher than average rates of good health outcomes in people who are at low risk of bad ones anyway. This can lead to inflated perceptions of how much benefit is possible.

The other problem is an over-supply of fatalism among many people who may be able to materially benefit from early detection. Constant bombardment about all the things they could possibly be worrying about might even make it more likely that they shut out vital information - which could make it even more likely that they ignore symptoms, for example.

Pitfall number 2: Over-diagnosis from detecting people who would never have become ill from the condition detected

This one is called length bias. For many conditions, like cancers, there are dangerous ones that develop too quickly for a screening program to catch them. Early detection is actually better at detecting ones that may never threaten their health. More people die with cancer, than of it.

So early detection means many people are fighting heroic battles that were never necessary. And some will actually be harmed by parts of some screening processes that carry serious risks of their own (like colonoscopies), or adverse effects of the treatments they got which they didn't need.

Add to those the number of people who are diagnosed as being "at risk" of conditions they will never have or which would have resolved without treatment, and the number harmed is depressingly huge.

This massive swelling of the numbers of people who have survived phantoms is spreading the shadow of angst ever wider (a subject I've written about in relation to cancer at Scientific American). Spend 10 minutes or so listening to Iona Heath on this subject - starting just past 2 minutes on this video. [And read @Deevybee's important comment and links about developmental conditions in early childhood below.]

Pitfall number 3: The statistical effect that means survival rates "improve" even if no one's life expectancy increases

This is lead-time bias. And it's why you should always be careful when you see survival rates in connection with early detection and treatment. Screening programs, by definition, are for people who have no symptoms (pre-clinical). So they cut the part of your life where you don't know you have the disease short. Even if the earlier diagnosis made no difference to the length of your life, the amount of time you lived with knowledge of the disease (disease "survival") is longer.

What we want is to move the needle on length and/or quality of life. For that to happen, there has to be safe and effective treatment, safe and effective screening procedures, and more people found at a time they can be helped than would have come from diagnosing the condition when there symptoms.

Here's an example. This person's disease began when they were 40 years old. They lived without any problem from it until 76 years old - then they died when they were 80. Their disease survival was less than 5 years. The proportion of their life that they "had" the disease was short.

Now here's the same person, with early detection that made no difference to when they died - but the needle on how long they have "had" the disease has shifted. So they now survive longer than 5 years with the disease. The "lead time" has changed, but survival in the way we mean it hasn't changed at all.

Randomized trials are needed to establish that in fact early detection and intervention programs do more good than harm - some do, some don't.

More Statistically Funny on screening - "You have the right to remain anxious" and on over-diagnosis: here and here.

Here's a fact sheet about what you need to know about screening tests. And here's a little more technical primer of the 3 biases explained here.

Sunday, November 17, 2013

Does it work? Beware of the too-simple answer

Leonard is so lucky! He's just asked a very complicated question and he's not getting an over-confident and misleading answer. Granted, he was likely hoping for an easier one! But let's dive into it.

"Does": that auxiliary verb packs a punch. How do we know whether something does or doesn't work?   It would be great if that were simple, but unfortunately it's not.

I talk a lot here at Statistically Funny about the need for trials and systematic reviews of them to help us find the answers to these questions. But whether we're talking about trials or other forms of research, statistical techniques are needed to help make sense of what emerges from a study.

Too often, this aspect of research is going to lead us down a garden path. It's common for people to take the approach of relying only, or largely, on a statistical significance test of the null hypothesis: the assumption that there is no difference. So if a result is within the range that could occur by chance alone, the assumption of the null hypothesis stands. But if it's not within that range, it's "statistically significant."

However a statistically significant result - especially from a single study - is often misunderstood and contributes to over-confidence about what we know. It's not a magical wand that finds out the truth. I wrote about testing for statistical significance in some detail at Absolutely Maybe. Leonard's statistician is a Bayesian: you can find out some more about that, too, in my post.

As chance would have it, there was also a lot of discussion this week in response to a paper published while I was writing that post. It called for a tightening of the threshold for significance, which isn't really the answer either. Thomas Lumley puts that into great perspective over at his wonderful blog, Biased and Inefficient: a very valuable read.

"It": now this part should be easy, right? Actually, this can be particularly tricky. The treatment you could be using may not be very much like the one that was studied. Even if it's a prescription drug, the dose or regimen you're facing might not be the same as the one used in studies. Or it might be used in conjunction with another intervention that could affect how it works.

Then there's the question of whether "it" is even what it says it is. Unlike prescription drugs, the contents of herbal remedies and dietary supplements aren't closely regulated to ensure that what it says on the label is what's inside. That was also recently in the news, and covered in detail here by Emily Willingham.

If it's a non-drug intervention, it's actually highly likely that the articles and other reports of the research don't ever make clear exactly what "it" is. Paul Glasziou had a brainwave about this: he's started HANDI: the Handbook of Non-Drug Intervention. When a systematic reviews shows that something works, the HANDI team wants to dig out all the details and make sure we all know exactly what "it" is.

For example, if you heard that drinking water before meals can help you lose weight, and you want to try it, HANDI helpfully points out what that actually means is drinking half a liter of water before every meal AND having a low-calorie diet. HANDI is new, so there aren't many "it"s explained. But you can see them here.

"Work": this one really needs to get specific. As I point out in the slides from a talk I gave this month, you really need to be thinking about each possible outcome separately - and thinking about the possible adverse effects too. There can be complicated trade-offs between effects, and the quality of the evidence is going to vary for each of them.

Think of it this way: if you do a survey with 150 questions in it, there are going to be more answers to some of the questions than others. For example, if you had 400 survey respondents, they might all have answered the first easy question and there could be virtually no answers to a hard question near the end. So thinking "a survey of 400 people found…" an answer to that later question is going to be seriously misleading.

Then there's the question of how much does it work for that particular outcome? Does a sliver of a benefit count to you as "working"? That might be enough for the person answering your question, but it might not be enough for it to count for you - especially if there are risks, costs or inconvenience involved.

And there's who did it work for in the research? Whether or not research results apply to a person in your situation can be straightforward, but it might not be.

And how high did researchers set the bar? Did the treatment effect have to be superior to doing nothing, or doing something else - or is the information coming from comparing it to something else that itself may not be all that effective? You might think that can't possibly happen, but it does more often than you might think. You can find out about this here at Statistically Funny, where I tackle the issue of drugs that are "no worse (more or less)." 

Finally, one of the most common trip-ups of all: did they really measure the outcome, or a proxy for it? If it's a proxy for the real thing, how good is it? The use of surrogate measures or biomarkers is increasing fast: you can learn more about why this can lead to an unreliable answer here.

So while there are many who might have told Leonard, "Yes, it's been proven to work in clinical trials" in a few seconds flat, I wonder how long it would take his statistician to answer the question? There are no stupid questions, but beware of the too-simple answer.

Monday, September 16, 2013

More than one kind of self-control

If you like reading randomized trials about skin and oral health treatments - and who doesn't? - you come across a few split-face and split-mouth ones. Instead of randomizing groups of people to different interventions so that a group of people can be a control group (parallel trials), sections of a person are randomized.

It's not only done with faces and teeth. Pairs of body parts can be randomized too, like arms or legs. These studies are sometimes called "within-person" trials. This kind of randomization means that you need fewer people in the trial, because you don't have to account for all the variations between human beings.

It has to be a treatment that affects only the specific area of the body treated, though. Anything that could have an influence on the "control" part is called a spill-over effect. There are still inevitably things that happen that affect the whole person, and those have to be accounted for with this kind of trial. Body part randomization is one of several ways a person can be their own control: the n of 1 trial is another way.

Randomizing sections didn't start in trials with people: it began with split-plot experiments in agricultural research. The idea was developed by the pioneer statistician, Sir Ronald Aylmer Fisher, who had done breeding experiments. He explained the technique in his classic 1925 text, "Statistical Methods for Research Workers."

It's great to see that neither blackheads nor treatment effects are hampering the Twilling sisters' style! They do seem to be at risk of susceptibility to the skincare industry's hard sells, though. Those issues are the subject of a post over at my Scientific American blog, called "Blemish: The truth about blackheads."

Sunday, July 28, 2013

Alleged effects include howling

When dogs howl at night, it's not the full moon that sets them off. Dogs are communicating for all sorts of reasons. We're just not all that good at understanding what they're saying.

We make so many mistakes about attributing cause and effect for so many reasons, that it's almost surprising we get it right as often as we do. But all those mistaken beliefs we realize we have, don't seem to teach us a lesson. Pretty soon after catching ourselves out, we're at it again, taking mental shortcuts, being cognitive misers.

It's so pervasive, you would think we would know this about ourselves, at least, even if we don't understand dogs. Yet we commonly under-estimate how much bias is affecting our beliefs. That's been dubbed the bias blind spot that we (allegedly) tend to live in.

Even taking all that into account, "effect" is an astonishingly over-used word, especially in research and science communication where you would hope people would be more careful. The maxim that correlation (happening at the same time) does not necessarily mean causation has spread far and wide, becoming something of a cliche along the way.

But does that mean that people are as careful with the use of the word "effect" as they are with the use of the "cause" word? Unfortunately not.

Take this common one: "Side effects include...." Well, actually, don't be so fast to swallow that one. Sometimes, genuine adverse effects will follow that phrase. But more often, the catalogue that follows is not adverse effects, but a list of adverse events - things that happened (or were reported). Some of them may be causally related, some might not be.

You have to look carefully at claims of benefits and harms. Even researchers who aren't particularly biased will word it carelessly. You will often hear that 14% experienced nausea, say - without it being pointed out that 13% of people on placebos also experienced nausea, and the difference wasn't statistically significant. Some adverse effects are well known, and it doesn't matter (diarrhea and antibiotics, say). That's not always so, though - a complex subject I'll get to on a future Statistically Funny, so watch this space.

If the word "effect" is over-used, the word "hypothesis" is under-used. Although generating hypotheses is a critical part of science, hypotheses aren't really marketed as what they are: ideas in need of testing. Often the language is that of attribution throughout, with a little fig-leaf of a sentence tacked on about the need for confirmatory studies. In fact, we cannot take replication and confirmation for granted at all.

Occasionally, the word "effect" is used to name a literal "hypothesis." That happened with "the Hawthorne effect." You can read more about that in my post, The Hawthorne effect: An old scientists' tale lingering "in the gunsmoke of academic snipers"Yes, Statistically Funny has a little sister at Scientific American now: Absolutely Maybe.

Sunday, June 30, 2013

Goldilocks and the three reviews

Goldilocks is right: that review is FAR too complicated. The methods section alone is 652 pages long! Which wouldn't be too bad, if it weren't that it is a few years out of date. It took so long to do this review and go through rigorous enough quality review, it was already out of date the day it was released. Something that happens often enough to be rather disheartening.

When methodology for systematic reviewing gets overly rococo, the point of diminishing returns will be passed. That's a worry, for a few reasons. For one, it's inefficient and more reviews could be done with the resources. Secondly, more complex methodology can both be daunting, and it can be hard for researchers to accomplish with consistency. Thirdly, when a review gets very elaborate, reproducing or updating it isn't going to be easy either.

It's unavoidable for some reviews to be massive and complex undertakings, though, if they're going to get to the bottom of massive and complex questions. Goldilocks is right about review number 2, as well: that one is WAY too simple. And that's a serious problem, too.

Reviewing evidence needs to be a well-conducted research exercise. A great way to find out more about what goes wrong when it's not, is reading Testing Treatments. And see more on this here at Statistically Funny, too.

You need to check the methods section of every review before you take its conclusions seriously - even when it claims to be "evidence-based" or systematic. People can take far too many shortcuts. Fortunately, it's not often that a review gets as bad as the second one Goldilocks encountered here. The authors of that review decided to include only one trial for each drug "in order to keep the tables and figures to a manageable size." Gulp!

Getting to a good answer also quite simply takes some time and thought. Making real sense of evidence and the complexities of health, illness and disability is often just not suited to a "fast food" approach. As the scientists behind the Slow Science Manifesto point out, science needs time for thinking and digesting.

To cover more ground, people are looking for reasonable ways to cut corners, though. There are many kinds of rapid review, including reliance on previous systematic reviews for new reviews. These can be, but aren't always, rigorous enough for us to be confident about their conclusions.

You can see this process at work in the set of reviews discussed at Statistically Funny a few cartoons ago. Review number 3 there is in part based on review number 2 - without re-analysis. And then review number 4 is based on review number 3.

So if one review gets it wrong, other work may be built on weak foundations. Li and Dickersin suggest this might be a clue to the perpetuation of incorrect techniques in meta-analyses: reviewers who got it wrong in their review, were citing other reviews that had gotten it wrong, too. (That statistical technique, by the way, has its own cartoon.)

Luckily for Goldilocks, the bears had found a third review. It had sound methodology you can trust. It had been totally transparent from the start - included in PROSPERO, the international prospective register for systematic reviews. Goldilocks can get at the fully open review quickly via PubMed Health, and its data are in the Systematic Review Data Repository, open to others to check and re-use. Ahhh - just right!

I'm grateful to the Wikipedians who put together the article on Goldilocks and the three bears. That article pointed me to the fascinating discussion of "the rule of three" and the hold this number has on our imaginations.

Sunday, June 23, 2013

Studies of cave paintings have shown....

The mammoth has a good point. Ogg's father is making a classic error of logic. Not having found proof that something really happens, is not the same as having definitive proof that this thing cannot possibly happen.

Ogg's family doesn't have the benefit of Aristotle's explanation of deductive reasoning. But even two thousand years after Aristotle got started, we still often fall into this trap.

In evidence-based medicine, a part of this problem is touched on by the saying, "absence of evidence is not evidence of absence." A study says "there's no evidence" of a positive effect, and people jump to the conclusion - "it doesn't work." Baby Ogg gets thrown out with the bathwater.

The same thing is happening when there are no statistically significant serious adverse effects reported, and people infer from that, "it's safe." 

This situation is the opposite of the problem of reading too much into a finding of statistical significance (covered here in Statistically Funny). Only in this case, people are over-interpreting non-significance. Maybe the researchers simply didn't study enough of the right people, or they weren't looking at the outcomes that later turn out to be critical.

Researchers themselves can over-interpret negative results. Or they might phrase their conclusions carelessly. Even if they avoid the language pitfalls here, journalists could miss the nuance (or think the researchers are just being wishy-washy) and spread the wrong message. And even if everyone else phrased it carefully, the reader might jump to that conclusion anyway.

When researchers say "there is no evidence that...", they generally mean they didn't find any, or enough of, a particular type of evidence that they would find convincing. Obviously, no one can ever be sure they have even seen all the evidence. And it doesn't mean everyone would agree with their conclusion, either. To be reasonably sure of a negative, you might need quite a lot of evidence.

On the other hand, the probability of something being extremely unlikely to be real based on quite a lot of knowledge - that there's a community of giant blue swans with orange and pink polka dots on the Nile, say - increases the confidence you might have in even a small study exploring that hypothesis.

Which brings us to the other side of this coin. Proving that something doesn't exist to the satisfaction of people who perhaps need to believe it most earnestly, can be quite impossible. People trying to disprove the claim that vaccination causes autism, for example, are finding that despite the Enlightenment, our rational side can be vulnerable to highjacking. Voltaire hit that nail on the head in the 18th century: "The interest I have to believe a thing is no proof that such a thing exists."