More on the "Truth Wearing Off" and my advice to epidemiologists of all ages

Andrew Gelman, a Professor of Statistics at Columbia, has a new post discussing the New Yorker article I mentioned last week.  I highly recommend that you look at the the article that he wrote in American Scientist discussing the statistical challenges in estimating small effects.

My favorite passage: "Statistical power refers to the probability that a study will find a statistically significant effect if one is actually present. For a given true effect size, studies with larger samples have more power. As we have discussed here, “underpowered” studies are unlikely to reach statistical significance and, perhaps more importantly, they drastically overestimate effect size estimates. Simply put, the noise is stronger than the signal."

Thus, when small 'underpowered' studies actually find an effect, it has to be a very large effect to reach statistical significance. So, small studies report overestimates of the effect.

One thing we know about before-after, quasi-experimental studies, which are commonly used in assessing infection prevention interventions, is that they are underpowered and over-estimate the effect compared to randomized trials.  Power is derived from sample size, effect size AND study design, among other things.

QE studies in our field also suffer from publication bias since many have been completed by clinicians who won't go through the trouble of reporting negative studies.  How many papers have you read in ICHE/AJIC/CID that mentioned ADI for MRSA not working?  Even if ADI for MRSA is the greatest control measure ever, which it might be, given a normal distribution of benefit, you would expect some studies to be negative, would you not?  Where are they?

Even, when negative studies do appear (e.g. Harbarth JAMA 2008 or Charlie Huskins hopefully soon to be published STAR-ICU trial) they are often not believed or even thought to be flawed!  Why? Nothing works 100% of the time and a negative study is NOT an erroneous result. A negative study is certainly not prima facie evidence of a flawed study. You must use all of the data, assess it based on quality and power and look for publication bias. (this is my advice to epidemiologists of all ages)

So, are we over-estimating the benefits of ADI and other interventions used in infection prevention?

Link: Gelman's post

Gelman and Weakliem American Scientist, 2009 (PDF)

Where did all of the significant findings go?

There is a really interesting piece in the New Yorker (Dec 13, 2010).  Worth tracking down a copy at your neighbors or dentist's office since the free online version is limited to the abstract.  Jonah Lerher describes in The Truth Wears Off, that initial studies often report large benefits from treatments or large associations between a disease and a specific risk factor which then can't be validated in future studies. 

There are many potential reasons for this 'decline effect' including publication bias - only publishing positive findings, especially in major or high-impact journals.  Dan had a nice post discussing positive outcome bias a couple weeks ago. Another issue might be selective reporting of results by investigators desperate to find strong associations so that they can get published and then get re-funded.  Certainly regression to the mean is important - ye olde bell-shaped curve.  One thing they don't mention is confirmation bias, which I think drives both NIH funding and publication decisions and could be responsible for some of the reduced effect sizes seen in later vs. earlier publications.

This 'decline effect' is troubling given what it says about the scientific process.  One wonders if changes in how science is funded and reported could impact this?