More on the "Truth Wearing Off" and my advice to epidemiologists of all ages

Andrew Gelman, a Professor of Statistics at Columbia, has a new post discussing the New Yorker article I mentioned last week.  I highly recommend that you look at the the article that he wrote in American Scientist discussing the statistical challenges in estimating small effects.

My favorite passage: "Statistical power refers to the probability that a study will find a statistically significant effect if one is actually present. For a given true effect size, studies with larger samples have more power. As we have discussed here, “underpowered” studies are unlikely to reach statistical significance and, perhaps more importantly, they drastically overestimate effect size estimates. Simply put, the noise is stronger than the signal."

Thus, when small 'underpowered' studies actually find an effect, it has to be a very large effect to reach statistical significance. So, small studies report overestimates of the effect.

One thing we know about before-after, quasi-experimental studies, which are commonly used in assessing infection prevention interventions, is that they are underpowered and over-estimate the effect compared to randomized trials.  Power is derived from sample size, effect size AND study design, among other things.

QE studies in our field also suffer from publication bias since many have been completed by clinicians who won't go through the trouble of reporting negative studies.  How many papers have you read in ICHE/AJIC/CID that mentioned ADI for MRSA not working?  Even if ADI for MRSA is the greatest control measure ever, which it might be, given a normal distribution of benefit, you would expect some studies to be negative, would you not?  Where are they?

Even, when negative studies do appear (e.g. Harbarth JAMA 2008 or Charlie Huskins hopefully soon to be published STAR-ICU trial) they are often not believed or even thought to be flawed!  Why? Nothing works 100% of the time and a negative study is NOT an erroneous result. A negative study is certainly not prima facie evidence of a flawed study. You must use all of the data, assess it based on quality and power and look for publication bias. (this is my advice to epidemiologists of all ages)

So, are we over-estimating the benefits of ADI and other interventions used in infection prevention?

Link: Gelman's post

Gelman and Weakliem American Scientist, 2009 (PDF)

Where did all of the significant findings go?

There is a really interesting piece in the New Yorker (Dec 13, 2010).  Worth tracking down a copy at your neighbors or dentist's office since the free online version is limited to the abstract.  Jonah Lerher describes in The Truth Wears Off, that initial studies often report large benefits from treatments or large associations between a disease and a specific risk factor which then can't be validated in future studies. 

There are many potential reasons for this 'decline effect' including publication bias - only publishing positive findings, especially in major or high-impact journals.  Dan had a nice post discussing positive outcome bias a couple weeks ago. Another issue might be selective reporting of results by investigators desperate to find strong associations so that they can get published and then get re-funded.  Certainly regression to the mean is important - ye olde bell-shaped curve.  One thing they don't mention is confirmation bias, which I think drives both NIH funding and publication decisions and could be responsible for some of the reduced effect sizes seen in later vs. earlier publications.

This 'decline effect' is troubling given what it says about the scientific process.  One wonders if changes in how science is funded and reported could impact this?

Conflicts of interest are not always financial. Role up your sleeves...and get your flu vaccine?

I joined this blog in December 2009.  There were several reasons for this including what I saw was a great need for conversation among hospital epidemiologists and infection preventionists around complex and important issues such as N95 masks in 2009 novel H1N1, ADI for MRSA and mandatory influenza vaccination of HCW.  That's why I was so excited to see that SHEA was finally releasing a position paper endorsing "a policy in which annual influenza vaccination is a condition of both initial and continued HCP employment and/or professional privileges."  I was so excited in fact, that I haven't read the document.  To be fair to myself, I've been pretty busy and I've seen several talks and debates on the issue. However, I was excited because it would awaken the debate again within the medical community and, more importantly, this blog.

Dan, Mike and I have all commented on the conflict of interest issue. It is true that several authors of the SHEA position paper have financial ties to vaccine manufacturers.  I agree, as I probably said 1000 times during high school debates, that 'perception is key' and that ideally we could produce documents such as this SHEA paper free of financial conflicts.  However, given that this document has been produced, what can and should we do with it?  How will we interpret it in the light of other organizations (e.g. AAP) coming to the same conclusion? Additionally, to be fair to the document and process, this isn't purely a SHEA position paper, but rather it was "approved by the Board of the Society of Healthcare Epidemiology of America and endorsed by the Infectious Diseases Society of America."

You could argue that we should post or even eliminate all of the potential "financial" conflicts of all of the board members of these societies and I would, at first pass, agree with you.  So now, in the future, we might have all board members and all guideline writers be free of all financial conflicts of interest. That may be theoretically possible for one subject, but all subjects? I doubt it.  I also don't think that "financial" conflicts of interest are the most important or result in the most bias.  I think it's not even close; but more on that in a minute.

Now if you think I've gone off my rocker (again), well, I was lead author on a SHEA guideline back in 2007 titled "Raising standards while watching the bottom line : Making a business case for infection control," and that guideline was conceived of, supported, edited, modified and approved by many members of the SHEA board.  In my opinion, a majority of those on the SHEA board must have pushed for this new flu vax position paper knowing what the outcome would be.  In fact, didn't SHEA produce a 2005 position paper containing a different recommendation? So why go through the effort to produce another document so soon, if they didn't know it would produce a different or this exact recommendation? Importantly, the SHEA position paper adheres to all current guidelines and lists financial conflicts of the authors. Nothing is perfect, but I don't think we should discount the recommendations for that reason.

On to another subject.  In April and June of this year I wrote posts discussing what I see is the most important bias in science and in life. That bias is confirmation bias.  My first post on the subject discussed how all of us that have pre-specified opinions, especially ones that are well known to others, root for results of new trials to support our pre-existing beliefs. It's just natural. This tendency for people to favor information that confirms preconceptions regardless of whether the information is true can influence our search for information, how we interpret information and even our memory.  Now, I'm not sure a definitive study has been done, but I suspect that if you have stated a strong public opinion for or against a certain "thing" it would take a lot of money to get you to change your mind and I haven't even mentioned status quo bias. Cognitive biases....if only the solution was so simple as listing or eliminating financial relationships!

While I'm on the subject of definitive studies, I will first state that I have great respect for the Cochrane reviewers and the SHEA position paper authors (and of course my co-bloggers).  However, no amount of genius can overcome the lack of studies/data/funding that exists for the evaluation of infection prevention interventions. So, again, even though I've not read the SHEA paper or the Cochrane reviews, I can definitively say that they are both wrong. Why? No one has completed the necessary cluster-randomized trial in 50-100 hospitals during different influenza seasons with different vaccine-virus matches in different countries with different acuity levels of the hospitalized patient populations etc, etc, etc.  No one will.

To me, the key issue around mandatory vaccine for HCW is not whether the vaccine works, as Mike discussed on Saturday. Rather, it is how much better HCW compliance would be under a mandate. I think most can agree that mandates greatly increase vaccine compliance, but if the data suggests that the vaccine doesn't work, then the question shouldn't be whether or not to mandate the vaccine. The question should really be whether we even offer it to HCW at all, or less seriously, even bother tracking compliance.  I think Mike's post or rather the Cochrane reviews have far more serious implications that stretch way beyond HCW mandates. To me though, there is enough data to support the efficacy and safety of influenza vaccine both in direct protection and also herd immunity. Thus, I think the key issue is compliance; but again, I haven't read it (yet).

So, what would I have done if asked to determine the benefits of mandatory influenza vaccine in HCW?  I might have completed a different type of research synthesis, altogether. I could have taken data like Mark Loeb's 2010 JAMA paper showing the benefits of herd immunity imparted on the unvaccinated by vaccinating children in small rural communities in Canada. Then, I'd have built a decision-analytic type model accounting for the non-linearity of influenza transmission in hospitals, adjusted for various levels of HCW vaccination compliance, completed numerous sensitivity analyses and then reported in which hospitals, in which countries and in which influenza seasons (H3 vs H1) we would expect influenza mandates to be most effective. Too bad that's not gonna happen.  Oh, and people wouldn't believe the model anyway.  It's just math for goodness sake and nobody trusts equations. No, most of us would much rather put our faith in conflicted human beings.  Go figure.

No links today; gotta spend time reading SHEA's new position paper

Confirmation Bias and Science as a Contact Sport

Just getting back from APIC - New Orleans. I've included this photo from Cafe Du Monde, where I broke "bread" with Titus Daniels, Tom Talbot and Keith Kaye. Thought I should avoid photos of people ingesting the beignets, especially those three guys.

While trapped in the Detroit airport (a freak storm flooded the gates and shut down the airport for a couple hours), I came across an interesting article in Ars Technica; a site normally associated with reviews of iPhones and Windows 7. The author Chris Lee discusses confirmation bias, a tendency for people to only look for data that confirms their prior beliefs, which permeates all of science and pseudo-science. He provides a number of nice historical examples of confirmation bias. I particularly enjoyed the end of the article where he describes what steps scientists can take to avoid confirmation bias including subjecting their work to as much internal and external criticism as possible. The "denier" section was good too. Can any of you think of examples of confirmation bias in infection prevention? Me neither.

Ars Technica article: (link)