Reproducibility: The Final Pilar of Trustworthy Science
This is my final post in a series inspired by Dr. Barker Bausell’s recent book, “Snake Oil Science: The Truth About Complementary and Alternative Medicine.” Since I began this series, the New York Times has published a rave review of Bausell’s book, which only further confirms the importance of Bausell’s contributions.
Although Bausell’s main thesis is that there are currently no large, randomized controlled trials (published in leading medical journals) demonstrating the effect of any CAM therapy beyond placebo, I have chosen to highlight some of his thinking about research methodology as it applies to the medical literature in general.
So far I have explained why most research (if not carefully designed) will lead to a false positive result. This inherent bias is responsible for many of the illusionary treatment benefits that we hear about so commonly through the media (whether they’re reporting about CAM or Western medicine), because it is their job to relay information in an entertaining way more so than an accurate manner (i.e. good science makes bad television).Then I explained a three step process for determining the trustworthiness of health news and research. We can remember these steps with a simple mnemonic: C-P-R.
The C stands for credibility- in other words, “consider the source” – is the research published in a top tier medical journal with a scientifically rigorous review process?
The P stands for plausibility- is the proposed finding consistent with known principles of physics, chemistry, and physiology or would accepting the result require us to suspend belief in everything we’ve learned about science to date?
And finally we arrive at R – reproducibility. If the research study were repeated, would similar results be obtained?
This third and final pillar of trustworthy science is a simple, but sometimes forgotten, principle. If there is a true cause and effect relationship observed by the researcher, then surely that cause and effect can be demonstrated again and again under the same conditions. Touching a hot stove burner always results in a burned hand. No matter how frequently you test this causal relationship, the result will be similar.
Sometimes conflicting results are obtained by repeating a study. When this happens, the reader should be careful in interpreting the conclusions – there may be a flaw in the study design, or it may be that the conclusions drawn were inaccurate. There could have been a false positive result, or no appreciable effect of the treatment under consideration, therefore leaving the results to chance. Flipping a coin gives you heads one minute and tails the next. Yet a person unfamiliar with coins could conclude (after one flip) that it has a head on both sides. In the end, therefore, one can be more confident in a study’s result if it is born out by other studies.
And so as we conclude this series, I hope that you now feel well equipped to perform CPR (credibility, plausibility, reproducibility checks) on health news. A little healthy skepticism can protect your brain from all the mixed health messages that barrage us each day. At the very least, now you’ll appreciate why most health news reports include an expert quote stating something to the effect of “it’s too early to know for sure if these findings are relevant.” That statement may be the most trustworthy of the entire report.
***
Next up: Shannon Brownlee’s book “Overtreated: Why Too Much Medicine Is Making Us Sicker And Poorer.” Shannon and I corresponded about this book two years ago, so I’m looking forward to seeing how it has turned out. Once I’ve finished it I’ll give you my thoughts here in this blog.This post originally appeared on Dr. Val’s blog at RevolutionHealth.com.
Return to article »