How studies are misleading

Cover of "Bad Science"

Cover of Bad Science


How many times do you hear the words, “studies show….”.  It’s amazing how the minute these magic words are uttered, we lie supine to whatever statistic or generalization follows.   Studies are vital to gaining wisdom because, assuming they are done properly, they provide us with valid and reliable generalizations upon which we can base our individual choices and public policy.  Without generalizations, all you have are platitudes.  Scientific generalizations however, require more than just validity and  reliability.  The extent to which any particular study is useful to a society is largely dependent upon its audience.  Studies must be scrutinized by a very logical and unbiased mind.  It takes a mind that is open to the premise yet maintains a bit of healthy skepticism.  But how do we create such minds?

I remember being taught the difference between fact and opinion in elementary school.  It was a simple concept to digest for my gradeschool

mind.  According to my 8-year-old daughter, “A fact you can prove and opinion is just something someone thinks.”  I listen to him with pride yet am terrified to know that if his formal education goes anything like mine did, that is the extent to which he will be taught about how to find truth in the world.  From here on out, he will be subjected to various studies and lead to take them at face value.  Sure college professors talk about challenging what you think you know to be true but that with which they wish you to challenge your  common sense is most likely the newest study, praised for it’s unlikely and counter-intuitive conclusions.  We are intoxicated with the esteem these studies seem to bring to their authors and seek to push the envelope and defy our own innate wisdom in search for novelty.

Clinical Practice

Imagine you are a member of some steering committee for a big-time HMO.  You are on a panel of fellow MDs discussing the screening criteria for STDs in pediatrics.  The factors to consider are as numerous as any clinical initiative can be; the confidentiality of the patient, the cost for screening, the emotional aspect of both the patient and their parents, the method of reporting, etc.  As you check your watch to see that lunch is 15 minutes away, your stomach rumbles.  You glance back up across the table and see that a colleague has this 8 page study in hand.  She makes the following claim as she slams it down on the table, “If we don’t screen everybody, we could miss 10% of our patients due to inaccurate self reporting on sexual behavior.”  “Wow! 10% is pretty substantial”, another doctor says.  You pick up the study and skim through it to read in the conclusion;

“More than 10% of young adults with a laboratory-confirmed positiveSTD result reported abstaining from sexual intercourse in the12 months before assessment and STD testing.”
You break for lunch and take a copy of the study to review for yourself.  The first thing you want to know is the total number of cases.  You look and find that the study was done with 14,012 pediatric patients.  Next you check to see that the researchers have controlled for age, race, socio-economics, etc.  They seem to have all their bases covered.  Then something occurs to you.  The claim stated that 10% of young adults with a laboratory-confirmed positive STD result reported no sexual intercourse in the 12 months before the study.  So what is the percentage of laboratory-confirmed positives to the complete sample?


946 tested positive for STDs.

946/14,012=0.07=    07% of the subjects tested positive

Now, of those who tested positive, the study finds that 10% falsely reported their sexual history.  Since the entire reason for reviewing this study is to see how many patients will be missed if your organization does not establish a policy of screening ALL pediatric patients for STDs you perform the following calculation:

964 tested positive

10% of 964 = 118

118/14,012 = 0.8%

Lunch is over and you go back in to the meeting and ask the budget committee if you are willing to take the resources from other clinical operations to screen all patients for STDs so that you will not miss 0.8% of the patients due to inaccurate self reporting.

The responsibility of the reader

There is a great burden of responsibility placed upon the reader of academic research.  One must realize that research is not free and is funded by any number of entities and interests.  Those who write the studies are usually paid professionals who have a masterful talent for reporting data to lead the reader to a desired conclusion.  There is no way of knowing if any of this was at work with the study offered in this example however if the reader hadn’t asked those other important questions regarding the true implications of the study, the hospital may have taken resources from a worthwhile cause to chase down 0.8% of the pediatric population.  This may or may not be the right decision but it must be made on the basis of the correct figures.

To conclude, I’ll quote one of my favorite thinkers:

“Studies either confirm common sense, or they’re wrong”   – Dennis Prager

One thought on “How studies are misleading

  1. Some studies are also inherently flawed because of the protocols followed. Then there are those that are not done objectively – these are done with a particular outcome in mind, usually funded by corporations with a vested interest (eg. Monsanto, pharma companies etc). Data has been known to be deliberately manipulated, omitted, or the protocols were set up to for a particular outcome (such as some of the studies “proving” that GMOs are safe). The lipid hypothesis that spawned the whole low-fat movement was also born out of one of these. This is just plain bad science.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s