How many times do you hear the words, “studies show….”. It’s amazing how the minute these magic words are uttered, we lie supine to whatever statistic or generalization follows. Studies are vital to gaining wisdom because, assuming they are done properly, they provide us with valid and reliable generalizations upon which we can base our individual choices and public policy. Without generalizations, all you have are platitudes. Scientific generalizations however, require more than just validity and reliability. The extent to which any particular study is useful to a society is largely dependent upon its audience. Studies must be scrutinized by a very logical and unbiased mind. It takes a mind that is open to the premise yet maintains a bit of healthy skepticism. But how do we create such minds?
I remember being taught the difference between fact and opinion in elementary school. It was a simple concept to digest for my gradeschool
mind. According to my 8-year-old daughter, “A fact you can prove and opinion is just something someone thinks.” I listen to him with pride yet am terrified to know that if his formal education goes anything like mine did, that is the extent to which he will be taught about how to find truth in the world. From here on out, he will be subjected to various studies and lead to take them at face value. Sure college professors talk about challenging what you think you know to be true but that with which they wish you to challenge your common sense is most likely the newest study, praised for it’s unlikely and counter-intuitive conclusions. We are intoxicated with the esteem these studies seem to bring to their authors and seek to push the envelope and defy our own innate wisdom in search for novelty.
Imagine you are a member of some steering committee for a big-time HMO. You are on a panel of fellow MDs discussing the screening criteria for STDs in pediatrics. The factors to consider are as numerous as any clinical initiative can be; the confidentiality of the patient, the cost for screening, the emotional aspect of both the patient and their parents, the method of reporting, etc. As you check your watch to see that lunch is 15 minutes away, your stomach rumbles. You glance back up across the table and see that a colleague has this 8 page study in hand. She makes the following claim as she slams it down on the table, “If we don’t screen everybody, we could miss 10% of our patients due to inaccurate self reporting on sexual behavior.” “Wow! 10% is pretty substantial”, another doctor says. You pick up the study and skim through it to read in the conclusion;
“More than 10% of young adults with a laboratory-confirmed positiveSTD result reported abstaining from sexual intercourse in the12 months before assessment and STD testing.”
You break for lunch and take a copy of the study to review for yourself. The first thing you want to know is the total number of cases. You look and find that the study was done with 14,012 pediatric patients. Next you check to see that the researchers have controlled for age, race, socio-economics, etc. They seem to have all their bases covered. Then something occurs to you. The claim stated that 10% of young adults with a laboratory-confirmed positive STD result reported no sexual intercourse in the 12 months before the study. So what is the percentage of laboratory-confirmed positives to the complete sample?
946 tested positive for STDs.
946/14,012=0.07= 07% of the subjects tested positive
Now, of those who tested positive, the study finds that 10% falsely reported their sexual history. Since the entire reason for reviewing this study is to see how many patients will be missed if your organization does not establish a policy of screening ALL pediatric patients for STDs you perform the following calculation:
964 tested positive
10% of 964 = 118
118/14,012 = 0.8%
Lunch is over and you go back in to the meeting and ask the budget committee if you are willing to take the resources from other clinical operations to screen all patients for STDs so that you will not miss 0.8% of the patients due to inaccurate self reporting.
The responsibility of the reader
There is a great burden of responsibility placed upon the reader of academic research. One must realize that research is not free and is funded by any number of entities and interests. Those who write the studies are usually paid professionals who have a masterful talent for reporting data to lead the reader to a desired conclusion. There is no way of knowing if any of this was at work with the study offered in this example however if the reader hadn’t asked those other important questions regarding the true implications of the study, the hospital may have taken resources from a worthwhile cause to chase down 0.8% of the pediatric population. This may or may not be the right decision but it must be made on the basis of the correct figures.
To conclude, I’ll quote one of my favorite thinkers:
“Studies either confirm common sense, or they’re wrong” – Dennis Prager
- Is Psychology About to Come Undone? (chronicle.com)
- The problem with preclinical research? Or: A former pharma exec discovers the nature of science and is amazed (sciencebasedmedicine.org)
- Double-Checking Our Science (thesituationist.wordpress.com)
- Slate Misleading Parents and the Public (tvsmarter.wordpress.com)
- Accreditation and the Tyranny of the Social Sciences (gratiaetnatura.wordpress.com)
- New study indicates mean people are born that way (fox6now.com)
- We have far too much faith in “science” (ourdinnertable.wordpress.com)