Here is a crash course in how to wade through the headlines in a not-so-foolproof list of steps.
A) When you see “study says” or “a new study…” in the headlines, remember that it is most likely one of several studies and likely not a definitive result.
B) Note the news source you are using. If it is on Fox News, Huffington Post, or MSNBC, the probability is high (but not 100%) that the results might be misrepresented.
C) Google or find a link to the original article and read the abstract (at least!). It’s one to two paragraphs but can definitely tell you a lot. Note the place it was published.
D) Ask the following questions:
Is this theory or empirical? If it is empirical, move on to the next questions. If it is theory, ask for data, because it may be nothing more than conjecture with little application to the real world.
Who is being studied, and how were they found? Was it a random sample? Were they selected from a group?
I can’t tell you how many times people believe a study based on people who volunteered for a study that compares them to a group that did not volunteer. If you see this, be extremely careful.
If you had all the power in the world, how would you measure what they are trying to measure? How far is their experiment or design from what you just answered?
Very often in social science, you can’t actually run an experiment on a human being. What we are left with has to be approached with statistics, clever comparisons between existing groups, or comparisons before and after policy changes, among other tools. Randomized control trials are used in medicine, but can’t often be used in economics. How far from a randomized trial do the authors stray?
Where are the data coming from? Are they collected in a survey, an experiment, or from real-world transaction or outcome data?
It’s important to remember that self-reported data have a margin of error. People may have reasons to change their answers based on what they think is socially desirable or what they think the person doing the study wants to hear, or they may not even know how to answer. For example, if I asked you how much your employer actually pays for your health insurance premiums, most of us might not know, so our answers might be all over the place.
What are the assumptions they are making about human nature, behavior, and action?
Remember, psychologists, sociologists, political scientists, and economists all examine a question differently with different tools, and some assumptions and tools may be inappropriate for the question at hand.
Are there factors that they did not take into account that confound their results?
It is impossible to express how frustrating it is to get into a conversation with someone who claims the “gender pay gap” is represented by the 77 to 100 ratio that is so often mentioned. What other factors go into how much someone is paid? Education, work experience, time taken off, life stability, etc… all may be connected to gender in ways that play into this gap. Without “controlling” for these factors, we can’t actually know what differences in pay are attributable to gender and which are connected to other factors.
Are their findings generalizable, or are they confined to the particular people studied?
E) Remember that even Harvard or Cambridge educated social scientists can get it horribly wrong if they aren’t careful with their assumptions or their statistics. Our understanding of human behavior is too important to defer to “proof by ‘I have a PhD’.”
F) Begin thinking about the policy implications for the result you are seeing. If the results are believable after asking the above questions, you may have to ask yourself what the results mean for your policy preferences. How does the research contribute to your understanding of the world?
G) How can this information make the world a better place? Knowledge in and of itself is interesting, but very few social scientists work because they only want knowledge. They want to see improvement in the human condition. They want knowledge to turn into wisdom.
Follow these steps, and you are on your way to bring a truly critical consumer of information. So whether the source is PBS, CBS, or TBS; Vox, or Fox–put on your thinkin’ caps!