There are not many moments in which social science is self-policing. Sure, journal articles have referees that attempt to spot problems. “Revise and resubmit” is very common.
But not all journals are of the same quality, and some social sciences train their pupils better in areas of technical expertise and reasoning. Each field has their own approach, and some are more appropriate or more comprehensive than others.
Most arguments in popular media that report on social science, when there is a study released that has results that upend a particular hyper-partisan position, simply appeal to authority in saying, “So-and-so from such-and-such university/institution/think tank says that there are serious flaws in the study.” Very rarely do we actually get to talk about what those flaws are, which we need to start doing.
That’s why it is so refreshing to see major media outlets beginning to talk openly about the issues in social science reporting, even between Old Media giants: the Washington Post and the New York Times.
Washington Post: Making time for kids? Study says quality trumps quantity.
New York Times (Justin Wolfers): Yes, Your time as a parent does make a difference.
Particularly, Wolfers gets into the metrics:
“This nonfinding largely reflects the failure of the authors to accurately measure parental input. In particular, the study does not measure how much time parents typically spend with their children. Instead, it measures how much time each parent spends with children on only two particular days — one a weekday and the other a weekend day.”
Measurements matter. Especially in something that affects children, what we can and cannot say about behavior and outcomes REALLY matters.
For an added bonus, I recommend this Freakonomics podcast on parenting, though I do disagree with Wolfers’ and Stevenson’s views on marriage.