The replication crisis has become fairly mainstream knowledge now in 2021. There have been many “debunkings” of social science research results, including some very famous ones. Here is a not-so-short list for just the field of psychology alone. Videos talking about them often reach millions of views, like this one. But what does debunking, or failure to replicate, really mean in the social sciences?

It may be natural to think that a debunking disproves the original result. If a study initially concluded that “X is good”, the subsequent failure to replicate may give us the idea that “X is bad”. But this is making the classic error of treating the absence of evidence as evidence of absence. What the failed replication really tells us is simply that “we don’t know if X is good or bad”, no more, no less. Maybe the attempt to replicate was flawed, and the debunking study will later be itself debunked. Or maybe both the original and replicating studies are flawed in their own ways. The point is, failures to replicate generally do not flip results, they merely reset them (back to we don’t know).

Then there’s the question of why the replication crisis primarily only hit the social sciences, leaving the natural sciences unscathed for the most part. Yes, there are widespread incentive issues in academia misaligning what research is supposed to do with what researchers want/need to do. And the peer review system, like all complex systems, isn’t really doing what it’s designed to do. But these are all problems with research in general, not just limited to the social sciences. Maybe the standard of work is lower in the social sciences. Or maybe the nature of the field just makes things harder to study, since any social science has to deal with people, who are notoriously unreliable. But I think there is something more fundamental here, which goes to the heart of the principle of reproducibility in the scientific method: the lack of uniformitarianism in the social sciences.

Also known as the Doctrine of Uniformity, uniformitarianism is the (often implicit) fundamental postulate upon which all scientific inquiries are made. It essentially states that the laws and processes which we observe in our current world are the same (or uniform) across both space and time. That means the laws of electromagnetism, planetary motion, quantum mechanics etc…, are the same whether you are on Earth or Mars, and have been the same back in ancient Egypt or millennia into the future.

Reproducibility, one of the key tenets the scientific method, relies on the uniformitarian principle. Yet this unprovable and unverifiable principle does not really hold in the domain of social science, across both dimensions. Spatially, things that are true for one population may be false for another. Temporally, things that are true for a group at a certain time period may no longer hold as time passes, since people and culture change over time. In a world without the Doctrine of Uniformity, the effectiveness of replication as a test of validity diminishes dramatically. Perhaps the best we can do is to hold lower default epistemic confidence for the social sciences, for both original research and debunks.