(April 26, 2022 at 1:22 pm)John 6IX Breezy Wrote:(April 26, 2022 at 12:30 am)vulcanlogician Wrote: I prefer those psychological studies that are replicable.
One more thing. I'm speaking from ignorance here since my knowledge of p-hacking is limited. But the argument I want to add is that replication comes at a cost—the more you run an experiment the more likely you are to find a false positive or false negative. Statistical tests are built on probabilities, so replication is like rolling a die: The more times you roll it, the more likely you are to land on your desired results. And I'm not sure there's a meaningful difference between a researcher running his own experiment multiple times (p-hacking) and other researchers running your experiment multiple times (replication). Again, I think Bayesian methods can help correct for this because it treats every experiment as informative. But I don't think psychology is currently treating replicability this way. In fact, I think these questions truly belong to the philosophy of science.
I'm not well versed enough in psychological science's methodology to say for sure, but it's my impression that the results are calculated within a disclosed margin of error. Others' results (though they may differ) can fall within this margin and still be considered a replication. That the conclusions set broad parameters to begin with, and are STILL seen to be non-replicable is the issue.
