I know not everyone wants to watch a 10 minute video, so I'll give the tl;dr below. But it is worth watching if you have a little time to spare.

***

In 2014 and 2015, studies were published which estimated that only between 46-85% of psychological studies were replicable. This is a serious problem.

The video presents this thought experiment: you are given a box of 100 thermometers but you know that a significant proportion of them is inaccurate. If only 2% of them are inaccurate, this is hardly a problem. But, if this number is 50% the odds of you getting an accurate reading is no better than a coin flip. That is a serious issue, since we rely in psychological science to furnish us with information that impacts, not only mental health treatment, but also public spending to address societal issues.

Some suppose that the issue stems from publication bias. In fact, this is the most oft cited culprit in discussions of the replicability issue. From wikipedia: "In published academic research, publication bias occurs when the outcome of an experiment or research study biases the decision to publish or otherwise distribute it. Publishing only results that show a significant finding disturbs the balance of findings in favor of positive results."

Of course, as human beings interested in expanding our knowledge, a bias towards new, significant discoveries is understandable. But, IMO, furthering our knowledge is not the only goal of science. There is a skeptical or "let's make sure" spirit that is also essential to all sciences. To me, the replication issue in science isn't just a problem with psychological science. To me, it demonstrates that our desire for new knowledge has over-extended itself... to the extent that (paradoxically) we eagerly publish falsities and turn down publication of more accurate studies on account of "you didn't discover anything new."

One solution to the problem of publication bias (which I personally like) is that we weight studies that disprove other studies as much as we weight those which genuinely discover new knowledge. The end result of such a valuation would be that any and all genuine discoveries that we make in a given field, will stand on more solid ground.

Any thoughts on this? And what are your thoughts on publication bias in science generally? (Not just in psychology.)

It seems that not being able to replicate results should, long term, make this a self-solving issue; if a particular finding can’t be replicated, it should have a winnowing effect on bad science. Any professional journal that publishes a non-replicable study should be required to repudiate that paper within a specified time frame (two years sounds about right). Something along the lines of:

‘In our April 2020 issue, we published the work of Dr. Emilio Hungaduna, et al (‘The Lachrymotic Effects Of Downward Tropism In 24 Months’ Post Neonates’). Since then, several other research teams have either refuted or failed to replicate Hungadunga’s results. To that end, this journal no longer supports the conclusions of Dr. Hungaduna’s team.’

I agree that publication bias (in all fields) is a problem. Negative results are still results.

Boru

The video is confusing hard science with soft science. Hard science could easily tell which thermometers are inaccurate. Soft science (of which psychology is one, google soft science) can't easily make that distinction and it is more dependent on fluctuating societal norms than actual repeatable hard science.

Repeatable results in psychology are more nebulous than other scientific disciplines. 

https://en.wikipedia.org/wiki/Hard_and_soft_science

Publication bias exists in both hard and soft science. Eventually they both work it out or self correct.

I've helped with several studies that took an "okay, we'll just see about that" look at another study.  All in agriculture, animal science, or biomedical engineering.

(April 5, 2022 at 2:33 pm)brewer Wrote: The video is confusing hard science with soft science. Hard science could easily tell which thermometers are inaccurate. Soft science (of which psychology is one, google soft science) can't easily make that distinction and it is more dependent on fluctuating societal norms than actual repeatable hard science.

Repeatable results in psychology are more nebulous than other scientific disciplines. 

https://en.wikipedia.org/wiki/Hard_and_soft_science

Publication bias exists in both hard and soft science. Eventually they both work it out or self correct.

I would say that replicability is equally important for the hard and soft sciences. Like, if it ain't replicable, it ain't science... period. True that a replication is a more nebulous thing in the soft sciences, but I would argue that replication is a key factor of all sciences. 

Even if publication bias "self-corrects" eventually, I still think it deserves our attention. It isn't just one result or one single fact that gets corrected. Researchers depend on scientific papers being accurate. One single erroneous result could impact dozens of studies because (for better or worse) researchers in psychology assume the results in publications are sound.

And that's the thesis of the video. They ought to be more reliable than they are. We could vastly improve on an 85% replicability rate (and it could even be lower than 85%). So I think addressing publication bias in a more immediate way is preferred over waiting for things to work themselves out eventually.

(April 5, 2022 at 4:22 pm)vulcanlogician Wrote:

(April 5, 2022 at 2:33 pm)brewer Wrote: The video is confusing hard science with soft science. Hard science could easily tell which thermometers are inaccurate. Soft science (of which psychology is one, google soft science) can't easily make that distinction and it is more dependent on fluctuating societal norms than actual repeatable hard science.

Repeatable results in psychology are more nebulous than other scientific disciplines. 

https://en.wikipedia.org/wiki/Hard_and_soft_science

Publication bias exists in both hard and soft science. Eventually they both work it out or self correct.

I would say that replicability is equally important for the hard and soft sciences. Like, if it ain't replicable, it ain't science... period. True that a replication is a more nebulous thing in the soft sciences, but I would argue that replication is a key factor of all sciences. 

Even if publication bias "self-corrects" eventually, I still think it deserves our attention. It isn't just one result or one single fact that gets corrected. Researchers depend on scientific papers being accurate. One single erroneous result could impact dozens of studies because (for better or worse) researchers in psychology assume the results in publications are sound.

And that's the thesis of the video. They ought to be more reliable than they are. We could vastly improve on an 85% replicability rate (and it could even be lower than 85%). So I think addressing publication bias in a more immediate way is preferred over waiting for things to work themselves out eventually.

And how do you propose to get a better accuracy/reliability rate? Who should decide what is reliable and what is hogwash? What is reliable on one society/culture will not be reliable in another (consider various Chinese medicine practices). As far as I know review and replication of published studies is the best tool. It might be slow and clunky but I can't think of a better option.

Maybe the integrity of the publishers should be scrutinized closer. I typically limit my psych info to reliable sources but I'm not sure how that would happen with the general population and access to the internet. Posting crap does not usually hurt publisher financially (financial being the driving force in the past). How would anyone stop the woo woo that gets posted and then sent to other's that will buy into the woo woo?

Lets face it, there will always be a certain percent of any population that believes unfounded shit, no mater where it comes from. Do I really need to point at religion or snake oil salesmen as an example?

(April 5, 2022 at 1:42 pm)BrianSoddingBoru4 Wrote: It seems that not being able to replicate results should, long term, make this a self-solving issue; if a particular finding can’t be replicated, it should have a winnowing effect on bad science. Any professional journal that publishes a non-replicable study should be required to repudiate that paper within a specified time frame (two years sounds about right). Something along the lines of:

‘In our April 2020 issue, we published the work of Dr. Emilio Hungaduna, et al (‘The Lachrymotic Effects Of Downward Tropism In 24 Months’ Post Neonates’). Since then, several other research teams have either refuted or failed to replicate Hungadunga’s results. To that end, this journal no longer supports the conclusions of Dr. Hungaduna’s team.’

Or better yet, publish those very studies that contradict the previous results and give those hard-working individuals who bothered to try to replicate the experiment some recognition of their own.

Quote:I agree that publication bias (in all fields) is a problem. Negative results are still results.

Exactly. Perceiving positive results as more significant than negative results stems from our (understandable) human biases and a desire for science to "progress." But this perspective is less objective than one that values negative and positive results equally. I think any scientist would agree with this sentiment. But the existence of publication bias itself reminds us that this principle doesn't fully demonstrate itself in practice.

---

(April 5, 2022 at 5:02 pm)brewer Wrote: And how do you propose to get a better accuracy/reliability rate? Who should decide what is reliable and what is hogwash? What is reliable on one society/culture will not be reliable in another (consider various Chinese medicine practices). As far as I know review and replication  of published studies is the best tool. It might be slow and clunky but I can't think of a better option.

We get a better replication rate by emphasizing replication more than we do. We emphasize new discoveries. Getting published is easier if you have some groundbreaking or eyebrow-raising result. I think the system being oriented that way has made our overall publications less reliable.

A small number of psychologists have even been caught manipulating results to make them more "publishable." The point I'm making is not that fraud is bad. (Of course fraud is bad.) The point I'm making is that it's problematic when we "desire" a certain kind of result that we deem fit for publication. Science ought to be more objective. A result is a result.

The "system" does have its merits, though. When a visionary, like Einstein, publishes a truly groundbreaking idea (Like Special Relativity) they are awarded more prestige and resources so they might go on to make even greater discoveries (like General Relativity). Einstein certainly deserved to be held aloft and given accolades for his work. But we don't want to award groundbreaking work at the expense of credible science. That would unravel the entire idea that makes it all worthwhile in the first place.

Quote:Maybe the integrity of the publishers should be scrutinized closer. I typically limit my psych info to reliable sources but I'm not sure how that would happen with the general population and access to the internet. Posting crap does not usually hurt publisher financially (financial being the driving force in the past). How would anyone stop the woo woo that gets posted and then sent to other's that will buy into the  woo woo?

Lets face it, there will always be a certain percent of any population that believes unfounded shit, no mater where it comes from.   Do I really need to point at religion or snake oil salesmen as an example?

We want to hold our academic journals to a higher standard than what the general public gleans from the internet, don't we? That's my point. Luckily, even at an 85% replicability rate, that puts psychology journals head and shoulders above what passes for fact on the internet. All I'm saying is that 85% is disturbingly low, and it suggests that something in the publication apparatus possibly needs attention and/or adjustment.

I think that for soft science 85% is probably about as good as it gets, to many non replicable variables. Don't claim a 'crisis' where there is not real crisis.

These psychological "professional" type people don't know shit.

(April 5, 2022 at 7:13 pm)Ahriman Wrote: These psychological "professional" type people don't know shit.