What testing do science based facts get through to be validated?

[The Grand Nudger]

I can sell you a tabletop fusion device, if you're interested.

[Gawdzilla Sama]

Given that the claims include the ability to actually use it both of you fall short. Tsk tsk.

[The Grand Nudger]

You can throw them at cats.

[John 6IX Breezy]

Your impression is maybe true for individual studies no-one bothers to redo for a while, but things which enter the scientific consensus of a field are usually vetted. I can certainly say that with confidence for my own field. Social psychology has had some troubles with some studies regarded as classics being problematic, but I'm not the person to defend social psychology here, I'll gladly defend physics and related areas which are extremely rigorous.

But being vetted is not the same as being replicated, correct? Most journals do require a reviewing process before publication. But my understanding of that process is that regardless of how rigorous (and sometimes flawed) that process is, the reviewers are doing everything except replicating the experiment, right?

[Alex K]

Your impression is maybe true for individual studies no-one bothers to redo for a while, but things which enter the scientific consensus of a field are usually vetted. I can certainly say that with confidence for my own field. Social psychology has had some troubles with some studies regarded as classics being problematic, but I'm not the person to defend social psychology here, I'll gladly defend physics and related areas which are extremely rigorous.

But being vetted is not the same as being replicated, correct? Most journals do require a reviewing process before publication. But my understanding of that process is that regardless of how rigorous (and sometimes flawed) that process is, the reviewers are doing everything except replicating the experiment, right?

Yes, having something peer reviewed before publication is really just a minimal quality check, not replication. But results which have merely been peer reviewed but not replicated will usually not be universally viewed as canonically accepted results by the community. When other groups try to use a result as a basis for their further work, they will notice whether it works. For example, if Charpentier and Doudna had messed up their CRISPR Cas9 research, thousands of people would have noticed by now.because the method is used daily.

[John 6IX Breezy]

Yes, having something peer reviewed before publication is really just a minimal quality check, not replication. But results which have merely been peer reviewed but not replicated will usually not be universally viewed as canonically accepted results by the community. When other groups try to use a result as a basis for their further work, they will notice whether it works. For example, if Charpentier and Dudna had messed up their CRISPR Cas9 research, thousands of people would have noticed by now.because the method is used daily.

I agree there could be an indirect form of testing when attempting to use the results to further other research. But I can see two problems that arise if we lean too much on this. First, is that a paper's reliability becomes a measure of its popularity; meaning that papers that have been cited the most are the only ones that give us some confidence that they are reliable, whereas the vast majority of papers that have only been cited a handful of times cannot be seen as reliable. And I don't think scientists look to see how popular a paper is before they use it. Secondly, future research rarely hinges entirely on a single piece of previous work; they often review the entire literature on a subject and gather all the relevant papers in support of their own research project. This means that if their experiment is unsuccessful, you have as many variables to blame for the failure as you do papers in your reference section, and it is not immediately obvious which one is responsible (not to mention the fault could be with your own experiment design).

[The Grand Nudger]

There's alot of good work in determining why something we expected to work fails, and in assessing a given experiments design, as well.

[Alex K]

Yes, having something peer reviewed before publication is really just a minimal quality check, not replication. But results which have merely been peer reviewed but not replicated will usually not be universally viewed as canonically accepted results by the community. When other groups try to use a result as a basis for their further work, they will notice whether it works. For example, if Charpentier and Dudna had messed up their CRISPR Cas9 research, thousands of people would have noticed by now.because the method is used daily.

I agree there could be an indirect form of testing when attempting to use the results to further other research. But I can see two problems that arise if we lean too much on this. First, is that a paper's reliability becomes a measure of its popularity; meaning that papers that have been cited the most are the only ones that give us some confidence that they are reliable, whereas the vast majority of papers that have only been cited a handful of times cannot be seen as reliable. And I don't think scientists look to see how popular a paper is before they use it. Secondly, future research rarely hinges entirely on a single piece of previous work; they often review the entire literature on a subject and gather all the relevant papers in support of their own research project. This means that if their experiment is unsuccessful, you have as many variables to blame for the failure as you do papers in your reference section, and it is not immediately obvious which one is responsible (not to mention the fault could be with your own experiment design).

Well, finding out why, that is exactly what doing research is all about. Many a grad student will spend years wading through all relevant factors to determine where something goes awry. The only problem is that there is a bias against publishing negative studies which can bias the significance of published literature. That's why there are efforts to make pre-registration of studies mandatory.