Let's talk about bias!!!

[Quick]

The most well known are probably blinds.  So lets say you want to account for observer expectancy bias?  This is what happens when you subconsciously influence participants in a study.  You shake your head...you motion with your hands.  You roll your eyes.  You glance sideways at your research partner.  

A screen will cover any visual ques.  If the people on either side of the screen are unaware of what you're looking for (or what they're looking at), we call that a double blind.  It becomes increasingly difficult to project your own biases through successive blinds.

---

That may be true, but depending on the branch of psychology, it is not always a given that these methods are incorporated to the extent that you are inferring. Because of this, it leaves a lot of the empirical evidence to represent personality, intelligence and bias wanting. This is also in the realm of what we consider "verifiable" metrics such as Big 5, IQ, and MMPI. In short, we know what we want to measure but there is still a whole bunch of things in our psyche that we have no idea about. Why this matters is that we should first have a good idea of the capabilities and temperaments of individuals who are conducting these other experiments to make sure people are not lying out their ass. Furthermore, with enough confirmation bias, this can overcome something even as foolproof as a double blind experiment if the right environmental triggers are present for the scientists to collaborate to skew the results. Not saying I have evidence of this happening, but just that it is still entirely possible.

Sure, sure..and to the extent that a study is well controlled it is a more credible study than one that isn;t.  To the rest...you don;t have to know everything to know something.  

As far as confirmation bias is concerned..theres a protocol for that too.  State your assumptions early, clearly, completely, and accurately.  Then, proceed to include all data collected..not just the data that confirms your assumptions.  

While scientific fraud is possible, the peer review system seeks to produce a situation in which the number of participants in the fraud would have to be impractically large, and all of their effort can only be equal to or lesser than a single counter demonstration.  

Let me put that into context.   If all 6.9 million scientists and engineers in the united states conspired together to advance the theory that my morning shit causes the sun to rise..... it would only take me observing myself taking a single shit at 4 am to falsify the entire edifice. I would demand all of the nobel prizes for everything.

Interesting. Very interesting.

Regarding peer review.. In some fields, peer review is susceptible to an echo chamber. Not all fields as much as others, and I would say the hard sciences are much less prone to this for both practical and psychological reasons.