Crisis in Psychology?

[Angrboda]

It's interesting where a person puts the line between forensics, anthropology, and sociology. Leaving aside those things in the soft sciences which don't seem to fit the criteria for soft science, the only way to solve the reproducibility crisis with respect to living human beings is to apply benfords law - but there are obvious practical difficulties there - and we still have to acknowledge the fact that living people are inherently duplicitous.  A thermometer expends no effort deluding itself, others, or enforcing a consensus of public opinion and Right Thought.  

Here's a thought experiment.  Early modern human remains are found, male, early 40's, signs of blunt force trauma to the skull and stone fragments between the ribs along the spine which have minor unhealed damage.  Further, the effects of the deceased contain numerous manufactured items.  Dried foods, medicinal herbs, binding material, resins, woven fiber, a number of flints and edges, woodcraft, bone awls and needles.  A small flute.  An axe, a bow, arrows in various states of manufacture from blank to finish.  The mans teeth show signs consistent with being worn down by fine inorganic particles in lightly processed grain, but are relatively free of other damage, cavities, or infection.  Other than the trauma listed above, the man is healthy and shows little evidence of a history of injury or persistent illness beyond having broken his arm once in childhood, and abnormal wear in the ball joints of his knees.  

Forensically - the man was murdered.  He has old injuries that have healed but none beyond (or at) the point at which he suffered blunt force trauma and, likely, several shots to the back.  It doesn't appear to be a robbery, the mans items were left at the scene.  If he died in a war, he was either very lucky or this was the first time he'd ever been in combat - both unlikely given his age.  Given the items he carried, left at the scene, he was likely a courier or independent tradesman.  The shots to the back were unlikely to have killed him, though the would have certainly incapacitated him.  It's unclear whether this was an assault in the immediate sense, or whether he was harried and coursed - but either way, he didn't live long after those shots to the back, as the ribs and spine never scarred over.  In the absence of any other remains or midden heaps or signs of civilization, we can only assume he was at some point between a and b when the worst day of his life happened.  If he was with anyone, they either didn't die there or survived - but either way they also left his items at the scene.  

So there's the scene.  I'd have to write a novel (and novels have been written) to fully describe the anthropological ramifications of a find like that.  We know that the man came from a community or civilization that could..for example, make or source all of those items.  We can put boundaries on the route of the man or reach of his community by cataloging the items in his possession and their pre-historic extent-of-origin.  We can determine the methods of production used to create the manufactured items, and how many man-hours they represent, by direct demonstration.  Similarly, we can determine what and when people in his community or within his reach must have been doing by the same.  If we find processed grains, we know what people were up to in spring summer and fall.  We can see that he must have eaten a relatively alkaline diet by the state of his teeth.  We can see that this period in time was relatively peaceful for this man by reference to the lack of repeat injury.  However, we can see that people were also motivated to murder even then, and not..at least in this case, merely to acquire goods.  He may have been executed, which engenders a whole hell of alot more than a mugging gone wrong.  The again, again...they also played the flute.  

All of this, is soft science, allegedly.

[John 6IX Breezy]

***


@John 6IX Breezy

What's your opinion on this? (Since it's in your field of study....?)

Well here's a question I've wondered about. What is the right conclusion to make once a study doesn't replicate?

Most people, I think, conclude that the first study is flawed and throws everything out. But I can't help but feel there is a temporal or sequential bias there. Would we be inclined to make the same conclusion if the studies came out in reverse order, for example? Meaning, if the study that didn't find any effect came first, and the one that found an effect came after? And then there is perhaps another option, which is to treat both studies as equally valid, as if they came out simultaneously, and then take some average of the two, no different from a meta-analysis. (In fact, I think this is what we would do if both studies came out positive.)

So, it's not immediately clear to me what our response to replication ought to be. And I should add that I think the root of the replication problem is statistics. You find similar issues everywhere that statistics is used, for example, fMRI research. This is particularly interesting because psychology is dominated by statistics, but it wasn't always the case. In fact, I think giants like BF Skinner who witnessed the emergence of statistics saw it as something corrupting the field. He thought our focus should primarily be on the individual, and statistics erases the individual.

ps. The following quote is from my undergrad statistics textbook:

---

"Behaviorism’s most famous advocate, B. F. Skinner, was quite opposed to statistics. Skinner even said, “I would much rather see a graduate student in psychology taking a course in physical chemistry than in statistics. And I would include [before statistics] other sciences, even poetry, music, and art."

Skinner was constantly pointing to the information lost by averaging the results of a number of cases. For instance, Skinner (1956) cited the example of three overeating mice—one naturally obese, one poisoned with gold, and one whose hypothalamus had been altered. Each had a different curve for learning to press a bar for food. If these learning curves had been merged statistically, the result would have represented no actual eating habits of any real mouse. As Skinner said, “These three individual curves contain more information than could probably ever be generated with measures requiring statistical treatment, yet they will be viewed with suspicion by many psychologists because they are single cases” (p. 232).

---

[Jehanne]

ps. The following quote is from my undergrad statistics textbook:

"Behaviorism’s most famous advocate, B. F. Skinner, was quite opposed to statistics. Skinner even said, “I would much rather see a graduate student in psychology taking a course in physical chemistry than in statistics. And I would include [before statistics] other sciences, even poetry, music, and art."

Physical chemistry includes lots and lots of statistics; in fact, such a textbook would likely have a substantial portion of it devoted entirely to statistical analysis, either in the introduction or the appendix (or, more likely, both.)

[vulcanlogician]

Well here's a question I've wondered about. What is the right conclusion to make once a study doesn't replicate?

That the results of the study aren't replicable and shouldn't be regarded as genuine scientific knowledge. Or maybe the replication attempt was flawed... but it's an either/or scenario. Non-replicable results don't allow us to confidently conclude anything. So they are as good, knowledge-wise, as no results whatsoever.

I expect more from psychological science.

Most people, I think, conclude that the first study is flawed and throws everything out. But I can't help but feel there is a temporal or sequential bias there. Would we be inclined to make the same conclusion if the studies came out in reverse order, for example? Meaning, if the study that didn't find any effect came first, and the one that found an effect came after?

If neither study is replicable, then neither one says anything. I prefer those psychological studies that are replicable. Even pessimistic estimates put these at at least around half of psychological studies. The portion of studies that aren't replicable need to achieve the standard set by those that are.

So, it's not immediately clear to me what our response to replication ought to be. And I should add that I think the root of the replication problem is statistics.

I disagree. The problem is the inability of [a certain portion] of these studies to be replicated. The size of that proportion is not the problem. We should only allow a tiny fraction of non-reproducibility (that imposed by chance or dumb luck). We shouldn't make systemic concessions within the science. We need psychological science to be reliable. If the things cited in the OP are true, it isn't.

[John 6IX Breezy]

We need psychological science to be reliable.

Hmmm I think we need a psychological science that's adaptive and individualistic. Yes, maximizing reliability is important. But keep in mind that the object of study isn't consistent the way atoms might be. A single individual is developing throughout the lifespan. And the species as a whole is randomly evolving from generation to generation.

Psychology, in other words, is the study of change and variability. A correlational study, for example, might attempt to explain variability in one distribution (stress) in terms of variability in another distribution (finances). Consider cross-cultural studies. If nothing else, these studies measure the degree to which a finding consistently fails to replicate from one population to the next.

Recently, I've been diving into Bayesian Statistics. I'm still grappling with it conceptually, but I think it's the correct approach here: You update what you believed about the old study with the evidence you uncovered with the new study. It is an adaptive approach. You can't throw out every study that doesn't replicate because they still hold information about the world no matter how noisy it is (assuming they are internally robust).

ps. And what does successful replication do for us if a given study is internally flawed? Replication doesn't protect or uncover deeper problems. It isn't a means of falsification. If your methodology is flawed, replication only perpetuates the flaw even if the results are consistent.

[Belacqua]

A correlational study, for example, might attempt to explain variability in one distribution (stress) in terms of variability in another distribution (finances). Consider cross-cultural studies. If nothing else, these studies measure the degree to which a finding consistently fails to replicate from one population to the next.

Yes, I think a failure to replicate may be an important finding in itself. If nothing else, it reminds us not to be too confident. 

I was imagining a study like the one you mention, which correlates stress against another variable. I can't even imagine how we could quantify stress reliably. I suppose there's a standardized checklist -- "On a daily basis, how often do you imagine your head exploding? Once a day. Twice a day. Every hour. Every waking second." 

But to correlate one person's stress with another person's stress, we'd need pretty much an infinite checklist of possible stressors. I've asked my doctor a few times for anti-anxiety meds, and never once has he asked me how much caffeine I drink in a day, or whether my wife scolds me. Is my anxiety really the same as yours? I don't think science can determine that.

Then there's the cross-cultural element. What we call "anxiety" in English gets translated as 不安 in Japanese. But those characters have their own variables. 不 is easy because it just means "not" or "lack," but 安 can be "safety," "calm," "peacefulness," and other semantically related things. When telling the doc that I have 不安, I very much don't feel that I'm describing what Americans call anxiety. (Personally, unscientifically, I feel 不安 in the pit of my stomach, and anxiety in my shoulders.)

Lately people have imported the word "sutoresu," from "stress." I think this came in via engineering, from stress checks on airplane wings, etc. But there's no guarantee that the nuance has travelled properly. Nor is there any guarantee that effective treatments would be the same.

[John 6IX Breezy]

Yes, I think a failure to replicate may be an important finding in itself. If nothing else, it reminds us not to be too confident. 

Right, in other words, failure to replicate is informative but not conclusive. The next step should be to explain why results varied between experiments just like we would explain why height varies between people. Replication isn't there to uncover questionable research practices—those can and ought to be filtered out prior to publication. So, I'm going to (naively) assume that if a study got published that it successfully passed quality control and its results are meaningful unless otherwise indicated.

As for confidence, this is where Bayesian inference comes in. We can ask what the probability is that a study is wrong given that it failed to replicate. Which takes into account the probability that any study fails to replicate when it is right (false negatives).

Lately people have imported the word "sutoresu," from "stress." I think this came in via engineering, from stress checks on airplane wings, etc. But there's no guarantee that the nuance has travelled properly. Nor is there any guarantee that effective treatments would be the same.

I was interested in phenomenology once because I saw it as a way to answer these questions about experience. People tend to have low-resolution perspectives of their emotions. So I had a personal project once (and still do) in which I construct a formula for categorizing and recognizing my own emotions. For example, I ask myself what the valence of the emotion I'm feeling is (negative). If it originates due to internal or external circumstances (external). What the emotion is wanting me to do (run away). Until I can reliably label each emotion (fear). My goal was to understand what something like jealousy feels like me for me as an individual by deconstructing it.

Maybe its been done before but such an approach would be useful in communicating emotions across individuals. Knowing whether you feel stress in your shoulders or your stomach is important information that shouldn't be lost in the word stress.

[John 6IX Breezy]

 I prefer those psychological studies that are replicable. 

One more thing. I'm speaking from ignorance here since my knowledge of p-hacking is limited. But the argument I want to add is that replication comes at a cost—the more you run an experiment the more likely you are to find a false positive or false negative. Statistical tests are built on probabilities, so replication is like rolling a die: The more times you roll it, the more likely you are to land on your desired results. And I'm not sure there's a meaningful difference between a researcher running his own experiment multiple times (p-hacking) and other researchers running your experiment multiple times (replication). Again, I think Bayesian methods can help correct for this because it treats every experiment as informative. But I don't think psychology is currently treating replicability this way. In fact, I think these questions truly belong to the philosophy of science.

[vulcanlogician]

 I prefer those psychological studies that are replicable. 

One more thing. I'm speaking from ignorance here since my knowledge of p-hacking is limited. But the argument I want to add is that replication comes at a cost—the more you run an experiment the more likely you are to find a false positive or false negative. Statistical tests are built on probabilities, so replication is like rolling a die: The more times you roll it, the more likely you are to land on your desired results. And I'm not sure there's a meaningful difference between a researcher running his own experiment multiple times (p-hacking) and other researchers running your experiment multiple times (replication). Again, I think Bayesian methods can help correct for this because it treats every experiment as informative. But I don't think psychology is currently treating replicability this way. In fact, I think these questions truly belong to the philosophy of science.

I'm not well versed enough in psychological science's methodology to say for sure, but it's my impression that the results are calculated within a disclosed margin of error. Others' results (though they may differ) can fall within this margin and still be considered a replication. That the conclusions set broad parameters to begin with, and are STILL seen to be non-replicable is the issue.

[John 6IX Breezy]

I'm not well versed enough in psychological science's methodology to say for sure, but it's my impression that the results are calculated within a disclosed margin of error. Others' results (though they may differ) can fall within this margin and still be considered a replication. That the conclusions set broad parameters to begin with, and are STILL seen to be non-replicable is the issue.

I found a short introductory video on what I'm describing. 

In either case I would organize the issue of replication into three questions. And I'd be interested in knowing your answer to these questions. (Personally, the more I think about the questions the more I see replication as problematic and of limited scientific value—which I know goes contrary to our inherited scientific beliefs.)

1. What counts as replication?
2. What function does replication have and how?
3. What should we conclude when replication fails or succeeds?