Philosophy Versus Science

[The Grand Nudger]

(AFTER a person has made a value choice -- e.g. "it would be good for me to cure diseases" -- then the methods one uses may well be informed by science. But the value choice itself is not a scientific choice.) 

This is a belief some people hold about valuation, not a fact about valuation.

[GrandizerII]

The tendency for only the brightest philosophy majors to take the LSAT or the GRE was also examined, and they did not find any statistical evidence that this would have mattered to the validity of this study.

They didn't find any statistical evidence because they never went and looked. All they ever did was say 'Nah, didn't happen. Let's move on.' They looked for something they called "interaction" without ever even describing what that is, much less how they tested for it. Then they just go on to say that there isn't any. There's no way for anybody reading that paper to decide for themselves if there is or isn't, because we have no idea how they decided that.

Whenever you see your stats returning glowing praise like their diagrams show, you need to be very, very careful. Odds are very good that you're doing something wrong and your data is contaminated with nasty, nasty artifacts. The real world simply doesn't behave like that. It's horrifyingly messy and on a bright day, you get data that's good enough to draw a line through. When your data supports your conclusions this impressively, a good researcher goes and does some very thorough statistics. If only to make the "I told you so!" much more rigorous.

I agree they should’ve provided some details in the text on what was done to determine no interaction effect. They do provide enough details for replicability, but they should have made more space to address the first part of your critique properly.

As to the rest of what you said, every study has its limitations (and quite a number are acknowledged in this study) and we have to be careful not to exaggerate or overgeneralise these findings. However, I still think they took great care with their work, and I don’t see any signs of deception in what they said. The data is there anyway, if another researcher wants to attempt to replicate the findings themselves.

[Paleophyte]

They didn't find any statistical evidence because they never went and looked. All they ever did was say 'Nah, didn't happen. Let's move on.' They looked for something they called "interaction" without ever even describing what that is, much less how they tested for it. Then they just go on to say that there isn't any. There's no way for anybody reading that paper to decide for themselves if there is or isn't, because we have no idea how they decided that.

Whenever you see your stats returning glowing praise like their diagrams show, you need to be very, very careful. Odds are very good that you're doing something wrong and your data is contaminated with nasty, nasty artifacts. The real world simply doesn't behave like that. It's horrifyingly messy and on a bright day, you get data that's good enough to draw a line through. When your data supports your conclusions this impressively, a good researcher goes and does some very thorough statistics. If only to make the "I told you so!" much more rigorous.

I agree they should’ve provided some details in the text on what was done to determine no interaction effect. They do provide enough details for replicability, but they should have made more space to address the first part of your critique properly.

As to the rest of what you said, every study has its limitations (and quite a number are acknowledged in this study) and we have to be careful not to exaggerate or overgeneralise these findings. However, I still think they took great care with their work, and I don’t see any signs of deception in what they said. The data is there anyway, if another researcher wants to attempt to replicate the findings themselves.

We'll have to agree to differ. IMO, that's a pretty disreputable junk study. Their biggest sins are explaining their methodology only in passing, if at all, and failing to show any of their work. To me, that's just another baseless opinion.

[GrandizerII]

I agree they should’ve provided some details in the text on what was done to determine no interaction effect. They do provide enough details for replicability, but they should have made more space to address the first part of your critique properly.

As to the rest of what you said, every study has its limitations (and quite a number are acknowledged in this study) and we have to be careful not to exaggerate or overgeneralise these findings. However, I still think they took great care with their work, and I don’t see any signs of deception in what they said. The data is there anyway, if another researcher wants to attempt to replicate the findings themselves.

We'll have to agree to differ. IMO, that's a pretty disreputable junk study. Their biggest sins are explaining their methodology only in passing, if at all, and failing to show any of their work. To me, that's just another baseless opinion

I don’t understand what you mean by “failing to show any of their work”? The data has been linked to (though you may have to pay to access some of it, like they did), the methodology is described in Sections 4 and 5, the code they used for analysis is linked to, results are reported and analysed, and there are tables and diagrams and an appendix.

I mean, sure, they don’t show the statistical tests in full detail, but that’s standard, and there may be some word limit imposed anyway.

Again, any researcher who is suspicious can test their findings using the same data and details they have provided.