Philosophy Versus Science

[The Grand Nudger]

(AFTER a person has made a value choice -- e.g. "it would be good for me to cure diseases" -- then the methods one uses may well be informed by science. But the value choice itself is not a scientific choice.) 

This is a belief some people hold about valuation, not a fact about valuation.

[GrandizerII]

The tendency for only the brightest philosophy majors to take the LSAT or the GRE was also examined, and they did not find any statistical evidence that this would have mattered to the validity of this study.

They didn't find any statistical evidence because they never went and looked. All they ever did was say 'Nah, didn't happen. Let's move on.' They looked for something they called "interaction" without ever even describing what that is, much less how they tested for it. Then they just go on to say that there isn't any. There's no way for anybody reading that paper to decide for themselves if there is or isn't, because we have no idea how they decided that.

Whenever you see your stats returning glowing praise like their diagrams show, you need to be very, very careful. Odds are very good that you're doing something wrong and your data is contaminated with nasty, nasty artifacts. The real world simply doesn't behave like that. It's horrifyingly messy and on a bright day, you get data that's good enough to draw a line through. When your data supports your conclusions this impressively, a good researcher goes and does some very thorough statistics. If only to make the "I told you so!" much more rigorous.

I agree they should’ve provided some details in the text on what was done to determine no interaction effect. They do provide enough details for replicability, but they should have made more space to address the first part of your critique properly.

As to the rest of what you said, every study has its limitations (and quite a number are acknowledged in this study) and we have to be careful not to exaggerate or overgeneralise these findings. However, I still think they took great care with their work, and I don’t see any signs of deception in what they said. The data is there anyway, if another researcher wants to attempt to replicate the findings themselves.

[Paleophyte]

They didn't find any statistical evidence because they never went and looked. All they ever did was say 'Nah, didn't happen. Let's move on.' They looked for something they called "interaction" without ever even describing what that is, much less how they tested for it. Then they just go on to say that there isn't any. There's no way for anybody reading that paper to decide for themselves if there is or isn't, because we have no idea how they decided that.

Whenever you see your stats returning glowing praise like their diagrams show, you need to be very, very careful. Odds are very good that you're doing something wrong and your data is contaminated with nasty, nasty artifacts. The real world simply doesn't behave like that. It's horrifyingly messy and on a bright day, you get data that's good enough to draw a line through. When your data supports your conclusions this impressively, a good researcher goes and does some very thorough statistics. If only to make the "I told you so!" much more rigorous.

I agree they should’ve provided some details in the text on what was done to determine no interaction effect. They do provide enough details for replicability, but they should have made more space to address the first part of your critique properly.

As to the rest of what you said, every study has its limitations (and quite a number are acknowledged in this study) and we have to be careful not to exaggerate or overgeneralise these findings. However, I still think they took great care with their work, and I don’t see any signs of deception in what they said. The data is there anyway, if another researcher wants to attempt to replicate the findings themselves.

We'll have to agree to differ. IMO, that's a pretty disreputable junk study. Their biggest sins are explaining their methodology only in passing, if at all, and failing to show any of their work. To me, that's just another baseless opinion.

[GrandizerII]

I agree they should’ve provided some details in the text on what was done to determine no interaction effect. They do provide enough details for replicability, but they should have made more space to address the first part of your critique properly.

As to the rest of what you said, every study has its limitations (and quite a number are acknowledged in this study) and we have to be careful not to exaggerate or overgeneralise these findings. However, I still think they took great care with their work, and I don’t see any signs of deception in what they said. The data is there anyway, if another researcher wants to attempt to replicate the findings themselves.

We'll have to agree to differ. IMO, that's a pretty disreputable junk study. Their biggest sins are explaining their methodology only in passing, if at all, and failing to show any of their work. To me, that's just another baseless opinion

I don’t understand what you mean by “failing to show any of their work”? The data has been linked to (though you may have to pay to access some of it, like they did), the methodology is described in Sections 4 and 5, the code they used for analysis is linked to, results are reported and analysed, and there are tables and diagrams and an appendix.

I mean, sure, they don’t show the statistical tests in full detail, but that’s standard, and there may be some word limit imposed anyway.

Again, any researcher who is suspicious can test their findings using the same data and details they have provided.

[Paleophyte]

We'll have to agree to differ. IMO, that's a pretty disreputable junk study. Their biggest sins are explaining their methodology only in passing, if at all, and failing to show any of their work. To me, that's just another baseless opinion

I don’t understand what you mean by “failing to show any of their work”? The data has been linked to (though you may have to pay to access some of it, like they did), the methodology is described in Sections 4 and 5, the code they used for analysis is linked to, results are reported and analysed, and there are tables and diagrams and an appendix.

I mean, sure, they don’t show the statistical tests in full detail, but that’s standard, and there may be some word limit imposed anyway.

Again, any researcher who is suspicious can test their findings using the same data and details they have provided.

If I have to burrow into their code to figure out how they did their math then they're doing it wrong. Worse, that code is only a small proportion of their stats. Sure, their data is available online. You have to pay for access, which is unfortunate but common. The difficulty is that there are things like "adjusted average" that their conclusions hinge on and no indication on how that average is adjusted. When you do something pivotal like that you need to explain it explicitly in the text, and that just isn't there. Something along the lines of "The average GRE scores were adjusted for pre-enrollment scores using the following common and well-worn methods... because of these reasons..." and if they aren't standard methods (which they get excited about telling you in the foortnotes? Really? I have to dig into the footnotes to find that this is pioneering work in the field?!?) then you'll want to include a lot more justification and detail. You'll especially want to do that if you are reporting surprising results because other authors are going to want to know exactly how your math got to this unexpected outcome. Finding philosophy at the front of the pack in almost every score tested should have triggered a more thorough review of the stats and a much more thorough description of the steps between raw data and inferences. I don't need to see it done out long-hand, that's ridiculous, but I do need to know what tools they're using and why. Simply listing the names of the software is insufficient.

I doubt that there's a word limit issue since this was a pretty brief paper as is. It's possible that they were writing for philosophers, but then they need to include these details in an appendix somewhere. You also have some pretty major biases in the data that are never addressed. Not once do they discuss the fact that all of the data are self-reported, which ought to have been a huge issue. Perhaps I'm missing something?

[Belacqua]

I don’t understand what you mean by “failing to show any of their work”? The data has been linked to (though you may have to pay to access some of it, like they did), the methodology is described in Sections 4 and 5, the code they used for analysis is linked to, results are reported and analysed, and there are tables and diagrams and an appendix.

I mean, sure, they don’t show the statistical tests in full detail, but that’s standard, and there may be some word limit imposed anyway.

Again, any researcher who is suspicious can test their findings using the same data and details they have provided.

I don't think the research paper tells us anything important. 

It's an attempt to claim that the successes of one field can be identified and quantified according to the values of a different field. 

The things which can be (allegedly) identified and quantified through this study are not the things that are important about philosophy. Philosophy is mostly about wisdom. It's about how a person lives. A wise man may or may not do well on standardized tests. He may be too interested in his passionate endeavors even to take a standardized test. 

So I think it's a category error. 

If practice discussing philosophical issues with smart, temperate, and generous people does increase a person's score on some standardized test, that is a fortunate side-effect, but unrelated to the goal of philosophy. 

Also, I think it would be interesting to compile a list of philosophy books which are still entirely relevant and worth reading, and have had their value in no way reduced by modern science. If we start making a list I think we'll find a lot of them. 

I'd say the Nicomachean Ethics is certainly one of them. Though the ethics it describes are certainly different from our own, one of the great benefits of the book is to get a clear idea of what an alternative ethics would look like. If we compare to what we have now, we see how Aristotle's version has certain benefits. It shows how our own ethical system is certainly contingent on our time and place, and not something that will be self-evident elsewhere.

Likewise After Virtue by Alasdair MacIntyre. Fascinating and challenging philosophy book which is, again, completely unaffected by changes in science. Many people will disagree with its conclusions, but will nonetheless benefit from reading it. 

Language Truth and Logic by A.J. Ayer. This is the clearest statement in English of the logical positivist view which many of our fellow atheists believe without knowing that they believe it. How Ayer spells it out, and how people since then have come to (mostly) reject his claims are valuable to know. 

The Birth of Tragedy from the Spirit of Music by Nietzsche. This is a post-Darwin book, written with the theory of evolution well in mind. It can certainly be criticized in terms of the accuracy of Neitzsche's reading of Greek culture, but advances in modern science have not lessened its message in any way.

The idea that somehow philosophy is not necessary any more because science has taken its place is belied by the existence of these, and many other books. They do different things than science, and they do things that science can't do and can't criticize. 

No doubt you can think of lots of others.

[Sheldon]

either pursuit would make me less useful in certain ways.

The idea that it is good to be useful is of course a philosophical idea. 

"How should one live?" is among the oldest and most important topics in philosophy, and "one should be useful" is one  way that people have answered that question. Valuing usefulness over other things is a position that has a long history and genealogy. Some people have disagreed, of course, but I would never argue against it.

(AFTER a person has made a value choice -- e.g. "it would be good for me to cure diseases" -- then the methods one uses may well be informed by science. But the value choice itself is not a scientific choice.)

So a method designed to objectively examine reality is no good at evaluating purely subjective choices, but then nor do such choices require evidence of any kind, so we're back to where you assented a poster was ignoring "evidence" unless it was scientific, yet no example of such evidence has been offered, only subjective claims which of course are not evidence, nor do they require any.

[GrandizerII]

I don’t understand what you mean by “failing to show any of their work”? The data has been linked to (though you may have to pay to access some of it, like they did), the methodology is described in Sections 4 and 5, the code they used for analysis is linked to, results are reported and analysed, and there are tables and diagrams and an appendix.

I mean, sure, they don’t show the statistical tests in full detail, but that’s standard, and there may be some word limit imposed anyway.

Again, any researcher who is suspicious can test their findings using the same data and details they have provided.

If I have to burrow into their code to figure out how they did their math then they're doing it wrong. Worse, that code is only a small proportion of their stats. Sure, their data is available online. You have to pay for access, which is unfortunate but common. The difficulty is that there are things like "adjusted average" that their conclusions hinge on and no indication on how that average is adjusted. When you do something pivotal like that you need to explain it explicitly in the text, and that just isn't there. Something along the lines of "The average GRE scores were adjusted for pre-enrollment scores using the following common and well-worn methods... because of these reasons..." and if they aren't standard methods (which they get excited about telling you in the foortnotes? Really? I have to dig into the footnotes to find that this is pioneering work in the field?!?) then you'll want to include a lot more justification and detail. You'll especially want to do that if you are reporting surprising results because other authors are going to want to know exactly how your math got to this unexpected outcome. Finding philosophy at the front of the pack in almost every score tested should have triggered a more thorough review of the stats and a much more thorough description of the steps between raw data and inferences. I don't need to see it done out long-hand, that's ridiculous, but I do need to know what tools they're using and why. Simply listing the names of the software is insufficient.

I doubt that there's a word limit issue since this was a pretty brief paper as is. It's possible that they were writing for philosophers, but then they need to include these details in an appendix somewhere. You also have some pretty major biases in the data that are never addressed. Not once do they discuss the fact that all of the data are self-reported, which ought to have been a huge issue. Perhaps I'm missing something?

Paleophyte, I think part of your beef with the study is that you're coming at it from a "hard science" perspective (perhaps because it's a quantitative study). But as you're aware, this is a social science study primarily written for philosophers (with the more technical stuff written for social scientists). So of course, it's not going to read like a physics study or anything like that.

I just had a look at the code, and it looks to me like it contains all the work required for analysis, including what you have been asking about. What are you suggesting is missing there?

The first two tables in the appendix contain some important stats such as intercepts and CI intervals and p-values. Now I have no idea why all of this was not included in the main text (because as someone who majored in psychology, I did have to report on this stuff in the main text itself, even if in the form of tables and such), but I don't really know what the write-up requirements are for "philosophy/social science" studies like this.

They do, however, point out (at the end of Section 2) how confounds are controlled for using "covariate adjustment", and they later mention in Section 5 what models overall were used by their analyses.

Causal inference research is not exactly new. What's uncommon, as the first footnote says, is the application of causal inference techniques to the study of the effects of philosophical education.

As for the results themselves, they're actually not that surprising ... or even that impressive when you remember that the baseline differences were adjusted for in this study. It has been known for quite some time that there is a noticeable correlative relationship going on between undertaking philosophical study and logical/verbal reasoning. They just hadn't been able to establish a causal relationship until now.

To be clear, the study does not show that philosophy majors are more intelligent than other majors. Rather, it shows that undergraduate philosophy boosts certain intellectual abilities and virtues higher than other undergraduate studies. Not all intellectual abilities/virtues, just some of them.

The point of this study was to show that what they had suspected for quite a while (thanks to observations and prior studies) is in fact the case.

By the way, it is not true that all data provided to them was self-reported. The standardized test scores themselves are not self-reports. The scores on the Habits of Mind and Pluralistic Orientation scales are self-reports, but as stated in one of the paragraphs from the study itself:

In short, these two different kinds of measures have complementary strengths and weaknesses. Standardized tests are “objective” in the sense that they are immune to reporting biases. They also capture a broad range of important abilities but might be thought to reflect a relatively thin conception of good thinking. On the other hand, self-reports can capture dispositions like curiosity and open-mindedness, that seem to be important aspects of good thinking. But these are less “objective” in the aforementioned sense. Given these relative advantages and disadvantages, we would ideally like to see converging evidence from both kinds of measures. That is, although either result would be interesting in its own right, evidence that studying philosophy improves both test scores and self-reported intellectual dispositions would provide particularly strong evidence that the discipline makes people better thinkers.

[GrandizerII]

I don’t understand what you mean by “failing to show any of their work”? The data has been linked to (though you may have to pay to access some of it, like they did), the methodology is described in Sections 4 and 5, the code they used for analysis is linked to, results are reported and analysed, and there are tables and diagrams and an appendix.

I mean, sure, they don’t show the statistical tests in full detail, but that’s standard, and there may be some word limit imposed anyway.

Again, any researcher who is suspicious can test their findings using the same data and details they have provided.

I don't think the research paper tells us anything important. 

It's an attempt to claim that the successes of one field can be identified and quantified according to the values of a different field. 

The things which can be (allegedly) identified and quantified through this study are not the things that are important about philosophy. Philosophy is mostly about wisdom. It's about how a person lives. A wise man may or may not do well on standardized tests. He may be too interested in his passionate endeavors even to take a standardized test. 

So I think it's a category error. 

If practice discussing philosophical issues with smart, temperate, and generous people does increase a person's score on some standardized test, that is a fortunate side-effect, but unrelated to the goal of philosophy.

It's not about the scores themselves, it's what those scores represent.

And how would you define wisdom if not in terms of such dispositions as intellectual humility and open-mindedness?

Again, the point of the study is to show the benefits of studying psychology (at least at the undergraduate level). Not just with regards to intellectual abilities, but also with regards to virtues that surely you would consider to be part of wisdom?

And what is exactly the goal of philosophy? I don't agree there is this one ultimate goal of philosophy that philosophers all/mostly agree on.

[Alan V]

either pursuit would make me less useful in certain ways.

The idea that it is good to be useful is of course a philosophical idea. 

(etc.)

Like so many other philosophers, you seem to be over-thinking this.  I meant useful in terms of priorities, not in terms of good versus evil.

I studied "wisdom" literature for a long time in my life. I concluded that most of it depended on the idea that God exists, which was left unsupported.