[Serious] Is ChatGPT a Chinese Room?

[emjay]

The way I've always looked at this question is to think of what we call 'understanding', whether phenomenally or neurally, as the measure of an idea/concept's integration and coherency. Coherency though is not the same thing as correctness, so you can be wrong in your understanding of something, because it's not a measure of truth but rather of how well things fit together, rightly or wrongly. So to me, to "understand" something is to be in a non-questioning state regarding that thing, but that understanding can still be wrong (as any experience of 'certainty' in the game of Mafia frequently attests ).

What beyond coherency would you say is required for truth?

That's a tough one, but one thing I can say is that the experience of certainty is often wrong, and not a reliable guide at all to the truth. In the game of Mafia you can have it all 'figured out' in a way that 'fits together' and you can often experience a sense of certainty that goes to the core of your being... as you might remember, I think you've played in the past? But then the 'flip' comes, and overturns everything you thought you knew. Or another example would be lateral thinking puzzles... puzzles where the only way to solve them is to think outside the box of your existing assumptions, the 'box' in that sense being an internally coherent understanding itself. So what makes one or other understanding closer to the actual truth?

This is just off the top of my head, so please forgive it if it's not well thought-out but from these examples it seems to be more a matter of degree, ie in both cases new information comes in that needs to integrated, and so the new representation better models the available data than the old representation. So I guess from this point of view this would make me some sort of relativist regarding truth because the brain in this sense is doing the same thing all the time, trying it's best to accurately model reality, or at least inasmuch as it's required to do so to survive... integrating ever more and more information with its existing model. So with each new integration it perhaps gets closer to some 'objective' truth, as a measure of integration, coherency, and perhaps add quantity of data (ie a lifetime's worth of observations about the world), but how to define that outside of what the brain seems to be doing, I don't know. I'll have to think about that.

ETA: Forget this, it's not a great answer and relativist is probably not the right word.