I can't wrap my head around this

[Mudhammam]

I hope that was sarcasm.

Not at all. The only way any person could have been someone else is if the self is separate in some way from the body. Since most people here don't believe this, then the OP might as well be "Why isn't this orange an apple?" It isn't an apple because it isn't an apple, and we don't question it because there's no innate possibility for an orange to be anything but what it is. Same for people, unless you believe in a soul.

The self as separate from the body... Not a coherent idea. My body, my insides and out, is the only thing that can be properly identified as me. To make "me" that indescribable thing floating somewhere over there, that is somehow in control of my "physical self," is to completely divorce the concept of "me" from that mysterious entity. It's no longer "me" or "you" once you cannot even in principle describe it, understand it, actually relate to it in anyway, or claim any semblance of control over it ("it" is hence separate from not only the body, but YOU, yourself).