Do you care about your significant other's sexual past?

[MJ the Skeptical]

Well, there are two things here, Godslayer:
1)

I mean, what you're saying is that I should change to whatever my potential partner wants.

That's not entirely true. You're making it sound like a zero sum game. There are things you can have limits on - be they soft or hard. You learn to negotiate. Understand that you'll never find someone who perfectly matches you sexually, so compromise with what the other person wants in bed is essential. This means telling each other what pleases you, or your fantasies, and not taking them personally. An example: anal doesn't do anything for me, but I'm willing to negotiate with it. That's a soft limit. Depends on the person and how they do it. A hard limit? I refuse to be hit in the face. (For those who think this is an obvious *NO*, you must have missed the BDSM references) You don't change yourself entirely to their wishes - they obviously already want to be with you, so clearly they already wish for most of you anyway. You simply compromise on the things you want to do or not. It's excellent self-exploration, and who knows - you might find out new things that you like. This process is what helped me figure out how I feel about women and BDSM.
2) Your anxiety
If your anxiety is crippling enough that it would make you think twice about being with someone who is otherwise great just because they'd had a lot of partners, my personal opinion is that it's possibly something you want to seek help with.
Beyond that, just the other day a friend of mine was talking about his own anxiety after our sexual encounters: He knows I enjoy BDSM, and that it's a complicated thing and he felt he was terrible at it because it's not his bag. However, he very much wanted to have sex with me, so he did his research and adjusted accordingly as much as possible. This translated mostly to bondage and rough sex...and that was fine. I knew he wasn't into BDSM, but the fact that he added the minimal amounts of control and pain elements to play to my kinks was fantastic! We had a great time, and I enjoyed myself immensely (as did he) and I have no complaints. He told me that so, when the anxiety crept up, he sat back and thought about it and realized that I already had known he wasn't into those things, and still chose to share my sex with him, so clearly I was getting something out of it. So what was there to be anxious about?
Anxiety is funny. It can make you think that everyone is lying to you. It can make you doubt your own virtues. It cripples you from enjoying someone to the fullest extent. I've had to get over a lot of my own anxieties in order to enjoy people I otherwise wouldn't have had a chance with because I would have sabotaged - and not a single one of them required me worrying about their sexual past. By the time you get into bed with someone, the time for being anxious is over. You should have already negotiated things with them - not only is the process ripe for dirty talk and fantasy exploration, but it gives you the chance to feel more skilled.
Teddy Roosevelt had an excellent quote: "Comparison is the thief of joy."
I don't compare the person in my bed critically against other people in my past. Sure, comparisons get made because you can't help it, but that doesn't mean they're negative. So this one guy didn't fuck me for an hour straight like one guy did...instead he spent the prior hour teasing me to the point of madness with his mouth. So one person is a bad kisser, but they sure as shit know how to use their hands and genitals. So one person has all the basic stuff down without being resoundingly special about it, but knows how to create a mood with his dominant personality to the point where I don't give a shit what he does because it's all going to drive me into a melty puddle of lust. So one person puts me in a dark, bitter, helpless subspace, but another doesn't like BDSM at all and sex with them is all sunlight and honey. The point is to appreciate the person you're with for what they are - because I assume you wouldn't be with them otherwise, and you should assume that they feel exactly that way towards you, or they wouldn't be pursuing sex with you.
"Love as thou wilt," as they say in the Kushiel books. Approach sex always with joy and appreciation and as long as health and safety are involved, you won't have to worry.

First of all summerqueen, we haven't formally met, but I'm Mike and I'm glad to meet you. I appreciate your well thought out post, thank you. But honestly, I don't have any disagreements with you here and I'm not exactly sure where you disagree with me other than you disagree with how I would handle things relationship-wise compared to yourself. While I definitely lose out on many relationships and friendships because of anxiety. I still feel like I have to stack up with all the other guys they've been with and I'm just not comfortable with that, probably because I've been rejected and now fear it, maybe I'm damaged goods, I don't have much success in dating and I'm probably pathetic for admitting that. I conceded that I am open to the possibility of being with someone who gets around. I'm not saying I won't make compromise, or go for things like soft limits as you mentioned. You made some good points in here and I appreciate you for it.

Idk, I kind of have tons of regrets in the dating department, kind of depressed cus of it.