Questions about genders ... male/female

[Angrboda]

... So even if a new gender were to have mutated, it would have very likely been ousted very quickly, as the likelihood that a single new gender would be able to reproduce would be pretty low.

The problem with this is that the same logic applies to one sex versus two. Sex is expensive, evolutionarily speaking. One sex alone, asexual reproduction, easily outperforms two sexes in terms of reproduction, so there has to be some leveling factor that justifies the added cost of two sexes. For a time it was thought that the increase in genetic diversity yielding reproductive advantages was that justification, but if I remember rightly, simple diversity on its own doesn't explain it.

Matt Ridley's The Red Queen: Sex and the Evolution of Human Nature goes into detail on the question and is probably the go to book for answers regarding the question.

See Wikipedia | Red Queen hypothesis.