Obviously I do not know your circumstances in terms of your marriage but for me (us) fidelity is the thing. Whether I was bi or not that wouldn't change. If your relationship is more open then I could see it becoming an issue but otherwise.....
In the meantime who you imagine is doing what to you in fantasy (during sex or not) is your own business and something you can choose to share or not. Is your sexual orientation really anyone's business but your own? If you want to make a flag out of it and wave it round for all to see that's fine - its your choice - but it really isn't essential and I can't see why you should feel guilty for choosing not to.
In the meantime who you imagine is doing what to you in fantasy (during sex or not) is your own business and something you can choose to share or not. Is your sexual orientation really anyone's business but your own? If you want to make a flag out of it and wave it round for all to see that's fine - its your choice - but it really isn't essential and I can't see why you should feel guilty for choosing not to.