I think you should just be open and explain why you don't what to do it, explain that you are not trying to be rude, you just don't share the same beliefs.
I am Christian and if someone said this to me I would understand and apologise for making you feel like you had to take part.
I am Christian and if someone said this to me I would understand and apologise for making you feel like you had to take part.