I never use ATMs unless I don't have a choice. I go to a store.
Most frequently, there's a Kroger around the corner I go to when I need cash. It's perfect because there are no ATM fees (I have to make a purchase to get the cash, but I can make it less than an ATM fee, and I actually get something for the money I'm spending). Also, I can get cash back in any exact amount I want instead of that stupid "multiples of $20" crap ATMs force on you. I can get four cents back on debit if I want. And, my card never leaves my hand, so the OP's problem never factors.
Most frequently, there's a Kroger around the corner I go to when I need cash. It's perfect because there are no ATM fees (I have to make a purchase to get the cash, but I can make it less than an ATM fee, and I actually get something for the money I'm spending). Also, I can get cash back in any exact amount I want instead of that stupid "multiples of $20" crap ATMs force on you. I can get four cents back on debit if I want. And, my card never leaves my hand, so the OP's problem never factors.