Wear it if it is a mit without a thumb. I have an old one with a thumb that is a PITA to flip so I don't wear that one. Just using it on the wheels now so it won't be a problem much longer.
I currently use the Meguiars mitt which only really has one side so I wear it. To be honest, even if I had a "two-sided" mitt i'd wear it - makes it a bit harder to drop.