If you do it in a light sweeping fashion rather than placing your palm over the towel and wiping, I don't see how it could be any more marr inflicting than a california cae duster would.
I would say no way. Even if you are lightly "dusting" your car's surface, you are still rubbing the dust across your paints surface with no lubrication. Take the 30 minutes and give your car a quick wash/dry!