Were you washing with the cotton towel or drying?
Either way, I'd get rid of it and use a chenille, sheepskin, or microfiber mitt to wash the car with, and a quality microfiber waffle-weave drying towel to dry the car with. I prefer these:
http://www.prestostore.com/cgi-bin/s...9280&pd=133820
And read the Autopia Guide to Detailing too.