Definitely both. Using dirty wash water, sponges that trap dirt, coarse wash mitts, bath towels to dry with, synthetics that collect dirt, and many other bad practices and poor tool choices contribute to the swirls and damage that occur while washing and drying. Using the double bucket method to wash (one clean bucket of water to rinse the wash mitt before going in for more soapy water) will help keep your wash water clean. Also, using a waffle weave towel and blotting the water off will help reduce any swirls or scratches that occur while drying.