If you're going to use a swirl remover by hand, I would suggest #9. You can use 3M SMR by hand, but it has fewer oils and fillers than #9, and is more effective by machine.
If you're doing this by hand, use linear motions, not circular. More difficult to introduce new swirls that way.