youre afraid to ruin the paintwork but you dont have a problem wet sanding?? you are much more likely to ruin the paint by sanding than using a rotary..
You can do it by hand, I imagine, but it will be a lot of work depending on how large an area it is. You just have to choose the right products and use them appropriately.