I have a bit of glue residue from racer's tape (much like duct tape) from my car. I was able to get it off using a bug & tar remover product, but it was evident that I created some scratches while getting it off. Looks like I'll have to polish those out now...
Your experience worries me, I was considering buying some racers tape to use for my next trip, now I'm not so sure.
On my last trip I used low adhesion 3M Blue Tape which worked like a charm but my car stood out like a soar thumb. I left very minor residue which easily came off with QD.
3M makes and adhesive remover that removes any and all my residue from using racers tape. Isopropyl alcohol usually works well also. The key I have found is to apply the liquid to the residue/glue and let it sort of soak in a little then apply more liquid to the applicator you are using to wipe it and usually it will come right off. Use a different part of the applicator for then next section. Rinse and clean the applicator surface frequently so that you don’t reintroduce the residue to the paint. This last part is probably the most important. You can easily get a sticky applicator that just ends up reintroducing the mess to the paint. Also the use of a hair dryer has helped me remove the tape with less residue being left behind. Of course wash and reapply any protectant as necessary.