It does exactly what you said. I always do a IPA or prep sol wipe down after polishing to see if any swrils or marring were missed. And no, fillers will not do any harm except the swrils will return.
IF you have enough paint to work with Better imperfect original paint than a need for a repaint in my book (but yeah, this isn't much of an issue with newer cars).