M105 with a polishing pad will probably leave it's own scratch pattern, though depending on your paint, a white pad with light pressure could possibly get you extremely close to LSP ready.
As stated, a final polishing and/or jeweling step will remedy this problem as well as bring out a superior gloss.
However, to answer your question, a light buff with a quality MF towel isn't going to do a single thing to your paint. M105 is a good product, but its not that good. If such a gentle buffing could produce enough heat and pressure to actually abrade clear coat, there would be absolutely no use for polishing pads, or mechanical polishers. YOu could just wipe the stuff on and buff it away like wax.
The whole point of using a machine, and pushing down on the pad is to create heat and pressure that will allow the abrasives in the polish to smooth out the clear coat. There is a big difference between putting your weight down on a Flex polisher with an abrasive polishing pad, and wiping away the residue (which contains lubricants and oils for this specific purpose).