Right, the plastic doesn't absorb the heat so the *paint* gets hot and sometimes that can cause problems. People have even had it "twist" right off the plastic!
But, that said, I've never had any problems like that and I even use rotaries on plastic parts. Just try to be gentle, at least at first, and don't ramp things up quite as aggressively as you would on metal panels.
Oh, and Welcome to Autopia, Lunker!