Imatk- While I hardly ever polish unless there's some marring to remove, I think that you probably oughta give it a gentle polishing. I wouldn't be surprised if you find it makes for a slight overall improvement. Something like the FPII would be a good choice, try doing a single panel and see if it looks any better.
Oh, and the whole "aren't polishes abrasive" thing *can* be confusing. I too agree that "polish" oughta equal "abrasive" but there are "pure polishes", as
Meguiar's calls 'em that are not abrasive and some chemical cleaners are called polishes too as
Danase pointed out.