If the grit seems to be stuck to your paint then clay would definatly help. Do a search on IFO (Industrial Fallout) and you'll learn real quick what it is. Clay should help a great deal. Head to AutoZone and pick up some
Clay Magic. That stuff is the best.
My first thought was that perhaps it was just wax that you buffed off and it was just sitting on top of the car. If you buff off wax with a towel that is 'full' of wax then sometimes it will leave little goobers on your paint. They will blow off after awhile though. If the grit that you feel is attached to your car then get some clay. If it feels like its just rolling around on top then get a clean towel and wipe it down. Either way I think your problem is pretty easy to fix.