gti- re stirring up dust or dirt.
I usually wash my cars in my driveway, which is concrete. So, by the time I blow dry the car (!), the driveway has been cleaned and rinsed by the prior washing of the car.
I have blow dried cars on grass many times without problem. I think I could even blow dry a car on dirt, I'd just have to skip blowing the lower parts of the lower panels of the car. The velocity of the air drops off rapidly with distance, so there is no real problem, particularly if you do not blow directly into the grass or dirt.
carguy makes a good point: you have to make sure to not bang the car with the blower. But, that same principle applies to any hard object we have around the car.
There are many legitimate views on drying. My own experience is that nothing I have tried is as easy, safe, thorough, or as quick as the blower.