Not all grout sponges are created equal. I have one I use for big tires, that'll give you an idea of how abbrasive it is. But I have soft black BMW paint to deal with and I use an extremely soft GS. I can find extremely soft GS locally.
Any grit that can cause scratches gets pressure washed off in the first place, for me anyhow. I foam then pressure rinse, and then GS with one of the quality car washs.
I do get light marring if Im not careful. But thats the whole idea about detailing. You have to be careful and use proper procedures.
Spilchy, Ive used a real sea sponge. The increased in cost turned me off. Plus, I can still get marring with a sea sponge if Im not careful. Plus, I cant get them locally. To me there was no benefit to keep using them.