Neothin- Nah, some of us do notice a big drop in vibration without the handle (and yeah, I was screwing it in all the way

). There's some proper explanation for it, regarding transmission of vibration from the source via a tangential appendage or something like that and it being exacerbated by the handle being hollow rather than solid, but I forget the specifics.
As to it being awkward, I suppose that it's just a matter of personal preference and getting used to it one way or the other.