I’ve even heard it said that with for instance LSD it can be easier to have a somewhat higher dose, because with that you’ll have no control, and you’ll often be beyond even trying to control what’s happening in your head.

I honestly think that’s precisely what’s scaring people, at least it would scare the heck out of me. It’s like putting yourself in a submissive BDSM position with somebody you barely know. Who in their right mind wouldn’t be scared as shit. Even if it ends up going well, the sensation of helplessness and fear might leave a permanent trauma on you.