Well yeah actually it is.
That's the fallacy of narco-libertarianism. Like everyone lives in a deserted farm in Montana and has no interaction with other people at all.
What difference does that make?
Are you afraid you'll see someone high on dope?
If you live in a major city, you do every day, even if you don't realize it.
But I dont usually have that person at the wheel of a car hurtling towards me. Or driving my bus. Or operating on me.