A man once thought I had taken a photo of him (I was actually zooming in on something far behind him) and he got quite angry about it. Fortunately with digital you can prove if you weren't doing something you are accused of.
The best way to get candid shots is to use a lens that is capable of zooming in from a distance, so you are less likely to be noticed. Then again where that isn't possible, if you are able to build up a rapport with the subject and can get them to pose for a few shots, you may be able to capture something. I find this easier to do in countries where I am clearly a foreigner and I show the picture to them and usually they are happy to let me go ahead. Ad astra per aspera
I suppose if you have the time to spend waiting for a good opportunity or to take hundreds of photos to get one publishable shot then that is less of an issue. I often find I don't have the time to hang about as much as I would like. Ad astra per aspera
In the UK every person has the right to take photos of other people in public places, so we don't have to ask for permission but I still feel incredibly rude for photographing people without asking.
Yesterday I went out photographing with a huge, clunky old Polaroid camera. Suddenly nobody cared if I was pointing the camera at them. It's weird - if I have my SLR with me, people notice, but if I have a small point&shoot, they don't care at all, and the same goes for a large Polaroid or even a Hasselblad. You have a normal feeling for a moment, then it passes. --More--
Speaking as some photographers' involuntary photography object, I say that's something you shouldn't get over :-) *Lunatic*, n. One whose delusions are out of fashion.
I spent yesterday evening persuading a room full of people to let me take shots of them for my friend for his birthday. It worked best when I joked with them, took a stupid shot and showed it to them saying surely they wanted to look better than that?! I don't think that would work in the street though! Ad astra per aspera