Ok. here's what I've found:
There are 3 types of people in the world. The first is people who actually beleive in magic, no explanation is necessary. These are usually young children, the ones you make a piece of paper float for, who take the paper,and try and make it float themselves for an hour
The second are people who don't beleive in magic, but enjoy it. They know it's slieght of hand, but appreciate the skill and the artistry of magic. These people enjoy a mystery, and enjoy trying to understand and work out how you did the trick. They discuss it with their friends after the show, and if they do spot it, they say "Ah!" to themselves and enjoy the rest of the show.
The third are the people who think that a magician isn't trying to entertain them, but trying to fool them. They feel stupid when they can't work out a trick, and try and turn the tables on you. These are the people who will turn cards on the table over you, grab the cards from you and shuffle them. To them, it's not entertainment, it's a challenge over who is smarter.
These people appear in all forms of entertainment. Before getting fully into magic, I worked as a stage hypnotist. People would shout out that everyone on stage were stooges, hypnotism isn't real. People who are like that will always be like that. One night, bored of a particularly nasty heckler, I brought him on stage and hypnotised him. Without going into detais Hypnotism is mostly about distracting the subject, so people trying not to be hypnotised are easier to hypnotise than a willing subject. They distract themselves by sitting there thinking "This isn't going to work".
I was working in a club, and had this guy running around the audience telling everyone I was his best buddy, that I was a genius, and trying to buy me drinks for the rest of the show.
At the end of the show, he 'woke' and the first thing he said was "Told you you couldn't do it." When he realised it was an hour later and everyone was laughing at him, he told everyone he was faking the whole thing.
I guess my point is, you'll always have hecklers. If someone doesn't show interest in seeing an effect, don't try to show it to them anyway. If you're doing a walkaround gig, do very simple magic to begin, and see which people to steer clear of.