Always remember, you are in control. No matter what happens, you are always in control. Think about that for a second...no I am serious, think about it. Let your audience examine everthing that you let them, and not examine everything you dont. If someone grabs, it means you did a poor job of convincing them of what you were tying to put on.
This goes right into you patter, techinique, and missdirection (if needed.) I will take the example of the Color Monte, but this idea can be applied to any effect in my mind. The story that I use, like many others, is how I got swindled by a guy on the street. I take the audience through the story. Magic is just a story that has pictures and thats all. The story has magical moments, like when the colors are on top, in the middle and on bottom and such, and then the kicker ending, but it is still just a story. Take them on a journey. If you cant do that, then get out of magic.
The point is, practice more. Practice more on your techinique and practice more on your patter. edited, write out your patter to make it flawless and in the end you will be more seemless with your presentation which in turn will not make your spectators think "Oh crap, he just did a move, I had better call him on it!"
A great quote from my high school drama teacher is this "If I must screw up, I will do so with energy and entusiasim." What this means, in my opnion, is that your audience will enjoy your preformance more if you dont worry about moves, patter or what you are doing physically, as long as you are having fun telling them a story. How many times have you seen an actor, singer, etc do what they do but they have that uneasy face on them. A face that says that they are worried about what they are doing. Like they havent rehearsed the act enough and you actually feel bad for them. Why? Because they are focused inward rather than outward. Now have you seen a singer, actor, etc doing what they do horribly , but are loving every second of it and having a blast? You settle back in your seat and ride with it. It might not be the best thing ever, but at least they are focused on giving you a good time. This bit of advice has gotten me through many a bad situation in magic (being heckler or whatever.)
Yes you will get hecklers. We all do. Saying you are going to do the "Appearing Cane and push them off" is not only impracticle, its down right stupid, childish, and unperfessional. There are many other ways to deal with hecklers (did you see what I did there...I didn't say "Stop Hecklers", I said "Deal with Hecklers.)
But going back to the start of my post, you are the one that is in control. And hey, if this guy keeps pestering you, you have the control to simply walk away.