I get this too but it's only when I'm trying something new for family and friends. As everyone has stated that's usually because it's new and I'm worried they my figure it out or notice the gimmic or slieght.
Plus they know I'm going to be doing something "tricky" so they watch me like a hawk. I know if I can fool them I'm fine for the street crowds. If not it needs more work.

Practice, number one cure for stage freight is practice. You cannot be confident with something you havent practiced enough. If you're not comfortable with things you need to spend more time with them. I would also recommend only performing in places and for people that you're comfortable with. Perform for family and friends, not people on the street. Work your way up. I know my first few stage shows I had to bring my girlfriend with me for support. I still get nervous when performing a new effect but with enough practice it definetly helps. Video taping yourself performing the effect and watching it is also a big plus. The other thing is that most people do not practice patter, if you dont, when it comes time to talk in front of people you're in trouble.

shaky,jittery hands mean lack of confident or afraid of failure

Warm-up your hands. Shuffle the pack a lot before you go into a performance. Practice your false-cuts and your flourishes. This gets your hands warmed-up and used to the feeling of doing what they're supposed to. It's kinda like how a singer warms up their voice before a show.

Just make the audience laught hat, will make you less tensed and less stressful.

you might just want to start with something easy yet effective. For example, 'Cut Here' by Jay Sankey is super simple and very good. It takes a lot of the initial pressure off.

you also may want to try to practice on your parents/close friends before performing for strangers or some one like that...

-Vinnie