Sponge balls, telling me that I could multiply them because there were holes in the songe that other sponges fit in. (2 sponge balls can fit into 1 sponge ball, all 3 are the same size, they are geniuses.)

You are going to get these from people. No matter how illogical it is, as long as they feel that they have an answer for it, there is no other option. For example .... think of a time when a new trick hitthe market and you tried to figure out how it was done. You had an idea that was complicated, intricate and almost James Bond-like in gimmickry ..... and find out that it was a DL. That is an exaggeration, but we ourselves have all been there.
No matter how well you perform, it is going to happen. But while we sit here and make fun of them, there are good ways and bad ways to handle it. If you've seen The Illusionist, then you know the good way. Everytime the Count (or whatever he was) saw a trick, he thought of some outlandish method of it being done. Edward Norton's character's reply was a simple, almost surprised, chuckle with "Well, that is one way to do it." Just the perfect mix of attitude and grace.
When someone says "I know how you did that!", I usually retort with a "So you know how well I did it, too!" or something along those lines. Most of the time, a line like this will suffice.
Or to the person that says constantly that "It's all tricks!" or "It's all Sleight of hand!" (which is an explanation that I love simply due to the fact that they haven't explained a thing!). I simply turn to them and say with an almost sad face " ...... you ... you mean ...... that I'm not magic?". It's kind of a joke and shoots them down for saying something so trivial. But I don't like to do that one too much because we are in the business of creating a magical moment for people. But a line like that is priceless every once in while. Because in reality, most people know that we are using sleight-of-hand or simple facts unbeknownst to most people, much like most people know that pro-wrestling isn't real, or movies aren't exactly 100% true. But during that time we are watching any of those things, we like to suspend our disbelief and pretend for a moment, THAT moment, that it is real. So for someone to go ahead and wreck other people's moment, I feel free to kind of knock them down a peg. Nothing too far, like the above comment is perfect. Just enough to show them that everyone KNOWS that but for this moment they want to believe something else. lol. Once in a while, posturing is needed. This is where I can get a little carried away, so I always have to watch myself. lol. And I have heard some doozies!

There was an effect on the market that worked exactly as your spectator described.

And there is a similar effect which operates under the same principle (ie. red sponge ball to red square and black sponge ball)

I know that effect, but when you let them inspect each sponge ball for 5 minutes, that explanation really doesn't work. One guy apparently followed me waiting for me to perform it again, he grabbed all three from the table and ripped them in half because he was pissed, then walked away. I'm glad I had a bag of 50 at home.

I'm not entirely sure that the people who constantly scream, "I know how it's done!" is doing it just purely out of spite, lack of attention, low self-esteem or some other psychological dysfunction. Any person with above-average education and intelligence will not be content to simply let the unknown (for that's what magic seems to most laymen, a big gaping lot of unknown factors) overwhelm him. He wants - needs - a reason, something for him to cling on to before his worldview crumbles and collapses. At the risk of making an obvious statement, having one's worldview collapse is a very traumatic thing.

To quote from the Art of Astonishment:



He goes on to write about how the magician's role is not to be 'novelty entertainer, or super con-man or Mr. Ego', but to serve as a guide to the primal experience of astonishment. The tricks are merely tools to help your spectator return to the state of astonishment, because at the end of a trick, nothing but that moment remains. All spectators watching something they don't understand, even those that don't say anything, will have their intellectual minds racing to find some sort of explanation: but no matter what, that moment of astonishment will remain.