Every time you ask a question, or give an answer, you'll see that on the left there is a number with an arrow pointing up and an arrow pointing down.
This is basically a +1 / -1 or thumbs up / thumbs down. The number that you see after your name is all of them added together. So, as of the time I'm writing this, all the questions and answers you've asked or answered have gotten 5 more thumbs down than thumbs up, so it's -5.