you can't both upvote and downvote the same item. the last option you selected is the only one that's counted at any given moment.
edit: wait, i see. if they're separate comments, you could just upvote the comment you like and downvote the comment you don't. i was assuming one comment that said upvote or downvote.