Perhaps you think too low of yourself? There will always be people who will prefer you to think this or that. Your parents, some dickhead politicians, your peers, … You don’t need everybody to like what you think.
Does the thought of the person, that you plan to downvote, crying themselves to sleep make you happy? Then you should downvote them. I think that’s what they wanted to say.