My general guideline is to ignore what I don’t like, downvote what I find harmful. Of course, the first one is emotionally motivated, while the second is rationally so; and therefore doing this requires a conscious choice. So, when I’m too tired, I might fail at doing this. But that’s the intent anyway.