I agree. I’d say that expressing a viewpoint that derails the conversation and people already know is wrong definitely does not add to the conversation, and should be downvoted.
Downvotes really shouldn’t be for comments or submissions that you disagree with, but for anything that does not add to the conversation. Reddit started off with this guideline too, but at some point votes turned into agree/disagree.