Most of the time the downvotes I would get on Reddit are on topics where I disagreed with a couple of the early people and they downvoted. This would cause a cascade of people piling on. Other times I could make essentially the same comment and get lots of upvotes.
Not a big deal either way, it is boring to always be aligned with everyone else as long as you are open to constructive discussions about it.