"Ginger" as a term is not, in itself, derogatory or hateful in my experience.
Describing gingers as soulless or hot-tempered is about the same kind of destructive as describing blondes as stupid, which is to say it's a silly stereotype that's often the territory of playful insults between friends, while some small minority of people do run it into the ground and cause real hurt.
(This might be exacerbated by tensions between England and Ireland in that specific area, but... for most of the world "ginger" is a pretty harmless thing.)