This is actually a pretty big assumption for some of the more outlandish claims. When you dig in a bit, you find many of them are kind of repetitive tropes and generally come from Senatorial-class writers about emperors whose policies were less friendly to that class.
Not to say they couldn't have happened, and even in those days better to exaggerate than to invent from whole cloth, but it seems like an unusually large number of "bad" emperors had the same kinks as each other or the same traits that the literature had always considered "unmanly."