I think the trouble with the conclusion you’re drawing is that it enables one to make sweeping statements about Muslims on the whole while maintaining plausible deniability in claiming that they’re only referring to “the bad ones.” In other words, sort of an inverse “No True Scotsman” fallacy.
Furthermore, I would wager that most people you’re referring to as “ex-Muslim” would still very much consider themselves to be Muslim, and even though you’re explicitly not addressing them in your claims, it’s not a huge leap that someone acting in worse faith would use your rationale as an excuse to generalize the entire demographic (including the so-called “ex-Muslims”).