Yes, that’s the fascinating thing. Using labeling as a mental shortcut for understanding the world is really useful, but it comes with a price tag.
It’s basically the same problem we have when labeling thins as “religion” or “some other stuff”. We might want to call something a religion, but it doesn’t quite match. We might want to label something else a non-religion, but it meets all the criteria. Those labels aren’t neutral either, so using them comes with some baggage.
Same thing with FOSS. If we label it a socialist concept, that label comes with some unfortunate connotations… Well, at least if you’re in a country where socialism is frowned upon.