I went to bed at 5-6 am for a few years. I didn’t have sleep issues whatsoever. Slept the same amount and felt just as rested (more if anything) as now where I have a “normal” sleep schedule.
There’s also people that, yknow, work at night, whose sleep might start at 9.
I don’t think it’s necessarily wrong to not limit yourself greatly on what u do based on if someones sleeping as long as you avoid very loud things (a pleasant fuck you to my mom (love you) for insisting she needs to vacuum at like 11 every second day when I had that sleep schedule), but your reasoning is just bull.