You are talking about what is visible on the surface right now.
When the idea of girls’ sports teams was first considered, men didn’t think of girls as wanting to play sports either; sports were for men/boys. There were people making the same argument as you “there aren’t enough girls interested in sports to form a team” etc.
Just because you only know a very small number of trans people doesn’t mean there aren’t more, and doesn’t mean there won’t be even more in the future.