It needs to be running all the time you need its services. If it’s not running it can’t serve you music.
Modern PCs and laptops don’t have an issue with that, even with Windows 10/11.
But a Raspberry Pi with a large HDD (with its own power supply) attached would be perfect IMO.
This is done to keep employees from sticking in unknown thumb drives that could install malware. Several critical systems on protected networks have been hacked in the past by leveraging human curiosity and placing a compromised thumb drive on the ground in the companies parking lot. Gluing shut the USB ports is a simple defense against that.
Yes, I had an MP3 player before I got my first smartphone.
Or, if you’re asking about apps, I used Vanilla Music Player. But nowadays I either use Spotify or a cassette player.