Sounds like you’ve been very unlucky. Even the open-source Nvidia driver should work out of the box and look OK. Performance is ass, but it’s good enough for a usable desktop experience (usable enough to install the proprietary nVidia driver, which at least on Ubuntu’s are just a few clicks in the GUI)
Instead of going Fedora, try PopOS. PopOS has a special ISO for nVidia graphics. Trying to “install” the Nvidia driver yourself on a live USB boot is not the way to go. I doubt it’s even possible.
I’ve been on (K)Ubuntu, and XBox controllers have literally just been plug and play. I could even use the KDE game controller settings page to compensate for the drift in my left joystick.
Another option is Bazzite, which is a version of Fedora Immutable (“Silverblue”) that comes with all the bells and whistles for gaming, including Nvidia drivers. However the immutable part may or may not be to your taste.