Yes. X11 these days usually auto-configures on its own (to my understanding, at least) - when you generate one with Nvidia’s settings it will add some stuff that is specific to the Nvidia driver, and thus once the card/drivers isn’t present, then X11 can’t start.
I had removed the drivers before swapping out the card in preparation, so I’m not 100% sure if said proprietary extensions doesn’t load because of the lack of drivers, or the lack of the card itself - probably both to be honest.
But either way, X11 wasn’t affected by the removal of the custom config, and there wasn’t ever one present until I made one via nvidia-settings (other than, it started working of course).