I’ve never tried to run multiseat the way you do here.
I do however succesfully run multiple computers in one chassi using kvm/qemu with pci-e physical passthrough on gpu and usb controller to my virtual fedora gaming machine (using vfio drivers in the host). Definitely more overhead than multiseat but I do enjoy the easy backup and restore I have on my gaming machine.
Level1techs.com has a ton of good information if you’re interested in virtualizing instead, such as forum.level1techs.com/t/…/119639