So first, let me be clear - I don’t know if an alternative to that software you first brought up. But some of our earlier CTFs had a similar issue with isolation.
We ended up spinning up new VLANs per contestant, each having a single Kali Linux VM with xrdp, along with each contestants target systems. Our router/fw blocked all access in/out of those VLANs, save for RDP/SSH traffic from our Apache Guacamole server on the DMZ.
So contestants would hit our portal (Guacamole), then from there connect into their own dedicated Kali instance and environment.
Later, we had to make additional fw exemptions for our scoreboard/docs, etc.
I throw CTFs for a living (among other things), and I’m happy to help out a fellow Infosec person.
What kind of infrastructure can you deploy? Is this going to be in the cloud, on-prem (via a hypervisor like Proxmox/vSphere, or hosted on a single laptop/server?