I assume as this is a Gaming mode, its purpose is not to avoid waiting for features. But close the entire desktop which may use up to 1GB RAM and a by of CPU. Which definetly impacts the game by some fraction. Doesnt matter how tiny, its just what gaming modes are having as focus I assume.
The next thing I would never see on a desktop is FSR which gamescope has.