I don’t want to step on your workflow too much since it somehow seems to work for you but your main issue stems from the fact that you clearly don’t work with your server as if it actually was a server.
You shouldn’t really have a desktop interface running there in the first place (let alone as root and then using it as a regular user). You should ask yourself what it actually solves for you and be open to trying different (and more standard) solutions to what you’re trying to achieve.
It’d probably consist of less clicking and using the CLI a bit more, but for stuff like file management you can still easily use https://linux.die.net/man/1/mc.
If you need terminal sessions that keep scrollback and don’t stop when you disconnect you should learn to use tmux or screen or something like that. But then again if you’re running actual software in there then you should probably use a service (daemon) for that.
As for whether it’s a security issue, yeah it most definitely is. Just like it’s a security issue to run literally any networked application as root. Security isn’t black and white and there are trade offs to be made but most people wouldn’t consider what you’re doing a reasonable tradeoff.