Portal basically is an interface/backend for flatpaks to interface with toolkits & DEs. If you don’t use flatpak, xdg-desktop-portal and associated backends should be removable. Even if you do, try removing the gtk and gnome backends w/apt. Hopefully it won’t try to remove a ton of stuff due to dependencies. Then, reboot and see if the slow loading problem goes away. If it does, you can try re-adding one or the other and see if it comes back.
Does logging in take forever as well?
Also after some cursory research, some people have had problems with portal on Mint after updates as well, just like on Arch. So… definitely try it.