If you have a public IP and can forward ports, exposing SSH (with key-based login) is quite safe. You can browse the server storage and copy files to/from your phone.
If you can’t open ports you will need something that punches out of NAT and intermediates a connection to your phone. Simplest way is to use a service like Tailscale, you install and start it on both the server and your phone and they will see each other from wherever they are.