Another option you could consider is rather than running the VPN on the pi and downloading through that, to run the VPN on your media server as you planned and instead use the pi (or your router if you can run openwrt) as the remote access point. Then you only need to worry about the performance needed for remote access
I’m not sure how tailscale works, but this is what I do with zerotier (i.e. run it on my router).