Just keep in mind that security through obscurity is not considered secure in itself.
Do you consider it to not be a helpful measure to take at all?
I have fail2ban configured - since it is reading from the auth.log, I guess I would not have to make any changes to the configuration there to have it work with a new port?