Matrix would be fantastic. Phone number accounts hosted on the carrier’s instance, and can federate with other instances.
Something like (for an American phone number): @0014204206969:matrix.att.com and have each carrier share which phone numbers are assigned to which carrier in some common protocol so all you would need is a phone number.