You’re not adding anything that wasn’t argued towards before. Soon or later, you have to trust something. There are ways to transfer keys by other means which you can use to corroborate.
The tradeoffs of this idea are just not worth it for 99% of the people.