You’re not mistaken, it is definitely possible with at least RSA, though, I would guess it may not always be possible. It also sounds like it’s still a bad idea unless you know all of the parameters used to generate the keys and can be sure what information is actually encoded in the keys.