I generally start looking to replace mine around the time that Google Maps starts becoming laggy. That's usually around the 3 year mark for me. After 4 years things get pretty bad.
Nexus 5 -> Pixel 2 -> Pixel 6a
Practically every app update grows its respective compute and memory footprint. And over time, it adds up. Combine that with the big jumps in resource usage that come with OS updates, and eventually things just start slowing down.