I upgrade when the opsys gets hopelessly outdated (as in apps no longer supporting it) or the device physically breaks. My last phone (Huawei Ascend P7) lasted for 7 years, but the Android 4.4 got just a bit too old, plus I cracked the screen a month after removing the battered to hell glass screen protector...
I don't care much about the phone not getting OS updates since I don't keep anything important on a phone in the first place and I don't care much about CPU/GPU performance since I don't run intensive apps on my phone—that's what my desktop and server are for. My current phone I bought last year will last probably for 5 more years.