We use Teams and if I notice that I’ve been on the yellow idle icon for too long I’ll give my mouse a jiggle, at which point I see all my colleagues are on idle too… it’s a very relaxed environment. I hope you can continue working remotely in the future!
I’m a remote employee in a similar position, I also finish my work in about 3-4 hours in any given day (mileage varies). My performance reports have been at a consecutive 100% and I don’t see why I should change my ways. Honestly I don’t give a damn if people think I’m slacking off if I go and do some gardening during the day, if anything finishing my work early just makes me very productive :)