I really understand that fear, and I do experience that with my outdoor cats. However cats tend to stick to their established territory and patterns and at least for mine, never go far and barely ever out of sight. In the summer being outdoor cats pretty much just means they sleep all day curled up in the garden.