The only reason I can imagine people actually caring enough to look is if you’re playing with sound enabled in a public place, put your headphones on if so. No one wants to hear the sound from someone else’s phone.
I guess the only other thing I can imagine is if you get too into it, maybe? If you’re frantically tapping your screen, that’s gonna draw attention too.