Though I don't think that's too big a loss when you're using it to sleep. If you used time stop for something that was both exhausting and wasting your time, that'd probably be worse honestly.
It would certainly become noticeable after awhile since as you said, it's one third of your life.