It gets really complex when your time traveling triggers an infinite time loop that you personally never experienced.
Example: You go back in time to warn yourself about a coming war or disaster, but you get interrupted before you can finish, so your other self panics and disaster proofs everything, unwittingly preventing the disaster. When the “war or disaster” never happens, you feel silly and stressed, so you go back in time to tell yourself not to worry so much.