The physics of time travel, I'm more confused than humanly possible?

There is no "physics of time travel". Time travel is a fantasy, and will always remain so. You don't seriously think there is any factual basis to all that television / Hollywood crap, do you?