He doesnt get his past memories. There is no such thing. He dreamed of himself doing what the past him was doing but to the present him, it was a dream. Much like we dreamed of doing something wrong to someone but we dont mind it too much since we didnt do it in reality.