Actually, memory and the perception of direction of time are fixed.
So you know that entropy in the universe is always increasing.
You also know that in order to create memory, one must use energy to record that memory. This can be rearranging magnetic tape from random state to an ordered state, or rearranging your neurons to for memory. Any action that creates memory (or reduce local entropy) uses energy and thus increase global entropy.
Ok, so, in order to remember the past, your perception of time must be moving in the direction of increasing entropy.
Now imagine the direction of time is reversed, and think about what you'd remember going backwards in time.