Start with the 11th Doctor.
It's a sort-of fresh start and while it does reference the past Doctors, if you're interested in knowing more - look up the episode on Wikipedia and I'm sure a bigger nerd will have linked to the past episode.
The biggest things that came back from past episodes were the Steven Moffat episodes from the previous seasons - those might be worth viewing just to catch up and also to see how good Doctor Who can be if in the hands of a good writer.
Previous Steven Moffat stories:
Series 1:
1.9 "The Empty Child"
1.10 "The Doctor Dances"
Series 2:
2.4 "The Girl in the Fireplace"
Series 3:
3.10 "Blink"
Series 4:
4.9 "Silence in the Library"
4.10 "Forest of the Dead"
Just listing those episodes reminds me of a lot of the stuff introduced in them that are referenced in later episodes.