The problem with a lot of file duplication tools is that they only consider files individually and not their location or the type of file. Often we have a lot of rules about what we'd like to keep and delete - such as keeping an mp3 in an album folder but deleting the one from the 'random mp3s' folder, or always keeping duplicate DLL files to avoid breaking backups of certain programs.
With a large and varied enough collection of files it would take more time to automate that than you would want to spend. There are a couple of options though:
You could get some software to replace duplicate files with hard links. This will save you space but not make things any neater - DupeMerge looks like it would do it on NTFS but I haven't tried it myself.
Another alternative would be to move your data to a file system that has built in de-duplication such as ZFS and let that handle everything.
Finally when I was looking at this myself what I found was that the problem was not individual duplicate files but that certain trees of files occurred identically in multiple places (adhoc backups of systems were a big culprit here). What you could do with but which I couldn't find and didn't get round to finishing writing was something that would CRC not individual files but entire trees of files/folders and report back the matches. If something does already exist to do that I'd be quite interested myself.