So your de-dupe ran for a week before you cut it out? On a modern CPU, the de-dupe is limited not by the CPU speed (since deduplication basically just checksums blocks of storage), but by the speed of the drives.
What you need to do is put all this data onto a single RAID10 array with high IO performance. 5TB of data, plus room to grow on a RAID10 with decent IOPS would probably be something like 6 3TB SATA drives on a new array controller. Set up the array with a large stripe size to prioritize reads (writes are going to be 'fast enough' on a RAID10, trust me). Once you have that hooked-up with your files copied onto it, you want to connect the drive to an OS that can natively deduplicate, like Windows Server 2012. If you must, you can set this box up as a storage server (with a low-end CPU, an old 'Core 2' should be able to keep up with 180MB/sec I/O), and keep your workstation separate. Reading this entire array (when full) through the CPU -should- take about 6-10 hours, deduplication will take slightly longer.
If you don't want to do deduplication at the block level, and you want to actually only have one copy of each duplicated file, you'll need to write scripts that do something like this:
1. Run through the data store and checksum each file (except for those ending in ".mychecksum" with AES128.
2. For each file, create an empty file named .."mychecksum" next to it. This will create the 'index' using the filesystem, which will be MUCH faster than having to read the data from inside each file.
3. Search through the store and concatenate all the ".mychecksum" files into a single CSV.
4. Run sed+unique on the file to see what will be nixed (i.e. Get a report)
5. Create another script that actually takes the output from step 4 and deletes ONE of the duplicate files. You can test by -renaming- ONE of the files to .deleteme and then deleting all those files after you confirm that it worked.
6. Repeat as necessary, possibly with a scheduled job.