40% increase in traffic might actually be a decrease in the amount of files, if you consider that more and more are 3D/3D/Lossless, ...

Found an easy solution at http://elonen.iki.fi/code/misc-notes/remove-duplicate-files/

OUTF=rem-duplicates.sh; echo "#! /bin/sh" > $OUTF; find "$@" -type f -exec md5sum {} \; | sort --key=1,32 | uniq -w 32 -d --all-repeated=separate | sed -r 's/^[0-9a-f]*( )*//;s/([^a-zA-Z0-9./_-])/\\\1/g;s/(.+)/#rm \1/' >> $OUTF; chmod a+x $OUTF; ls -l $OUTF

This solution is based on the md5 checksum, it is probably the fastest and least reliable of checksums; sha256 or sha512 would be better but will require much more time to run.
This solution requires you to run some version of linux (a knoppix or ubuntu live CD should do).

In my opinion, patience is your tool of choice. The issue with the program you tried, is that you won't know if it works well before you use it - so maybe you might want to search info on how it runs, so you can make sure it will sort your files as you wish. The way I would (will actually, cause I'm heading towards the same problem) is to make a simple script that calculates checksums for all files - and then find dupes based those checksums. I would choose this approach for two reasons: 1. The computer can't understand the contents of files - so there is no point checking it. 2. Checksums are faster to compare than files (though they are longer to calculate - but probability shows that is the way to go as you will need to make all possible comparison between pairs of 4.8million elements - or 4.800.000 (close to 250 000 000 000). Calculating checksums is a CPU intensive task, so the faster the cpu, the faster it will go - depending on the most acceptable factor (time or errors) are the checksums methods you can use. Checksums are not perfect and to reduce errors, you might want to use two checksum algorhytms, although the time required will more than double. I come back to my conclusion, patience, waiting one week was nice, but seen the amount of data you need to sort, I wouldn't be surprised to see the computer running 6 months (or 2 years) to do the sorting.

Python? PHP? they must infringe some patents somewhere... be a fruit ninja, ban the apples!

After over 20 years of shitty operating systems no wonder they need a toilet \o/

My sarcastic answer wasn't at all oriented towards your post but rather toward the actual patent system.

If inventors --- sorry - investors --- keep on using patents like they are ( shape - weight - size ), then we shouldnt be surprized they start to patent colour - material and why not natural states: 'My tablet is a solid: his should be liquid or gas to be different!'

If it's like motorola defy ( http://www.motoroladefy.eu/ ), then i''d rather stick to HTC :D

This is forgetting that the iPad 2 has exactly the same size and colour as the plastic board I used to cut bread for years before it came out!