Stress testing hard disks is a particular bugbear of mine, after having some really bad luck with early hard disks. Over the 15 years that I've been doing it I've had to send back loads of hard disks and flash cards because they failed my tests, either breaking completely or returning single bit errors in your data. Mostly the manufacturers will take disks back if you can get their stupid Windows program to return an error code. Sometimes it takes a bit of arguing but ultimately the manufacturers want to keep you happy. Flash disks with single bit errors are the hardest to send back in my experience.

Here is the latest generation of my stress testing code (re-written in Go recently): https://github.com/ncw/stressdisk

(Interestingly the stressdisk program sometimes finds bad ram in your computer too!)

I generally thrash every new hard disk or memory card for 24 hours to see if I can break it before trusting any data to it!

I also run a long smart test too.

Somewhat paranoid, yes, but I really, really hate losing data!

I use Photobox ( http://photobox.com/ ) for this purpose. They are cheap and quick, but only in Europe. They also allow you to upload photos with FTP rather than some stupid application which is really really convenient!

That is a good thought, and photorec does an excellent job of finding pictures and videos by searching through your sectors - definitely worth a try.

http://www.cgsecurity.org/wiki/PhotoRec_Step_By_Step

I'd be asking myself why lots of files became corrupted from one dodgy file system event. Assuming HFS works like file systems I'm more familiar with, it will allocate sequential blocks for files wherever it can. This means that a random filesystem splat is really unlikely to corrupt loads and loads of files. You might expect a file system corruption to cause a load of files to go missing (if a directory entry is corrupted) or corrupt a few files, but not put random errors into loads of files.

I'd check to see whether files I was writing now get corrupted too. It might be dodgy disk or RAM in your computer.

The above might be complete paranoia, but I'm a paranoid person when it comes to my data, and silent corruption is the absolute worst form of corruption.

For next time, store MD5SUM files so you can see what gets corrupted and what doesn't (that is what I do for my digital picture and video archive).

I was going to hold this back until the competition was finished, but it seems the cat is out of the bag!

Here is my solution and a writeup of exactly how I got there.

http://www.craig-wood.com/nick/articles/how-i-solved-the-gchq-challenge/

I agree!

My personal limit is 10 lines for a shell script. If is longer than that I convert it to Python.

Python scripts have the advantage that they work on Windows too, and they have lots of os independent abstractions for file names, processes etc.

Why learn an arcane language like sh when you can learn a nice well structured language like Python and write better scripts?

A few years ago I would have used Perl rather than Python, but I'm converted now ;-)