Speeding up Firewire File Transfers?

Milo_Mindbender asks: "I've got a pretty common problem: copying a ton of files from an old Windows XP computer to a new one. After noticing how long transfers were taking over my 100mbps Ethernet, I hooked up a IEEE1394/Firewire cable and things were much faster. Strangely though, Windows is still only using about 10% of the cable's 400mbps bandwidth. Does anyone know any tips/tricks for speeding this up or any Shareware mass-file-copy tools that would be faster than Explorer/file sharing? Right now, the older machine is setup with Windows file sharing and the new machine is copying from it, neither machine is using much CPU and the disks are nowhere near their max speed. The number and size of the files might be what's slowing it down, since it's gigabytes of files in the 100-200k size range."

Linux with NFS or maybe ghost?

[Anonymous Coward]

Get 2 knoppix discs and use nfs. Firewire and ethernet arn't whats slowing you down. I always get 92mbit transfer rate from my desktop (ext3) to laptop (ext3 and fat32). You do run the risk of losing file attributes though.

Is there any enterprise software you can use like ghost?

All in all Explorer sucks ass at file operations. You would probably get beter transfer rates with ftp.

Some things to try

[Tycho]

What is the manufacturer of Firewire controllers in the computers you are using? VIA controllers are usually not the best Firewire controllers. Texas Instruments controllers are usually better. For that matter, depending on the situation try compressing the files. Also, do not depend on time remaining in Windows I have found it wildly inaccurate at times. Windows seems to estimate the time remaining to be way too high, so YMMV, literally.

File size is the problem

[inio]

Your file size, and disk seek time, are the problem. Lets say your drive has a 5ms seek time (that's pretty damn fast). writing each file actually requires three writes: to the file allocation tabe, to the directory, and the contents of the file itself. Assuming the writes take another 5ms, that's 20ms per file. that limits you to 50 files per second. At 200kiB per file that's about 10 megs per second.

holy crap

[fishdan]

how is THIS not well known? MSFT blows. Thanks for the great tip

Try searching for an rsync clone for win32

[rwa2]

Well, I could think of a lot of ways to speed it up under Linux using various combinations of rsync, and... well, really just rsync. See if there's a good rsync clone for Win32 that will preserve your precious file attributes. Even running it under cygwin may be better in the long run, especially because inevitably (speaking from experience) your large copy will be interrupted halfway through by an "unreadable file" or some such rubbish, and you'll find yourself having to try to fix it and start the copy all over again from the beginning, or else trying to just transfer the remaining directories you think you're missing.

Using cygwin's rsync via ssh: (after running "ssh-host-config" on your new box and setting a "passwd" as Administrator )

rsync -azve ssh --progress /cygdrive/c/pr0n/ Administrator@newxpbox:/cygdrive/c/pr0n/

will do the trick, and you can just keep running it over and over again until all the files are mirrored. It will take a long time to buld a list of all the files you need to transfer, but it will only tranfer the files you're missing, and will attempt to do some compression (which should help because you're more IO bound than CPU bound, but just remove the -z if your CPU is pegged). Plus, you'll find rsync & scp damn useful for many other common tasks you take on.

The bottleneck is probably your windows filesystem, and cygwin's extra abstraction layer will only make that worse. But using rsync under cygwin means you only have to transfer the files once - which will be a much bigger time saver than trying and failing to do the entire transfer several times.

If you were doing this often, I'm guessing you might see an improvement if you defragment your old drive first, but you obviously don't really want to waste time on that for a once and final transfer.

Also, the Windows TCP/IP stack is typically tuned for 2 - 10Mbps links. Here's some information on how to fix that: http://rdweb.cns.vt.edu/public/notes/win2k-tcpip.h tm [vt.edu] It's mainly geared towards improving throughput on high-capacity WAN links, but parts are also relevant to achieving decent performance on 100Mbps+ networks as well. Also remember that a lot of network drivers suck too and are incapable of pushing the throughput even to a fraction of its rating... that's been a factor too, especially on cheap windows crap. An updated NIC driver /might/ get your net transfer to catch up with your firewire transfer somewhat.

Since you're getting 40Mbps / 400Mbps firewire, you're really not doing too bad. Converting to bytes, 5MB/s is a decent fraction of the 20MB/s to 50MB/s raw speed of your older hard drives, and actually seems reasonable given that you're sending lots of small files and not a few big ones where you can actually make good use of your drive's readahead cache.

Re:archive then move?

[mgv]

Have you tried to archive/compress them first [gzip/zip/etc], then move the big file over? Lots of small files take longer to move than fewer larger files.

Is it just possible that you are confusing bits with bytes per second? 400 Mb/s is about 40 MB/s (or pretty close, especially as you rarely the full theoretical 50 MB/s that you would think this would equate to).

Michael