Speeding up Firewire File Transfers? 187
Milo_Mindbender asks: "I've got a pretty common problem: copying a ton of files from an old Windows XP computer to a new one. After noticing how long transfers were taking over my 100mbps Ethernet, I hooked up a IEEE1394/Firewire cable and things were much faster. Strangely though, Windows is still only using about 10% of the cable's 400mbps bandwidth. Does anyone know any tips/tricks for speeding this up or any Shareware mass-file-copy tools that would be faster than Explorer/file sharing? Right now, the older machine is setup with Windows file sharing and the new machine is copying from it, neither machine is using much CPU and the disks are nowhere near their max speed. The number and size of the files might be what's slowing it down, since it's gigabytes of files in the 100-200k size range."
archive then move? (Score:4, Informative)
Re:archive then move? (Score:1)
archive/compress them first [gzip/zip/etc], then move the big file over?
Pr0n jpgs do not compress very well. WTF not just let it run overnight?
Re:archive then move? (Score:4, Insightful)
Another thing is that even without looking at third party tools, you should be using XCOPY in preference to windows explorer.
There is an Exchange server utility that is optimised for moving gigantic files very fast; doubtless you can find similar programs about.
xcopy? (Score:2)
Re:xcopy? (Score:3, Informative)
See my earlier post about XXcopy, http://www.xxcopy.com./ [www.xxcopy.com]
Re:xcopy? (Score:2)
-Jar.
Re:xcopy? (Score:2)
Re:archive then move? (Score:5, Funny)
To put that in perspective, you would need to weld fourteen quadrillion VW Beetles end to end, then use the resulting Beetle Bar as a lever and an object with the displacement of eleven million Libraries of Congress as the fulcrum in order to give xcopy the same Windows command-line file copying power as Robocopy.
Re:archive then move? (Score:2)
Re:archive then move? (Score:2)
Better, use xxcopy [xxcopy.com]. Similar CLI, free; avoids the common problem of long/short file names getting scrambled. The "pro" version apparently has network features, but I've never used that.
Re:archive then move? (Score:2)
I much prefer xxxxxxxxxcopy.
Re:archive then move? (Score:2)
Perhaps you mean ROBOCOPY.EXE (RobustCopy)? It is a tool in the Wind2k/2k3 Admin toolkit designed to do just what you describe: Faithfully, accurately, (and efficiently) copy large files from one place to another.
Re:archive then move? (Score:2)
We used to run a Windows to Windows performance lab to collect file transfer statistics. FTP beat the pants off of any Windows networking thing, even XCOPY.
Re:archive then move? (Score:2)
I wonder how the microsoft backup too would work in this situation. Backup to a file, copy the file, then restore from it. I've done it in the past, though not for the same reason, and it's worked...not sure about speed.
Clearly, moving the disk would be the best option. I often use a firewire Wiebetech Drive (useful to have around if you often find the need to do such things) dock to perform a similar ta
Re:archive then move? (Score:2)
Re:archive then move? (Score:3, Interesting)
Is it just possible that you are confusing bits with bytes per second? 400 Mb/s is about 40 MB/s (or pretty close, especially as you rarely the full theoretical 50 MB/s that you would think this would equate to).
Michael
No, archive WHILE moving! (Score:2)
Also, booting the machine from a Linux boot CD and mounting the drive read-only and using tar to move the files is probably the fastest method. I do this all the time to recover/backup machines at work.
boot gentoo install cd...
mount network share/external drive
mount local drive '-o ro'
cd
tar -vcf
Move the old hard drive, then copy (Score:4, Informative)
Re:Move the old hard drive, then copy (Score:5, Informative)
Do you mean the software named "Magic Folders" [google.com]? Or perhaps you have some kind of Folder of Holding [wikipedia.org] with compression created by a high-level Magic User. Most likely you mean the Special Folders [wikipedia.org] that are used by Windows, but then again you may just be spouting about something you actually know little about.
Or maybe you should just explain yourself and not flame the mods... You might even get modded +Insightful or +Informative then and you would have the advantage of explaining your term to the person you were answering - thus being genuinely helpful.Re:Move the old hard drive, then copy (Score:2)
If you've got a system you don't care about, and another system disk you were planning on wiping anyway, install the second disk and try various copy/delete/move operations between the original special folders and the newly installed ones on the second disk. Use Explorer, magic is not quite as strong using xcopy from the c
Re:Move the old hard drive, then copy (Score:5, Informative)
I looked in TweakUI and it calls them "Special Folders" as well. My point is: Be careful about how you present things that you may only have cursory knowledge of here on Slashdot. I was being funny about it, but there are planty of users out there who will ream you for bad information. Instead of being authorative and telling the mods what to do, you could have replied in the form of a question such as "You could try, but won't windows have problems with the magic folders?" or something like that. Further, you could disclaim being an authority and just post something along the lines of "I think that...".
Ok, I'm done being a slashdot post nazi now :D
From my experience, XP stores the hard locations (ie: c:\Documents and Settings\BrynM\My Documents") for the special folders it needs in the registry. I've never had a problem slapping a previously used drive into a machine to copy files (I did this exact thing to recover files after and IDE failure that was corrupting NTFS just a couple of weeks ago). If you've had problems or know of them, then please post that.Re:Move the old hard drive, then copy (Score:2)
Re:Move the old hard drive, then copy (Score:2)
Mods: If you don't know what "magic" is, please mod some other comment. Thank you.
Normally I avoid posting on the techie parts of slashdot, as I know approximately fuckall in the power/knowledge sense when it comes to the esoteric hardware and coding problems that are usually discussed. I just read, hopefully learn something, and avoid sticking my foot in my mouth. But, there was something about this that really rubbed me sideways in an uncomfortable "AAAHHAGH, you're rubbing be with stingling nettles!!
Re:Move the old hard drive, then copy (Score:2)
I replied further upthread to the guy who knew what I was talking about. What I *really* hoped would happen was that a few hackers would try to find out what I (and Microsoft) meant by "magic", but on Googling right now, it is not as obvious as it used to be in 1995, when the magic was first conjured. Here be dragons:
http://en.wikipedia.org/wiki/Special_Folders [wikipedia.org]
Re:Move the old hard drive, then copy (Score:2)
Sorry I was cranky. I can understand not wanting to be burned by roving retardo-mods. Thanks for the info, it was helpful!
Re:Move the old hard drive, then copy (Score:3, Informative)
If you want to know where a folder really is on the drive, just do a search for a file you think is in the drive.
Re:Move the old hard drive, then copy (Score:2)
You got beat down in high school? I'm sorry...
I didn't have that problem because all those bitches knew I was speaking the truth when I told them I was "down with the AK" (ala Ice Cube)
Re:Move the old hard drive, then copy (Score:2)
i had a problem once copying an entire user directory, which totally screwed up a machine because it suddenly had magic folders appearing and disappearing
Re:Move the old hard drive, then copy (Score:2)
Aha! You did have problems. Please post what happened so the rest of us may avoid it. Even if you weren't sure what was going on when it happened, you might get replies that help clear things up. I hope you don't think I'm nitpicking you, I'm just trying to help you and I have the Karma to burn on helping other posters (modded to oblivion for being offtopic). Believe it or not, I've learned quit
you don't need those old files anyway. (Score:2)
Well, then don't put anything you want to transfer between drives in the recycle bin.
I've done this type of copy--physically move the source drive into the computer with the target drive--many times with Windows 98, 2000, and XP. I've copied files and folders to and from 'My Documents', 'Desktop', 'Application Data', and other special folders. I've never had an issue with this approach.
Whatever.
Here (Score:5, Informative)
Firewire is crippled in Windows by default. You need the patch here [microsoft.com] to restore functionality.
Re:Here (Score:4, Funny)
holy crap (Score:2, Interesting)
Re:holy crap (Score:2)
http://en.wikipedia.org/wiki/Firewire#Operating_s
Re:Here (Score:2, Informative)
Article seems to imply that only 1394b (FW800) controllers are affected. Still, limiting to S100 is a pretty dumb move on MSFT's part (insert tinfoil rant about MS being in league with Intel over FW vs USB2 here).
As a side note, I use an external FW drive on my HP laptop running XPSP2 Pro, and I can pull >35MB/s from it. I do not have any machines with 1394b so I can't confirm that there is an issue with that, but my 1394 port is definitely not running at S100.
Re:Here (Score:2, Informative)
Until Microsoft gets a truthful disclosure I refuse to install it.
Why bother with a patch? (Score:2)
Why not just boot Knoppix or some other CD with a driver that does not suck?
"Performance of 1394 devices may decrease..." (Score:2)
seriously, though, thanks for the pointer; I'll be (re-re-re-re-re-re-re...-re)patching my XP box tonite!
Re:Here (Score:3, Funny)
It's the hard drive, not the fire-wire (Score:1)
To get full firewire transfer goodness, you need a raid of fast drives, on both systems.
Re:It's the hard drive, not the fire-wire (Score:3, Insightful)
You are confusing MByte/s and MBit/s. Firewire is 400 MBit/s, while SCSI is 320 MByte per second.
Re:It's the hard drive, not the fire-wire (Score:5, Informative)
A modern SATA drive can do just shy of 70 megabytes per second, which is 560 megabits.
Re:It's the hard drive, not the fire-wire (Score:4, Informative)
They're just about as fast as SATA drives, since ATA-100 is still faster than the sustained speeds of the drives (100 is megabytes in this case). This is why ATA-133 never caught on -- it's faster than any of the drives you'd connect it to. It wasn't SATA's speed that made it popular, it's the numerous other advantages (thinner cable, cheaper, hotplug, etc).
Re:It's the hard drive, not the fire-wire (Score:2)
Any modern IDE drive (40G+) should be able to sustain 40 - 50M/s for sequential reads and writes.
Re:It's the hard drive, not the fire-wire (Score:2)
Re:It's the hard drive, not the fire-wire (Score:2)
hmm...
$ dd if=/dev/zero of=zero bs=1M count=10K
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 161.013 seconds, 66.7 MB/s
Looks pretty sustained to me.
Re:It's the hard drive, not the fire-wire (Score:2, Informative)
I'm not sure why you're transfers aren't that fast, for me firewire from my external harddrive is just as fast and getting stuff off my fileserver *6 disk raid 0*. I have yet to
Re:It's the hard drive, not the fire-wire (Score:2)
Sleep on it. (Score:3, Funny)
IF you wake up. Muahahahahahaaa....
Find the bottleneck (Score:2, Informative)
Re:Find the bottleneck (Score:2)
Since the poster already mentioned CPU, I suspect they know enough to look at the basic utilization stats. Most likely, however, the limiting factor is either the hard drive speed or the fact that Windows explorer is a piece of crap.
What are you moving now? (Score:4, Funny)
That's quite a collection of pr0n!
Re:What are you moving now? (Score:2)
That's quite a collection of pr0n!
You've obviously never seen my pr0n archives. Which reminds me, anyone know of good prices on RAID cards? Newegg maybe?
Some things to try (Score:2, Interesting)
Re:Oxford even better (Score:2)
That's not a FireWire controller. Oxford makes bridge chipsets from FireWire to ATA. A FireWire controller generally refers to the bridge between an peripheral bus and FireWire, e.g. PCI to FireWire. My advice: TI chipset. Accept no substitutes. In particular, though, avoid NEC from what I've read.
As for the performance problems, if you have SP2 and FW800 ports (or a controller that's FW800-capable under the hood but uses FW400 PHYs), make sure you install the FireWire hot patch [microsoft.com] from Microsoft or Fir
File size is the problem (Score:5, Interesting)
Re:File size is the problem (Score:2)
You're asking Windows to create, write, and close maybe 500 files per second. Windows file creation isn't that fast. What's the file system format on the destination side?
Try transferring a 1GB file and report how long that takes.
FileSystem is the problem (Score:2)
The key feature here is lazy allocation. It not only keeps your drive from getting as fragmented, it also means that when it does decide it has to write, it's writing all the files at once, and can make intelligent decisions like, write all the metadata out, then write all t
Re:It's a good thing you don't write Windows... (Score:3, Informative)
Re:It's a good thing you don't write Windows... (Score:2)
Windows can't delay metadata writes on FAT32. See my comments earlier about synchronous metadata writes in FAT32 on Windows due to the whole floppy disk legacy.
Re:It's a good thing you don't write Windows... (Score:2)
Tar and Robocopy.. (Score:2)
If you don't mind the files being in one glump.. use a Win32 port of Tar..
Both options above seem to speed up Firewire (or any) transfer.
Try searching for an rsync clone for win32 (Score:4, Interesting)
Using cygwin's rsync via ssh: (after running "ssh-host-config" on your new box and setting a "passwd" as Administrator )
rsync -azve ssh --progress
will do the trick, and you can just keep running it over and over again until all the files are mirrored. It will take a long time to buld a list of all the files you need to transfer, but it will only tranfer the files you're missing, and will attempt to do some compression (which should help because you're more IO bound than CPU bound, but just remove the -z if your CPU is pegged). Plus, you'll find rsync & scp damn useful for many other common tasks you take on.
The bottleneck is probably your windows filesystem, and cygwin's extra abstraction layer will only make that worse. But using rsync under cygwin means you only have to transfer the files once - which will be a much bigger time saver than trying and failing to do the entire transfer several times.
If you were doing this often, I'm guessing you might see an improvement if you defragment your old drive first, but you obviously don't really want to waste time on that for a once and final transfer.
Also, the Windows TCP/IP stack is typically tuned for 2 - 10Mbps links. Here's some information on how to fix that: http://rdweb.cns.vt.edu/public/notes/win2k-tcpip.
Since you're getting 40Mbps / 400Mbps firewire, you're really not doing too bad. Converting to bytes, 5MB/s is a decent fraction of the 20MB/s to 50MB/s raw speed of your older hard drives, and actually seems reasonable given that you're sending lots of small files and not a few big ones where you can actually make good use of your drive's readahead cache.
Pay attention to units (Score:4, Informative)
Are you sure you aren't confusing mbps [wikipedia.org] with MBps [wikipedia.org]? 400mbps is equal to 50 megabytes per second, and "12.5% of the cable's bandwidth" sounds suspiciously like your description of the problem, "about 10% of the cable's bandwidth".
Re:Pay attention to units (Score:3, Funny)
Are you sure you aren't confusing mbps [wikipedia.org] with Mbps [wikipedia.org]?
Last I checked, 400mbps is equal to 0.4 bytes per second. I remember getting speeds like that back in the days of dial-up. Like back in the day when you picked up the phone and read off ones and zeros to your friend on the other line as he copied your fortran program by punching holes in his punch card. Oh yeah, and uphill both ways. In the snow.
; )
[/
Re:Pay attention to units (Score:5, Funny)
Re:Pay attention to units (Score:2)
[*] Assuming you meant an analog millenium, not a digital millenium, which would be MiY.
Makes sense (Score:2)
Some ideas (Score:2)
As far as copying faster. You might want to try robocopy from the Windows 2003 resource kit [microsoft.com] or xxcopy [xxcopy.com]. I've tried xxcopy and it seems to buffer things well, such that I can do a sustained 25 MB/s or so when backing up files to my 500 Gig
why use firewire? (Score:2)
You'd still need to use something like xcopy32 (or boot in linux and use tar - if both drives are fat32)... or find a windows version of tar (url:http://unxutils.sourceforge.net/)
Bits and Bytes (Score:2)
Similar Problem on Macs (Score:4, Funny)
(Admit it. You knew this [wikipedia.org] was coming.)
Re:Similar Problem on Macs (Score:2)
Re:Similar Problem on Macs (Score:2)
Re:Similar Problem on Macs (Score:2)
I say, EXCELLENT!!!
Cheers,
ElGanzoLoco
Have you tried this? (Score:3, Informative)
Gigabit? (Score:2)
The fastest way to do this is to put the old drive in the new machine (or perhaps an external drive enclosure if we are talking about a laptop) and copy that way.
If you are worried about special file or folder attributes then use MSBackup to copy the drive to a backup file as it will preserve everything.
Why use Firewire? (Score:2)
Re:Why use Firewire? (Score:2)
Optimize for performance (Score:5, Insightful)
Bring up the properties of the firewire disk in "Device Manager". Go to the Policies tab and make sure it's set to "Optimize for performance".
Re: (Score:2)
Re:Optimize for performance (Score:2)
NSCopy (Score:2, Informative)
If you want more speed, I'd say get FireZilla (an FTP client) and FireZilla Server (an easy to use FTP server), both open source and free. Set up the server on the "source" computer, and download as fast as you can! It will use the bandwidth much better.
One of the other suggestions about moving the hard drive would
Your calculations are off (Score:2)
One option is to just pull the drive from the old machine and use it as the slave drive. I use this when moving large files. Another option is to have a gigabit card, now around $14 everywhere. Newer PCs already have gigabit cards. Just use a crossover cable if you wont buy the (also cheap) gigabit switch.
As far as firewire is concerned, I've never used it to tra
When I transfer files between computers... (Score:2)
initiated from files; location: /someplace ; tar xvf - )'
tar cvf - file1 file2..filen | ssh user@host '( cd
initiated from files' destination: /someplace ; tar cvf - file1 file2..filen)' | tar xvf -
ssh user@host '( cd
Not exactly a new trick but one that bears repeating. You get prompted for a password a
Math (Score:2)
Synctoy (Score:2, Informative)
Whitepaper: http://www.microsoft.com/downloads/details.aspx?fa milyid=49818CF1-2287-40EA-8A6F-57BD8695F23D&displa ylang=en [microsoft.com]
Download:
http://www.microsoft.com/downloads/details.aspx?fa milyid=E0FC1154-C975-4814-9649-CCE41AF06EB7&displa ylang=en [microsoft.com]
FIREHOSE tool (Score:2)
"Unlike RAID striping, FIREHOSE striping load balances the network devices so every ounce of bandwidth is utilized. Combine a 400Mbit firewire eth device with a 100Mbit eth device to get 500Mbits of power. Combine 10 100Mbit ethernet ports for a gigabit pipe. The number of dev
Re:Use FTP (Score:1)
Re:Use FTP (Score:3, Informative)
user@destination $ netcat -l -p 12345 | tar -x
user@source $ tar -c
Re:Use FTP (Score:2)
What I actually usually do is run a pass of rsync over the directories. If the files are the same, rsync will do very little work, if there is anything needs fixing, it will get done.
Re:Linux with NFS or maybe ghost? (Score:3, Funny)
Even IP Over Carrier Pigeons [ietf.org] would yield some sort of improvement.
If you’re gonna use an operating environment that shits all over your desktop anyway, ya might as well go all out.
Linux with SCP (Score:2)
Re:Many Major Problems: (Score:2, Redundant)
Firewire 400 is 400 mbit/s. A modern 7200 rpm SATA desktop drive can sustain just shy of 70 mbyte/s, which is 560 mbit/s.
Re:Many Major Problems: (Score:2)
Just for the first 50 gb or so.
Re:Many Major Problems: (Score:2)
Re:Many Major Problems: (Score:2)
Re:Linux is Faster (Score:2)
50% Troll
50% Offtopic
Topic is "speeding up FireWire transfers". I have done that by using Linux instead of Windows. I told my experience in detail, including how to switch.
TrollMods love Windows, or maybe just going slow.
Re:Linux is Faster (Score:2)
Re:Parent perhaps not Troll (Score:2)
You know, an Anonymous Coward actually able to understand the info in a post, rather than just jumping into OS flamewar mode when seeing a point about OS choice affecting the task.
Oh, right - that's unheard of on Slashdot. I must have you con