Fully Open Source NTFS Support Under Linux

lord_rob the only on writes "The Linux NTFS project has released a beta version of its fully open source userspace (using FUSE) 3G-Linux NTFS support driver. According to the developer, this driver beats hands down other NTFS support solutions performance-wise (including commercial Paragon NTFS driver and also Captive NTFS, which is using windows ntfs.sys driver under WINE)." That's right, writing to NTFS even works. Soon it'll mean one less recovery disk to keep around, I hope.

Great news.

[LinuxGeek]

This gives us another tool that can be used to repair windows systems that have been hit by some of the newest rootkits [f-secure.com] that can hide from detection when windows is running. Can't hide from a Linux boot disk and with complete write support, now these can be cleaned and studied more effectively.

Performance

[Reality Master 101]

Unless I missed it, I notice the performance numbers are only single process. I'm suspicious of this because user-mode filesystems (as under microkernel operation systems) typically crash and burn performance-wise under simultaneous load, not under single-user use.

I know that user-mode is easier to debug, but they really should turn this into a kernel module.

Re:Great news.

[maxume]

Are the root kits so insidious that it is dangerous to put them in a throw away windows box as a non boot disk? That would seem like a fairly effective way to study them. I don't think I would want to go through that as a repair procedure, but for research, a disk swap seems easy enough.

Re:Performance

[Reality Master 101]

Any sources or references for that?

Performance problems are a well-known fundamental problem with microkernel architectures that use user-mode processes. If you're interested in the subject, there are lots (and lots) of discussions about it (hint: your instincts above are wrong). Google is your friend.

Re:Performance

[zataang]

I fail to understand your logic. Why would you have multiple people accessing a Windows partition? Either you have a Windows server, or you have a Linux server. I just don't foresee having an NTFS partition on a Linux server (which serves multiple users) being so actively used as to cause performance problems. Also, being in the user space has its own advantages in terms of robustness. Given that ntfs is not documented (well?), I would be much more assured if the module stays out of the kernel. Why do I need to suffer random crashes?

EVEN BETTER NEWS

[dsginter]

Think of the implications:

A given distro can now come with a handy Windows InstallShield Wizard and INSTALL UNDER WINDOWS and BOOT/SHARE the same partition.

This is huge. Who wants to be the first to make a Linux ActiveX malware distro?

Re:Performance

[ArbitraryConstant]

While an NTFS kernel module with awesome performance would be nice, we haven't even had reliable writing before this (not without the Windows NTFS driver). It wasn't just a lack of performance, it was so bad that you'd need to do stuff like keep an FAT32 partition for transferring files. You won't run your high-traffic website on this, but it is still extremely helpful.

For the purposes of making a dual-boot system less painful, it's great. Now all we need is a Windows driver for Reiser...

Re:Cue drumroll for New! Improved! MS file system

[schmiddy]

This is the cue for Microsoft to roll out a new! improved! disk directory format.

I think it's great that Linux has gotten this far with NTFS reading/writing reverse engineering. I've used the shaky NTFS support for quite a while. One key use is when you forget the Admin password on Windows, and you're either locked out of the system or have only user-level privileges, you can use a Linux bootdisk the open the (otherwise hidden) password file and blank it out. The NTFS drivers have always had dire warnings about writing to NTFS possibly resulting in corruption, so this step forward is great.

However, I think you're exactly right – one way or another, MS will find a way to tweak the filesystem, probably in Vista, maybe even as an auto-updated "security fix" to XP, that will make compatibility even harder. Look at how long it's taken us to get this far. Linux has been working on NTFS support since.. at least Win2k, which is a looong time. And now we have to worry about MS putting us back to square one.

I think it's a little sad that Linux has to waste so much time being compatible with MS software (.doc support, NTFS support, FAT support, samba). I'm not in any way saying we shouldn't be doing this, since everyone wants compatibility, but it's a real PITA we have to waste so much time playing catch-up with MS. Imagine where OpenOffice et al. would be today if they didn't have to worry about reverse engineering the miserable .doc format. We could wish that MS would play nice and ues open standards, but there's a snowball's chance in hell of that happening. They know exactly what they're doing, and how much it sets us back. Perhaps someday MS will be forced to play catch-up with us for a change. I guess you could say the IE team is doing that right now, but it's not like we're implementing non-standard features in our browser.

Re:Great news.

[Schraegstrichpunkt]

You also can't hide from a different installation of Windows that has the infected disk mounted.

In theory (assuming a sufficiently naive theory) that is true. In practice, all it takes is Explorer and something like a few WMF files. Heck, Explorer renders HTML for its thumbnail view, so it probably wouldn't be too hard for an attacker to find an exploitable bug somewhere in that code path.

Re:No 64-bit

[david.given]

Let's try reading a 32-bit value from the disk surface and masking out the top 3 bits (random file-systemesque example chosen off the top of my head).

uint32_t read32bit(off_t offset)
{
uint8_t buffer[4];
fread(buffer, 4, 1, disk);
return buffer[0] | (buffer[1]<<8) | (buffer[2]<<16) | buffer[3]<<24);
}

{
uint32_t value = read32bit(offset) & 0x1FFFFFFF;
}

Totally portable, totally trivial, and efficient --- any reasonable compiler will optimise out the 'return' line if it can.. (I'm assuming little-endian. May contain typos.)

This is basic programming skills. There is nothing the least bit hard here, and these days, people should be doing this kind of thing as a matter of course.

Re:Great news.

[Thing 1]

You also can't hide from a different installation of Windows that has the infected disk mounted. Rootkits hide themselves by hooking into the running kernel/fs drivers - inspect the disk with a clean install and they can't hide then either.

Interesting approach: install VMware Server (free), install Windows into a VM (free if you have 2003--IIRC*, Microsoft allows 4 instances, 1 host and 3 virtual), then connect the physical drive to the VM. Not sure whether VMware will bypass the drivers and allow you complete physical access as I haven't tried it but that's one of the options when creating a new virtual hard drive.

You probably don't want to run the VM from the same drive that you attach to it, though... I haven't tested this, but it might be a nice option for investigating without taking down any services that may happen to be running on the potentially-infected PC.

* -- is this the sound made by a crashing car?

FUSE is too slow

[caseih]

The latest knoppix CD uses an older version of this NTFS driver (read-only if I'm not mistaken) via FUSE and it is *slow*. Rsyncing an entire disk for backup purposes can take days (yes days). Disabling the fuse-ntfs system in knoppix and mounting using the read-only NTFS kernel-level driver is several orders of magnitude faster. So I think this driver is good for sharing data and doing emergency stuff, but it is no where near fast enough to think about using it as a root file system or anything. Knowing this latest driver is faster than Paragon's driver is good news; paragon's driver must have been even slower.

When the ntfs driver is stable, I hope it will be put in the kernel (at least as a native file system). Then we can consider adding a unix layer on it and install linux to the same drive as Windows, for those that want to dual-boot.

Re:That's a pity

[bfree]

You could use this to boot from an ntfs partition if you wanted, just use a kernel based read-only driver to get to an initial ramdisk which includes enough support to remount the partition with this. Even without this you should be able to use a loopback filesystem on ntfs even with the older ntfs drivers as their write support is at least meant to be solid if you are not creating or changing the size of files.

Re:Still using FAT32

[Anonymous Cowled]

DOS natively supports fat32, but does not natively support NTFS - tha's what NTFS4DOS is for (google it). Incidentally, for Windows, NTFS is a far superior fs than FAT...

Re:Performance

[Reality Master 101]

Look at the keyword "fundamental" here, THAT's the myth and the fact that several people, AST being one of them, have proven that there is no such "fundamental" difference is the "fact" here.

AST himself said at the site the poster above linked to, "In this paper we argue that for most computer users, reliability is more important than performance and discuss four current research projects striving to improve operating system reliability."

If performance is exactly the same or better than monolithic kernels, as you claim, why would AST make an issue that reliability is more important than it, unless performance WAS an issue? Why wouldn't he write a paper titled, "Having your cake and eating it too... better performance AND better reliability. Why microkernels have won the war."

The answer is because they AREN'T and NEVER will be for general purposes. Sure, you can find isolated tests or isolated projects where they might do better (and the cost of doing better is generally insane complexity), but it's just foolish to argue that they're anywhere close in performance in the general case.

Look, extraordinary claims require extraordinary evidence. Show me the big database servers running on microkernels. Show me the big web servers. Show me big mail servers. And show me how the performance compares to the monolithic kernel operating system on the same hardware.

Sure, microkernels "work", but who cares? I can get DOS to "work". Show me something that works *better*. Or to put it another way, when microkernels are truly better, you won't need to sell everyone, they'll sell themselves. So far, they haven't for general purpose operating systems that care about performance.

Re:This is great news

[Anonymous Coward]

A filesystem really only has to work with a specific operating system and patch level.

What about external USB/Firewire hard drives? If NTFS file systems are no longer compatible between WinXP / WinVista and Win2000, there's going to be some gnashing of teeth by end-users who use those devices. Especially users who use those drives to move large amounts of data between multiple systems. (Segmenting a 300GB drive into 32GB partitions so they can be formatted by Windows with FAT32 isn't vary useful. Unless you get a 3rd party program to let you format larger partitions.)

Microsoft doesn't have a "portable" large drive filesystem other then NTFS that can easily be used on external drives larger then 32GB.

Making things harder for end-users is not in Microsoft's best interest, which is why there's unlikely to be any major changes to NTFS. (Not saying that MS won't shoot its customers in the foot, but it's improbable.) Doing so would drive (more) customers into the arms of Apple, Linux or Solaris.

Re:It is good news ... But ...

[puffing_billy69]

The entire reason Microsoft is the most successful business in the world is that they stay compatible with more stuff for longer than anyone else.

Nonsense: NTFS is a Windows Filesystem. It's never had to be compatible with anything else, and they've clearly made no effort in letting it be. They have made changes in the past that have had consquences. (I'm not saying they made those changes to be evil, the point is, XP broke Nortons).

And who said they'd need to break XP NTFS to break Linux NTFS? I have no doubt at all the current version of NTFS is already prepared for whatever other layers of obfuscation they might add. Mark my words, NTFS was designed only to allow Windows to store and retrieve files, not you. Windows will be your vehicle for storing and retrieveing those files, they've worked damned hard at making it that way, and they'll continue to work damned hard.

Re:Yay

[julesh]

Random aside:

NTFS was actually launched in 1993, 13 years ago, when Windows NT 3.1 (really 1.0, but the version was matched to the MS-DOS-based Windows 3.1) was released.

It's interesting to note that this means XP (which identifies itself internally as NT 5.1) is actually NT release 3.1.

3.1 is typically the best version of any microsoft product (except DOS; 3.3 was generally regarded as better). Version 4 (e.g. Win95, DOS 4.00, ...) is often a complete flop, frequently requiring a quick followup release (W95OSR2, DOS 4.01) to rectify serious problems with it. At this point consumers start to lose cofidence and MS look for a new direction in order to convince people that their software isn't all that bad.

So, when Vista flops, what are MS going to replace it with?

Re:Great news.

[Jeremy Allison - Sam]

No, you can execute directly out of the alternate data stream so long as you end the stream name in .exe....

See here:

http://www.infosecwriters.com/texts.php?op=display &id=53 [infosecwriters.com]

for details. It shows a process listing with myfile.txt listed as a running process.

Scary stuff :-).

Jeremy.