"iSCSI killer" Native in Linux

jar writes "First came Fibre Channel, then iSCSI. Now, for the increasingly popular idea of using a network to connect storage to servers, there's a third option called ATA over Ethernet (AoE). Upstart Linux developer and kernel contributor Coraid could use AoE shake up networked storage with a significantly less expensive way to do storage -- under $1 per Gigabyte. Linux Journal also has a full description of how AoE works." Note that the LJ article is from last year; the news story is more recent.

Cheaper?

[DSW-128]

I guess I don't really see how it's cheaper that iSCSI? Sure, there's less overhead from the lack of TCP/IP, so you may not need as massive a network to drive it equally. But I've been under the understanding that iSCSI doesn't require SCSI drives, so you could build an iSCSI target out of the same machine/drives as an AoE host, correct? For some applications, I think the lack of TCP/IP might be a benefit - less opportunity to hack. (Then again, I'd expect anybody deploying something like this or iSCSI would drop the few extra $$$ to build a parallel network that transports just storage.)

Reliability

[Neil Watson]

People often forget there is a considerable difference in the reliability of ATA drives versus SCSI. If you are going to use some sort of ATA based SAN be prepared for disk failures much sooner than if they were SCSI.

Yes!

[mihalis]

I like the look of this technology. The great thing it has going for it is that most of the non-hard-disk infrastructure (switches and cabling) leverages the tremendous investment in ethernet. That is great.

The thing that needs work, in my view, is that the bit that links the disks and the rest isn't cheap enough. In fact what would be awesome here is if, say, Seagate provided disks with native ATAoE connectors built-in. They might have to buy Coraid for that to happen.

In case anyone thinks I'm out of my mind here, don't forget that disks can already be had with ATA interface, SCSI interface, FCAL interface, SATA, SAS - that's five and there are probably more. Yes they might be a bit more expensive, but if they come in under the combined price of "regular ATA disk" + Coraid ATAoE disk adapter then you'd come out ahead. Someone like Seagate would, I think, have the industry-wide clout and respect to succeed in making this an open standard. Something that will be a challenge for Coraid for a long time (I have nothing against them, btw, they are friendly and their mailing list didn't spam me when I signed up).

When I was on the OpenSolaris pilot project I tried to get people interested in using this with Solaris. I think it might be great for ZFS, for example. At that point the real storage wizards were more interested in iSCSI, but I respectfully disagree, OpenSolaris + ZFS + cheap storage = awesome file server. Emphasis on the cheap. As Sun people will admit, their previous attempts at RAID were more like RAVED (Redundant Array of Very Expensive Disk). Coraid does have a Solaris driver, so this is definitely feasible.

iSCSI killer?

[apharov]

In the context of using this in low-cost environments with Linux I can hardly see how this could kill iSCSI. Last week I implemented an iSCSI setup for about 500 euros (target serves out 500GB disk space for non-critical backup) using standard components, FC5, iSCSI Enterprise Target [sourceforge.net] and Microsoft iSCSI Initiator.

Works great and is a lot (>10x) faster than the about similarly priced NAS device that was used for the same task before.

Bootable?

[Anonymous Coward]

Is it possible to boot WindowsXP via AoE or iSCSI? I want a diskless WindowsXP box.

Re:More for business?

[riley]

Storage Area Network solutions are not under the $1/GB. Running a network filesystem (NFS, SMB, Coda, etc) are running a local filesystem over networked storage are two different things, fulfilling two different needs.

iSCSI and AoE don't necessary directly benefit the small/home server market, but for the things that SANs are traditionally used for (data replication across geographically separated sites without any changes to the application software) there could end up being a big win in cost.

Re:Not an iSCSI killer, here are the reasons why n

[Cyberax]

You can use Ethernet-based multipath IO, a lot of switches can be stacked to provide redundancy (and load-ballancing).

AoE is a COOL thing exactly because it's a 'dumb' technology. You can buy a switch, a bunch of disk drives and AoE adapters, a small Linux PC - and your storage system is ready. There is a lot of existing RAID manipulation and monitoring tools for Linux, so RAID configuration is not a problem.

You also can boot from SAN, it's not a problem. Just add required modules and configs to initrd and place it on a USB drive.

I can't tell if this is clever or stupid...

[YesIAmAScript]

A wise man once told me there is a fine line between them.

ATA is a crappy protocol, even when local. It's only good for squeezing that last $0.03 out of the controller cost. Once you are using ethernet cables ($1) and links and PHYs on each end ($4 each), it makes a lot more sense to put some brains back in. Use SCSI. Heck, even ATAPI optical drives (the optical drive in your computer) uses ATAPI, which is SCSI in packetized ATA transfers.

Also, I'm a bit nervous about the packet CRC validation being done in the ethernet controller/layer itself. The problem is that if an ethernet switch between you and the storage device stores packs and forwards them (as all smart switches do), it may also chose to regenerate the CRC on the way. If it corrupts the packet internally and generates a new, valid CRC for the new, corrupt packet, you have undetected corruption. I'd be a bit nervous about that for my hard drive.

I do think using GigE is a smart way to attach hard drives to servers. I look at the back of an Apple XServe and see two GigE ports and a fibre channel card. Why can't one GigE port be used to attach to the network and one to the XServe RAID? Why do I need to get a multi hundred dollar card to attach to the XServe RAID when that GigE port is fast enough? It'd sure save a lot of cost, and hopefully reduce the price ot the end user.

Anyway. I'm pro GigE attachment, not sure I'm for this AoE.

Re:More for business?

[rf0]

iSCSI is slightly differnet as rather than presenting a file system, it presents a hardware device. So you show it a 1TB device over the network (e.g /dev/sdb) then the client machine can partition that disk up as if it was local. Thats the advantage over just a shared network filesystem