"iSCSI killer" Native in Linux

jar writes "First came Fibre Channel, then iSCSI. Now, for the increasingly popular idea of using a network to connect storage to servers, there's a third option called ATA over Ethernet (AoE). Upstart Linux developer and kernel contributor Coraid could use AoE shake up networked storage with a significantly less expensive way to do storage -- under $1 per Gigabyte. Linux Journal also has a full description of how AoE works." Note that the LJ article is from last year; the news story is more recent.

Will it catch on?

[andrewman327]

From TFA:
Some significant caveats mean that not everyone is so keen on the technology. For a start, it's a specification from Coraid, not an industry standard. Its networking abilities are limited. And its detractors include storage heavyweights such as Hewlett-Packard and Network Appliance.

So will this ever develop into a real standard or will it remain the sole domain of one company? I do not know if I want to invest time and money into it if the latter is true. From a comp sci point of view this is a great approach to networked storage. It uses what people already have to make storage reletively cheap. I am going to wait to see where this technology goes. Maybe it will blossom and become a serious contender.

Re:Reliability

[dfghjk]

how is that relevant to the discussion of protocols?

reliability of SCSI versus ATA is largely imagined and the rest is intentional. drive manufacturers want you to believe their enterprise drives are more reliable and right now those drives are largely SCSI.

iSCSI can talk to ATA drives

[Anonymous Coward]

iSCSI is a protocol. ATA disks are a physical medium. They work together, and you get the benefits of SCSI commands with the price of ATA disks. Just because iSCSI is the protocol does NOT mean that you need to use SCSI disks. You might even be talking to a RAID of ATA disks and not know it.

So, why would you need AoE? It's already cheap, and been for sale for some time.

Not an iSCSI killer, here are the reasons why not

[cblack]

1) Complexity for RAID and volume management is not centralized and is pushed to individual hosts. One of the main benefits of SAN technology is that you can just carve out storage from a single interface and assign it to a server and the server simply sees it as a block device. With AoE each drive is addressed separately by the server, which means it is up to the server (and server admin) to figure out how to handle distributing over multiple drives, handle drive failures, and expanding volumes. This is huge.
2) It is not a standard and is only really supported by one vendor. This may change in the future but it is significant right now. It is registered with the IEEE but that hardly makes it a peer-reviewed standard with input/improvements from many experts.
3) No boot from SAN. Until someone makes some sort of mini bootstrap system on a CD or a hardware card implementation of AoE that can be addressed as a block device admins will be unable to host the root filesystem and/or C: drive on an AoE SAN
4) No multipath (that I can see). Perhaps I misunderstand this, but it seems like there is no way to do multipath IO with this system. That is, all the drives are single-connected to a network. If that network switch goes down, all drives on that network are inaccessible.
So AoE looks like a neat technology for pushing drives out of the box and potentially sharing them among hosts, but there is no intelligence there. It is just dumb block addressable storage with no added availability or management, and therefore is far from being an iSCSI or FC killer.

Re:Another "Killer"

[slimjim8094]

So was MP3 (at least implementations) and it was around longer and more widely supported by programs/devices.

Not so much cheaper

[err666]

Ok, so the Coraid people are selling their ATA over Ethernet 15 slot version for $3,995.00. That's apparently around EUR 3133. I can get something proven iSCSI based from Promise here in Germany for 4.499,- (a Promise M500i). Ok, that is almost 50 percent more expensive, but the iSCSI solution is supposed to work under all operating systems (Linux, *BSD, Windows, etc.) more or less out of the box, while for AoE you will have to buy drivers for Windows, and has generally worse support for other operating systems.

Now, suppose you will really use this baby and you want to have *lots* of storage.

So you buy 15 SATA drives, like say Seagate ST3750640NS for EUR 444 each. Now the difference between AoE and iSCSI becomes less:

AoE solution: EUR 9793
iSCSI solution: EUR 11159

Now the iSCSI solution is only 14 % more expensive.

Now it would be clear for me to go for the "safe" path of something proven and widely supported like iSCSI instead of AoE. The infrastructe you need will be the same anyway (Gigabit Ethernet, Gigabit ethernet switch).

Re:Bootable?

[wild_berry]

My inexpert guess would involve getting a Tyan Thunder/Tiger motherboard with LinuxBIOS and compiling and configuring your own ATAoE support. Windows would need to think it's a local disk; LinuxBIOS could pretend that it was.

put it back in the oven

[jhackworth]

perhaps an interesting idea, but just because I can build a computer out of old, recycled clock parts doesn't mean it is going to become my server. Also, iSCSI adoption has increased something like 40% this year. Windows support for iSCSI will improve dramatically with the next revision, and iSCSI costs are only going to decrease.

Also, consider management of one of these AoE boxes. What sort of tools are out there to simplify provisioning, deployment, snapshots and backup, etc. In order for this to go anyplace but the basement of 'the IT guy at work' a whole lot more stuff will be required. Oh yeah, and that probably isn't going to happen with 1 vendor controlling the market.

AoE is not fully baked yet. Put it back in the oven and let me know when the timer goes off.

Re:Another "Killer"

[Anonymous Coward]

"Yeah, just like OGG took off because it was open source. Hah."

It won't take off anytime soon because Microsoft and Apple's native players do not support it (why not?). If they did, it would help. Imagine not needing anybody's special plugin. MS/Apple just want you to use their media files and don't care what a hassles it creates for the user in the mixed environment of the internet.

Re:Cheaper?

[Zephiris]

The really silly thing about this is that they claim it's "lower overhead" than TCP/IP because people are having to buy "expensive TCP offloading engines" for iSCSI, when a few seconds of research provided, namely on Wikipedia (http://en.wikipedia.org/wiki/ISCSI), that plain NICs can outperform the offloading ones, and sure, it's obviously going to be lighter than TCP/IP, however, ATA over Ethernet only has basic authentication (MAC addresses, which can be forged cheerily), can't be routed, and isn't very available. It's -only- usable for Storage Area Network, not really for general remote drive (or part of a drive even) access. At currently, only Linux support is available. iSCSI is supported by Windows, Linux, Solaris, among others. Even FreeBSD is working on a native implementation. Windows Vista will even include a fully built-in/native support for iSCSI. I can't imagine why they complain that iSCSI is 'more expensive' to implement, when their primary product for ATA over Ethernet is a 'special drive enclosure' (according to their documentation, you can't even use AoE with standard networking hardware, interfaces, routers, etc) with special networking hardware which can house up to 15 ATA drives. The enclosure itself (with nothing else) costs about $4000. You could build ten high-end machines dedicated to serving iSCSI requests to multiple drives each for that (five if they use actual SCSI), and still use standard networking hardware, and still have it accessable from a network across the world, with things like actual user authentication.

The whole ATA over Ethernet thing seems like trying to blow smoke up the arses of some very rich and silly people. At the same time, the technologies are rather different, too. If you just want to build a SAN? Sure, go for HyperSCSI or AoE, maybe, but if you actually want remote drive access? Why would you want any of this? They shouldn't be trying to utterly replace iSCSI. It's absurd. As far as I see it, iSCSI is more of a general and free/open replacement for things such the old 'X drive' remote service, and network filesharing like SMB/NFS. Websites can (and are starting to) offer iSCSI targets to offer remote drives for backup. It can also be used for cheap SAN, or more-or-less replacing SMB/NFS over a network. It does all of this rather well.

It seems to me that the company behind ATA over Ethernet is becoming rather desperate to resort to such claims.