Nevyn's Journal: "Scalable" network IO

So, as some of you may know, I'm pretty interested in scalable network IO. So I was pretty excited about the recent /. article. Alas, it was written by Felix "I write broken code" von Leitner of dietlibc and libowfat "fame".

Apart from the fact it mainly talked about dealing with network IO events, it completely screwed up the definition of the poll() API ... to the point where all the poll() related data is worthless (see socket_poll, which does what he says you can't with poll ... and faster). This includes the point about fork() being faster then poll() on Linux (which I'd put money on not being true). I was also sad to see alarm() even as a consideration for a timeout API, IMO do not use alarm() ... ever.

It only touched on iovec's and scatter gather IO from the application POV, which is probably due to zero support for that from the DJB designed buffer_* string IO APIs. Also the connect() calls were all blocking, and I think the data traveled using httpbench never used more than a single socket at once (Ie. there weren't ever multiple sockets seeing data at once). Sigh, I guess I'll have to finish my text for all that stuff ... and here I was hoping someone would do it all for me :).

On the other hand, there were a couple of interesting points I didn't know...

• epoll now does Level triggered events by default (stupid, stupid, stupid...). People should just learn to deal with edge triggering properly (of course I'm slightly biased as Vstr helps out a lot here).
• DJB has an IO API
• DJB wants to do non-blocking IO, I'd assumed DJB was firmly in the fork() and blocking IO model after qmail.
• Felix has written a read only LDAP server ... it's very possible that even being Felix this will be better to salvange than anything from OpenLDAP anytime soon.
• The description of the O(1) schedular was pretty good.
• The description of level triggering events and edge trigger events, was also pretty good.
• It's probably always good to have advice out there telling people not to use threads, even if I'm not sure the numbers are right.
• It is nice to have a simple reference for all the IO event mechanisms, this would be better somewhere on c10k really though, IMO. esp. as it's more likely to be accurate.
• I didn't realise you could make BSD TCP_NOPUSH actually do anything useful, although TCP_CORK is vastly nicer it appears you might be able to get generic code to use TCP_NOPUSH on *BSD with most of the gain.