Better Networking with SCTP

5-0 writes to tell us that IBM DeveloperWorks has an interesting look at the key features of SCTP in the Linux 2.6 kernel and the ability to deliver multi-streaming. "SCTP is a reliable, general-purpose transport layer protocol for use on IP networks. While the protocol was originally designed for telephony signaling, SCTP provided an added bonus -- it solved some of the limitations of TCP while borrowing beneficial features of UDP. SCTP provides features for high availability, increased reliability, and improved security for socket initiation."

Re:multihoming?

[isj]

I don't know if you missed something - I didn't RTFA.

Heartbeats are optional. Some real-time applications probably want to use heartbeats every 10 seconds, while other can disable them completely.

The multihoming has nothing to do with routing table size. The multihoming feature is used for providing better connectivity.
Imagine your laptop with WiFi. If the application (say, FTP download) used SCTP instead of TCP then the download would not break when your laptop moves from one access point to another and switches ip-address. SCTP survives that.

Re:How long...

[sjames]

Easy that: as long as it took IPv6 to be adopted into the mainstream.

Probably not that long. The problem with IPv6 is that too many entities are involved in a successful v6 deployment and too many changes have to happen at different levels.

OTOH, SCTP requires only a client and a server that want to use it.

Re:INIT floods

[Jonner]

Read about SYN Cookies [cr.yp.to]. This is a method of avoiding SYN DOS attacks that has already been implemented in Linux (and probably elsewhere) for a while. I think SCTP just integrates the same concept into the official protocol specification. Once the SCTP server sends the INIT-ACK, it doesn't have to keep track of that association until the client sends a COOKIE-ECHO.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Re:multihoming?

[romiz]

Did I miss something?

This is an transport layer, not a network layer. It is only necessary in endpoints, such as clients and servers, and it might be a good thing if firewalls understood it. But the routers don't interpret it, so there won't be any change on backbones, except a slight increase in traffic with a few more keep-alive packets.

Re:What's not said

[jhermans]

That's total bullshit. MS has nothing to do with this. In fact, no-support of SCTP in MS operating systems will be thee biggets hurdle for the introduction. It's *Linux* that is driving ther adoption !

Disclaimer: I'm using SCTP in my job for several years now.

Re:Credit where due

[tpgp]

While the protocol was originally designed for telephony signaling...

In other words, it started out in the hands of AT&T, Bell Labs, Northern Telecom, Alcatel, et. al.

Utterly incorrect.

If you had only taken ten seconds to check wikipedia's sctp page [wikipedia.org] you would have found it was developed by the Internet Engineering Taskforce's SIGNTRAN [wikipedia.org] working group.

The IETF is an open, all-volunteer standards organization and couldn't be further in spirit from the monopolies you mention.

Give credit where it is due indeed.

(oh and this protocol was defined in 2000 - far later then the telephone signalling I suspect you're thinking of)

SCTP vs TCP benchmarks

[Anonymous Coward]

FYI,

http://www.altoriopreto.com.br/mestrado/index_en.p hp [altoriopreto.com.br]

Re:How long...

[zm]

> Sctp is new

Nope. I worked on SCTP implementation in year 2000.... Nortel had it in 1999.

zm

FreeBSD, Darwin

[Midnight Thunder]

Seems like there is an implementation [www.sctp.de] for FreeBSD and Darwin (underlying OS used by MacOS X) too, according to this page.

Other possible future TCPs

[rev_karol]

There's also Scalable-TCP, High-Speed TCP, FAST-TCP, BIC-TCP, H-TCP. Each with their own advantages. Check out the site. These guys are doing interesting evaluations. H-TCP is specifically what they work on:
TCP Evaluation Discussion [hamilton.ie]
Interesting plots too [hamilton.ie]
The end result is that TCP is not particularly suited to high-speed networks.

Re:INIT floods

[lagfest]

Who says you have to track the cookies? Just make a hash of the client's ip address, port, and a key that changes every 20 seconds. Now you only have to save a history of the three latest keys.

In fact, that's pretty close to how it's done according to SCTP for beginners [uni-essen.de]

The server receives an association setup request (an INIT chunk) usually in the CLOSED state, and analyzes the data contained in that chunk. From that it generates all the values needed at its side to enter an established association, and generates a secure hash of these values and a secret key (e.g. with the MD5 or SHA-1 algorithms). The values are then put into the so-called COOKIE, along with the derived message authentication code (MAC). This COOKIE is returned to the sender of the INIT chunk in an INIT-ACK chunk. The server remains in the CLOSED state, and forgets all about the received INIT chunk.

Re:Linux to Linux

[lasindi]

I may run Linux myself, but in almost everything I do on my desktop (that isn't itself Linux-related) I am interacting with non-Linux machines. I'm forever "losing out" because I can't receive MSN special features. Sure I could do webcam with what was gnomemeeting (it looks awesome) but does anyone run it? Thankfully now I have friends riding Firefox and one using Jabber (googletalk).

But yes all my friends use windows!

So will such features help Desktop Linux?

Short answer: It might "help Desktop Linux" in general, but it will fix zero interoperability issues and it will do nothing to the problems you listed.

Long answer: You need to learn a few things about network protocols, my friend. Even if SCTP, TCP or UDP had anything to do with your problems, SCTP is not implemented on Windows. Most if not all of the programs you're using use TCP or UDP, and the issues of compatibility you're experiencing have nothing to do with these protocols. The programs you mention have their own protocols that run over TCP and UDP. Seriously, go and learn how to program BSD sockets [softlab.ntua.gr] and you'll understand where TCP and UDP are in the network protocol heirarchy. Once you've done that, maybe you could help out projects like Kopete and Gaim to fix your problems.

Re:Goodbye TCP?

[KDR_11k]

The SYN flood happens because the server has to keep track of the connection until it times out and there's a limit to the number of connections the server can keep track of, with the cookie method the server gets an INIT, sends out the cookie and forgets about the connection. Only when that is returned the server opens a connection. Sure, you could go through the proper protocol for starting the connection but that forces you to tell the server the real IP of your DoSing client instead of putting a number of fake IPs in there. The server could just start ignoring your client for a while and your DoS has failed.

Re:How this compares WRT DCCP?

[ChristopherX]

DCCP and SCTP are not very related. DCCP is an improvement on UDP - DCCP is an unreliable protocol that improves upon UDP by being aware of, and throttling back upon, network congestion.

Does this matter with TCP/IP offload and iWarp?

[soldack]

For all its problems TCP/IP is everywhere. This fact has made it the networking technology to use even when it doesn't make technical sense. For example, folks use it in high performance computing and in storage (iSCSI) where there are much better methods available technically. Its commonality (along with ethernet's popularity) often make TCP/IP over ethernet the cheapest solution to many problems (while not the best).

I used to work on InfiniBand where the reliability/congestion detection protocol (Reliable Connected and Link Level flow control in IB terms) are in hardware. This scales to 20 Gbit connections between hosts quite well. Other examples of hardware protocols include myrinet (invented by myricom) and qsnet (from quadrics) and scalable coherent interface current pushed by Dolphin Interconnect. All of these folks struggle to compete with good old TCP/IP over ethernet. Except for the parts of the HPC world, TCP/IP over ethernet wins. In the storage landscape, Fibre Channel, SAS, and SATA seem to be holding out but iSCSI sure is trying.

The performance issue is real though and very few systems can saturate a 10 Gbit TCP/IP etnernet link without massive host CPU overhead. One solution floating around is that instead of trying to make new protocols to replace TCP, we should imitate the competition and put hard work in hardware. TCP/IP offload NICs (TOE) are becoming increasingly more popular. With RDMA technology layered on top of it you get iWARP. For storage you get iSER (ironically from an IB company!). This technology is being adopted by both the MS and Linux camps so it seems to have a good shot. In fact, many of the interfaces used by IB work about as well over iWARP cards. Things like Message Passing Interface, Direct Access Provider Library, Sockets Direct Protocol (SDP), and iSER do not know the difference between iWARP and IB or anyone else.

Software can just post a full size message and it gets sent out the wire without copying, segmentation, timers, resends, or other CPU hogs. This kind of stuff really helps with large messages. With SDP, apps can be made to take advantage of it without changes to the application. MS is also providing a standard way for just TOE NICs without RDMA abilities to work with the OS. Linux doesn't seem to have a standardized way for TCP/IP to be offloaded entirely but is supporting RDMA and SDP.

The things SCTP seems to offer is more explicit understanding of the difference between failure and congestion and multi-home support. This could make load balancing over multiple paths between hosts pretty interesting. The problem I see is that is that it is competing with the established TCP that now has many of its warts fixed with hardware offload. SCTP will still have the issue of a CPU handling segmentation/reassembly, massive amounts of interrupts, timer/retry overhead, etc. It also seems to have a higher overhead for connection establishment (although that is mitigated by being able to send data during the end phases). Is this a solution looking for a real problem? Pehaps not. Does this really have a chance of being taken up? I am not too confident.
-Ack

Interesting. Not a bad idea

[Animats]

It's always been a bit strange that TCP, which is a streaming protocol and ignores message boundaries, is the standard for request/response message type traffic. You have to add a framing protocol on top of TCP to do messaging, which is what everybody does.

The first attempt in the IP world to add a protocol of this type was Reliable Data Protocol, in 1984. (See RFC 908 [faqs.org]). But that never went anywhere. Since then, nobody has really addressed this. There was ISO TP4, but that didn't go anywhere either, althoug it was fully supported in Windows NT.

SCTP has reasonable congestion behavior, like TCP, so it's an improvement over UDP-based protocols in that regard. Moving some UDP-based protocols to SCTP could be a step up. That's where it could be most useful. It might make sense to put remote procedure call type protocols that now use UDP onto SCTP. If a protocol has to do retransmission, it's better to do it at the transport layer than at the application layer.

The "multihoming" thing seems badly placed, because that's not properly a transport layer function. But I haven't really looked at that.

John Nagle

Kernel space name resolution not required

[butlerm]

SCTP does have an option for using name resolution to do multihoming, however for practical reasons it is almost universally unimplemented. SCTP multihoming works just fine without it. IP address lists for multihoming are exchanged during the standard connection (association) establishment process.

State cookies are not stored on the server at all, but rather are echoed from the client back to the server as a effective means of SYN flood style DoS attack prevention.

SCTP (properly implemented) is radically superior to TCP for a large class of applications, basically anything that needs low latency reliable message exchange. The lack of message boundary information in TCP causes considerable pain for implementers of upper layer protocols - notably RDMA/RDDP and iSCSI. The running solution for efficient hardware implementation of RDMA and iSCSI over TCP involves *inserting* markers every 512 bytes or so in the middle of a data stream so that the receiver can re-synchronize it efficiently.

The primary SCTP RFC is RFC 2960 for those who are wondering.

Re:What's not said

[Anonymous Coward]

You, Sir, are a blatant troll. I happen to know at least one of the developers of SCTP (was one of my professor at university), and Microsoft has nothing to do with this. Stop spreading bullshit.

Sounds similar to TIPC

[AaronW]

This sounds somewhat similar to TIPC [sourceforge.net] which we're using in some projects where I work. Like UDP it is message based, but it provides a reliable message transport. It also runs in the kernel as a protocol stack. It does have some differences, though. It is not based on a source and destination, but rather a publish/subscribe mechanism which sounds similar to the SCTP multi-homing support. With the publish/subscribe, one or more clients can indicate that they're interested in a certain service. When that service becomes available or disappears on the node, cluster, or network (depending on the scope of configuration) the client stack will automatically notify it.
It also has the concept of priority in it, so that messages may be prioritized.
Unlike SCTP, however, it does not run on top of IP but is its own protocol that runs directly over the wire, which means that it cannot be routed across an IP network. It is great as an internal embedded messaging protocol, but not as useful when a network is involved.
TIPC is also not connection oriented. There is no connection setup required to send messages much like UDP.

-Aaron

Low message delivery latency

[butlerm]

SCTP excels at low latency delivery of small messages. TCP's head of line blocking is a serious problem in many applications. SSH tunneling is a good example.

The main advantage of using SCTP over multiple TCP connections is connection establishment time as well as server overhead. You can create an association with hundreds of streams in the roughly the same time as a single TCP connection, with little or no overhead for unused streams. Then when you want to initiate a new non-blocking transaction you can send a message on a new stream without the three-way handshake of an extra TCP connection.

In addition, a single SCTP socket can handle reliably delivered messages on thousands of streams from hundreds of associations. No need to use select()/poll() on a long list of file descriptors.

Especially on the WAN, yes

[butlerm]

SCTP and Infiniband focus on different areas. IB is largely a high performance HPC / cluster network architecture for LAN applications, where SCTP is a transport protocol designed to operate efficiently under WAN conditions (significant packet loss, high RTTs).

SCTP is a more efficient RDMA/iWARP transport than TCP, but the differential advantage of SCTP over TCP is much lower in a LAN environment due to the low RTTs, so RDDP/TCP dominates so far despite the bizarre marker insertion scheme (MPA). Same goes for iSCSI.

The interrupt issue has largely been solved - on Linux NAPI dynamically switches between interrupt and polled mode to reduce this overhead to negligible levels. Message signalled interrupts also help considerably.

What would be much more helpful (and economical) for iSCSI, SCTP, and RDDP is NIC CRC32C checksum generation. CRC generation is quite expensive in software but trivial in hardware.

SCTP wasn't originally designed for load balancing a single association via simultaneous multi-path transfer (SMT). It can be done, but it requires some loss detection algorithm changes. Someone still needs to develop a option to coordinate this at association establishment time.

One advantage of SCTP over TCP is that on a per stream basis, SCTP connection establishment overhead is much lower than TCP - basically O(1) instead of O(N) in the number of streams.

SCTP uses AIMD congestion control

[butlerm]

Congestion control is an area where SCTP is much like TCP. SCTP uses AIMD on a per destination address basis. However any of the the alternative congestion algorithms for TCP would behave similarly with SCTP.

Of course given the additional message boundary information available in SCTP, further improvements could be made.

Protocols that can benefit from SCTP

[Skapare]

Some of the protocols that could benefit from SCTP include:

• IRC ... put each channel in its own stream to minimize lost packet retry bottlenecks. This is especially valuable in server to server trunk links.
• HTTP ... multiple page requests, each in a separate stream, avoids the flood of multiple TCP connections that can use many processes on the server, and avoids the wait of sequential chunks in persistent connections.
• SMTP ... get your Nigerian business deals, body part enlargement products, replacement ink cartridges, notifications of winning in lotteries you never played, stock investment advise, and those all important sexual drive enhancement drugs, all at the same time.

Linux support for TOE

[butlerm]

Actually, most iWARP/RDMA stuff doesn't have a software interface to TCP at all - the hardware handles not only TCP, but three or more layers on top of it (at least MPA, DDP, and RDMA, plus iSCSI in some cases). That type of interface is not a problem. What is controversial is using TOE for conventional TCP applications using kernel space dispatch.

This is a bit of an end run around the Linux kernel bridging, routing, and filtering layers, which is the primary reason why support for it won't get merged in the kernel socket layer until RNICs can at a minimum do IPtables like IP address filtering and proper dispatch so that some packets can be routed through the kernel layers on an Ethertype and IP protocol / address / port specific basis.