That is bullshit. Here is a guy that benchmarked the Intel X520 10G NIC that wrote a small piece titled "Packet I/O Performance on a low-end desktop": http://shader.kaist.edu/packetshader/io_engine/benchmark/i3.html [kaist.edu]
His echo service manages to do between 10 and 18 Gbit/s of traffic even at packet size of 60 bytes. And there is plenty of optimizations he could do to improve on that. The NIC supports CPU core affinity so he could have the load spread on multiple cores. The memory bandwidth issue could have been solved with NUMA. But even without really trying we are hitting the required response time on desktop hardware.
That's without data ever being accessed from userspace, no protocol stack, average packet size being half of the maximum, and there is a good possibility that the measurements are wrong, because then it would be easier to implement the whole switch by just stuffing multiple CPU cores into the device, and the whole problem would not exist.
The simple fact is that after the packet has been transferred over the 10G link it will go through a PCI Express (x8) bus and be processed by the Linux OS - the same OS that you earlier claimed to be running on the control plane of the switches designed by your company. The only difference here is that I would probably get a faster system CPU than would be in your hardware.
Actually in that driver it was not processed by anything -- copying is performed while in kernel. Data is mapped to userspace-accessible memory but userspace does not even touch it, everything happens in kernel, with network stack being completely bypassed. The only thing less relevant (but perfectly suitable for a switch if it was indeed that fast) would be forwarding done entirely in interrupt handlers.
Another important question is, it's still unknown what the latency of that thing is -- with enough buffers it can get very high, and we did not even take into account all the delays in hardware, sitting in the queues in the switches that forward them over the management network.
As to the blocking issue, only packets from the same (new) flow would be queued.
Except there is no new flow because it will be only created after the controller will produce a rule to identify it, so any packet may belong to it. And there is a matter of somehow expiring old rules based on lack of traffic that matches them. This works very well when the whole NAT or load balancing can be reduced to predefined static mapping with a hash. What is done already with managed switches, even cheap ones. There is no benefit in making fancy controllers do that.
Also given that protocols like TCP do not just suddenly burst out 10G of packets
But protocols like UDP (and convoluted monstrosities on top of them with multiple dependent mappings, like SIP), do.
, the next packet following the initial SYN packet is not likely to arrive before the SYN has been processed by both switch and controller and forwarded long ago.
Except there may be a DoS in progress that fakes those packets. On a host, you can have an algorithm that prevents them from eating your memory, so only real mappings (with both sides completing the handshake) end up being persistent. On a switch you have no way to implement such an algorithm because it does not fit into rules you can define by your protocol, what means, either your simple switch has to become a router, or it will not be able to provide such functionality without falling a victim to a simple SYN flood.
And again packets to other destinations will not be blocked while we wait for the controller and somehow I get the impression that you think they would.
There is also a matter of number of rules, size of internal RAM in ASICs that contains them (and it's a different kind of RAM, not cheap and not large, think CPU cache), and time that it takes to update it. All for nothing because CPU in the switch can do the same, better, without any "globally standardized" formats for rule-passing protocol, without data ever showing up on management interfaces, without any of those hare-brained schemes. As I said, there is a place for standardization of switch ASIC control, and it will be a great way to make open source firmware for high-end switches and routers feasible. But then the standard should not be crippled or shoehorned into a management protocol that is supposed to work over the network.