|
North American Network Operators Group Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical Re: Software router state of the art
> Date: Wed, 23 Jul 2008 16:51:50 -0400 > From: "William Herrin" <[email protected]> > Sender: [email protected] > > On Wed, Jul 23, 2008 at 3:59 PM, Kevin Oberman <[email protected]> wrote: > >> The first bottleneck is the interrupts from the NIC. With a generic > >> Intel NIC under Linux, you start to lose a non-trivial number of > >> packets around 700mbps of "normal" traffic because it can't service > >> the interrupts quickly enough. > > > > Most modern high performance network cards support MSI (Message Signaled > > Interrupts) which generate real interrupts only in an intelligent > > basis. and only at a controlled rate. Windows, Solaris and FreeBSD have > > support for MSI and I think Linux does, too. It requires both hardware > > and software support. > > "ethtool -c". Thanks Sargun for putting me on to "I/O Coalescing." > > But cards like the Intel Pro/1000 have 64k of memory for buffering > packets, both in and out. Few have very much more than 64k. 64k means > 32k to tx and 32k to rx. Means you darn well better generate an > interrupt when you get near 16k so that you don't fill the buffer > before the 16k you generated the interrupt for has been cleared. Means > you're generating an interrupt at least for every 10 or so 1500 byte > packets. You have just hit on a huge problems with most (all?) 1G and 10G hardware. The buffers are way too small for optimal performance in any case where the RTT is anything more that half a millisecond, you exhaust the window and stall the stream. I need port move multi-gigabit streams across the country and between the US and Europe. Those are a bit too far apart for those tiny buffers to be of any use at all. This would require 3 GB of buffers. This same problem also make TCP off-load of no use at all. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: [email protected] Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751 Attachment:
pgp00016.pgp
|