North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Re: GigE performance on BSD

  • From: Richard A. Steenbergen
  • Date: Sun Jun 18 07:59:16 2000

On Sat, 17 Jun 2000, RJ Atkinson wrote:

> > > >I've been able to get DAMN NEAR line rate with sendfile(2) zero-copy
> > > >transfer, and some optimizations for the entire packet output path, using
> > > >FreeBSD and some AMD K7 800s, without using jumbo frames. 
> > > 
> > > Curious.  
> > > sendfile(2) is using TCP ?  Any TCP extensions present ?
> > > This was off-the-shelf FreeBSD ?  Which version ?
> > > Which GigE card (NetGear GA-620 ?) ?
> >
> >Based off 4.0-STABLE, using some work I'm doing on the FreeBSD TCP/IP
> >stack (mainly cleaner code and a few trivial optimizations at this point,
> >nothing earth shattering), some additional optimizations and shortcuts
> >through the stack based on IP Flow which I'm writing, using back to back
> >NetGear GA620s, 512k send/recvspace buffers and a 1MB socket buffer, and a
> >really quick in-kernel ack & discard in the receiving end.
> Ah, so this was neither an off-the-shelf software configuration
> nor a normal user application.  I see.
> >The last time I tried it with standard userland transfers was on back to
> >back p3 500s which pulled about 450Mbps between a GA620 and an Intel GE.
> This is consistent with what I would have expected from a normal
> user application on a modern UNIX over BSD Socket API for that speed
> of Intel CPU.  Folks using CPUs with correct byte-ordering (e.g.
> non-DEC MIPS, SPARC) get about 1 Mbps/MHz under UNIX. Folks seem
> to get about 0.8 Mbps/MHz under NT/Intel (assuming a ~1518 byte MTU).

sendfile(2) is a current off-the-shelf method to improve things
dramatically. The other question is what you do with the data once you
receive it (in the example of an ftp, there is no equivilent recvfile(2)
to do zero copy transfers). And of course, mbufs and checksums are the
rest of the problem (for BSD derivatives, linux has its own very special
brand of problems :P).

Richard A Steenbergen <[email protected]>
PGP Key ID: 0x138EA177  (67 29 D7 BC E8 18 3E DA  B2 46 B3 D8 14 36 FE B6)