North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Re: from the academic side of the house

  • From: JP Velders
  • Date: Sun Apr 29 08:30:22 2007

On Thu, 26 Apr 2007, Simon Leinen wrote:

> Date: Thu, 26 Apr 2007 19:11:37 +0200
> From: Simon Leinen <[email protected]>
> Subject: Re: from the academic side of the house

> [ ... ]
> Another host issue would be interrupts and CPU load for checksum, but
> most modern 10GE (and also GigE!) adapters offload checksum
> segmentation and reassembly, as well as checksum computation and
> validation to the adapter if the OS/driver supports it.

Correct, though sometimes the performance can vary wildly depending on 
driver and firmware. I had to turn off all the checksumming with 
certain driver versions for the Myrinet 10GE cards, later drivers 
performed OK.

Also, having multi-cpu and multi-core machines helps enormously, 
binding a NIC/driver to a certain core, and having the receiving 
process on the neighbouring core can dramatically increase throughput.

But per adapter/driver differences in which offloading actually 
increases performance sometimes still is a black art, necessitating 
painstaking testing...

> The adapters used in this record (Chelsio S310E) contain a full TOE
> (TCP Offload Engine) that can run the entire TCP state machine on the
> adapter, although I'm more sure whether they made use of that.

I believe that they did not use it. In the past we also ran into 
problems and performance dips when using the accellerated Chelsio 
cards. Without TOE accelleration things worked much smoother.

Kei's team did modify iperf such that they used mmap's to gain more of 
a zero-copy behaviour, depending on drivers this can also give much 
better performance, akin to say Myricom's claims for Myrinet...

Kind regards,
JP Velders