North American Network Operators Group
Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Re: Best utilizing fat long pipes and large file transfer

  • From: Sean Knox
  • Date: Fri Jun 13 13:43:53 2008
Many thanks for great replies on and off-list.

The suggestions basically ranged from these options:

1. tune TCP on all hosts you wish to transfer between
2. create tuned TCP proxies and transfer through those hosts
3. setup a socat (netcat++) proxy and send through this host
4. use an alternative to plain netcat/scp for large file transfers

My needs are pretty simple: occasionally I need to push large database files (300Gb+) around linux hosts. #4 seems like the best option for me.

People suggested a slew of methods to do this: RBUDP, gridftp, bbcp, and many others, with programs either sending with reliable UDP or breaking large transfers into multiple streams. Because it was easy to use right away, I tried RBUDP and was able to copy a tarball at about 700Mb/s over a 20ms delay link, and when factoring in destination disk write speed, isn't too bad a starting point. GridFTP and bbcp look very useful too; I'll be exploring them as well. The presentation links Kevin O. sent were very interesting.

I've looked at HPN-SSH before but haven't played with it much. I'll definitely try it out based on the feedback from this thread.

Thanks again.

sk


Sean Knox wrote:
Hi,

I'm looking for input on the best practices for sending large files over a long fat pipe between facilities (gigabit private circuit, ~20ms RTT).
I'd like to avoid modifying TCP windows and options on end hosts where possible (I have a lot of them). I've seen products that work as "transfer stations" using "reliable UDP" to get around the windowing problem.

I'm thinking of setting up servers with optimized TCP settings to push big files around data centers but I'm curious to know how others deal with LFN+large transfers.

thanks,
Sean