BBCP another High Bandwith File Transfer Utility

BBCP is a file transfer utility currently in alpha used mainly for transferring files (Huge Files) through high Bandwidth Links.

Usage

To transfer the local file /local/path/largefile.tar to the remote system remotesystem as /remote/path/largefile.tar:

bbcp -P 2 -V -w 8m -s 16 /local/path/largefile.tar remotesystem:/remote/path/largefile.tar

Where:
“-P 2” , produces progress messages every 2 seconds.
“-V” , produces verbose output, including detailed transfer speed statistics.
“-w 8m” , sets the size of the disk I/O buffers.
“-s 16” , sets the number of parallel network streams to 16.

bbcp assumes the remote system’s non-interactive environment contains the path to the bbcp utility. This can be determined by with the following command:

ssh remotesystem which bbcp

If this is not the case the “-T” bbcp option can be used to specify how to start bbcp on the remote system. For example:

bbcp -P 2 -V -w 8m -s 16 -T 'ssh -x -a -oFallBackToRsh=no %I -l %U %H /remote/path/to/bbcp' /local/path/largefile.tar  remotesystem:/remote/path/largefile.tar

Often during large transfers the connection between the transfering systems is lost. The “-a” options gives bbcp the ability to pick up where it left off. For example:

bbcp -k -a /remotesystem/homedir/.bbcp/ -P 2 -V -w 8m -s 16 /local/path/largefile.tar remotesystem:/remote/path/largefile.tar

To transfer an entire directory tree,

bbcp -r -P 2 -V -w 8m -s 16 /local/path/* remotesystem:/remote/path

When transferring files to the Cray XT3 (jaguar) at NCCS, it is necessary to specify a particular jaguar node as the destination host because the hostname jaguar.ccs.ornl.gov actually points to a server load balancing device which returns node addresses in a round robin fashion. For example:

bbcp -r -P 2 -V -w 8m -s 16 /local/path/* jaguar3.ccs.ornl.gov:/remote/path

Documentation

More information on bbcp can be found by typing “bbcp -h”