University of California

Data Transfer

There are several ways to transfer files to ShaRCS system at both SDSC and LBNL. From UNIX systems, secure copy (scp) is recommended for small files. The following is an example of an scp transfer from a local machine to SDSC Cluster

% scp <source_file> <username>@<path>:/to_dir/<target_file>

To use secure copy from Windows platforms, download a copy of WinSCP (freeware). The scp command is appropriate for transfers of either small files or a small number of larger files.

Transferring large files with bbFTP (currently only available on South Cluster)

For transferring large data files, typically those bigger than 2GB in size, we recommend using the installed bbFTP software. This uses multiple transfer streams and compression in order to speed up the transfer process, making transfers up to 10 times faster than can be achieved using regular scp.

Installing

Linux/UNIX

To use bbFTP from a within a Linux/UNIX environment, install the latest version of the bbftp-client. Source code and installable packages are available from the bbFTP downloads site. Installation instructions can be found here. (You may need the assistance of your local system administrator to complete this step.)

Windows

To use bbFTP from within a windows environment, a client compiled with Cygwin is available (the file bbftp-client-cygwin-3.2.0.zip). To use it, you must have a working version of cygwin with ssh installed on your windows machine. (You may need the assistance of your local system administrator at this stage.)

Usage

bbFTP provides an FTP-like interface for transferring files. For example, to download the file output.dat in the directory /username/data/ from ShaRCS and put it in /home/username/output.dat using 10 data streams, execute:

bbftp -s -e 'get /username/data/output.dat /home/username/output.dat' -u username -p 10 -V <node path>

To upload the file input.dat from your local machine to your home directory on ShaRCS, execute:

bbftp -s -e 'put input.dat/users/username/input.dat' -u username -p 10 -V <node path>

More documentation can be found at the bbFTP website, or by typing 'man bbftp' on the ShaRCS machines.