GridFTP Data Transfer¶
GridFTP is a command line service for parallel movement of data. For most transfers, we recommend Globus, which uses the same underlying gridFTP but adds reliability, performance, and ease of use. However, users occasionally find the flexibility of GridFTP useful.
Availability¶
GridFTP provides a high performance transfer mechanism to move data in and out of NERSC. GridFTP is available on the following systems:
| Hostname | Description | Recommended Use |
|---|---|---|
dtn0[1-4].nersc.gov | High performance data transfer nodes with access to all NERSC Global File systems (NGF) | Almost all data transfers needs into & out of NERSC |
garchive.nersc.gov | Single node system connected directly to the NERSC HPSS tape archive | Remote transfers into & out of HPSS |
Transferring Data with GridFTP¶
Use globus-url-copy to move data with GridFTP:
Syntax: globus-url-copy [-help | -usage] [-version[s]] [-vb]
[-dbg] [-b | -a] [-q] [-r] [-rst] [-f <filename>] [-s <subject>] [-ds
<subject>] [-ss <subject>] [-tcp-bs <size>] [-bs <size>] [-p
<parallelism>] [-notpt] [-nodcau] [-dcsafe | -dcpriv] <sourceURL>
<destURL>
Initialize your proxy certificate:
myproxy-logon -s nerscca.nersc.gov
Copy a file from your workstation to dtn01:
globus-url-copy file:///path/to/file \
gsiftp://dtn01.nersc.gov//path/file
Copy a file from your workstation to HPSS:
globus-url-copy gsiftp://garchive.nersc.gov/path/file \
file:///path/to/file
Copy a file from dtn01 to HPSS ("third party copy" without directly logging in to either system)
globus-url-copy gsiftp://dtn01.nersc.gov/path/to/file \
gsiftp://garchive.nersc.gov/path/to/file
Performance Optimization¶
For optimal data transfer performance, you may need to tune certain parameters for your network. In the example below we are using 4 parallel streams with a TCP block size of 4MB:
globus-url-copy -p 4 -tcp-bs 4MB file:///path/to/file \
gsiftp://dtn01.nersc.gov//path/file
Firewalls¶
If you have problems using GridFTP across a firewall (eg. your transfer hangs without moving any data), you may need to ask your network administrator to open a range of ports in your firewall. Once this is done, you will need to set this range in your environment so that GridFTP clients are aware of this.
For example, to use the port range 60000 to 60064 set the following environment variable, before starting your client:
export GLOBUS_TCP_PORT_RANGE=60000,60064