Globus on NIH HPC (Biowulf)
Quick Links

NIH HPC Globus Endpoints

Globus is a service that makes it easy to move, sync, and share large amounts of data. Globus will manage file transfers, monitor performance, retry failures, recover from faults automatically when possible, and report the status of your data transfer. Globus uses GridFTP for more reliable and high-performance file transfer, and will queue file transfers to be performed asynchronously in the background.

Globus was developed and is maintained at the University of Chicago and is used extensively at supercomputer centers and major research facilities. [Globus website]

No matter how you transfer data in and out of our systems, be aware that PII and PHI data cannot be stored or transferred into the NIH HPC systems.

See the links in the Quick Links menu at left for details.

Transferring between Biowulf and the NCI DME

NCI's HPC DME has a frontend that allows Globus transfers to and from the NIH HPC Data Transfer (Biowulf) endpoint. The actual Globus transfers are performed by the NCI DME platform, so each Biowulf user needs to provide access to their Biowulf space to the NCI DME Globus group. The Biowulf user needs to set up a shared endpoint, share it with the NCI DME group, and then use the NCI DME software to start the transfer.

Below are the steps to set up a shared endpoint on Biowulf and share it with the NCI DME Globus group.

  1. Point your web browser to the Globus File Manager window. For the Collection, enter 'NIH HPC Data Transfer', and for the Path, enter /data/username or /data/shared_dir_name. If you are transferring data into Biowulf, it is best to set up a new folder for this data.

    Select the directory to be used for the transfer, and click 'Share'.

  2. Enter a meaningful name for the shared endpoint and click 'Create Share'.
  3. Select 'Add Permissions', and then click 'Group'. You will be sharing this endpoint with the NCI DME group,
  4. Enter 'HPCDME' in the Search box, and select one of the NCI endpoints. The Biowulf staff do not administer these endpoints, so you may need to check with the NCI DME group which endpoint to select here.
  5. If you are transferring data from NCI DME to Biowulf, give both read and write permissions to the NCI DME group.
  6. In the Overview tab, you will see the UUID of this new shared endpoint. Copy it.
  7. Now go to the NCI DME interface, select a Globus transfer, and enter this UUID. You should now be able to transfer data between the NCI DME and the NIH HPC (Biowulf) endpoint.