High-Performance Computing at the NIH
GitHub YouTube @nih_hpc RSS Feed
Storage on Biowulf & Helix

There are several options for disk storage on the NIH HPC; please review this section carefully to decide where to place your data. Contact the NIH HPC systems staff if you have any questions.

Except where noted, there are no quotas, time limits or other restrictions placed on the use of space on the NIH HPC, but please use the space responsibly; even hundreds of terabytes won't last forever if files are never deleted. Disk space on the NIH HPC should never be used as archival storage.

Users who require more than the default disk storage quota should fill out the online storage request form.

NOTE: Historical traces of disk usage and file counts are available through the User Dashboard.

Summary of file storage options
  Location Creation Backups Space Available from
/home network (NFS) with Helix account yes 8 GB default quota B,C,H
/lscratch (nodes) local created by user job no ~850 GB shared C
/scratch network (NFS) created by user no 75 TB shared B,H,C
/data network (GPFS/NFS) with Biowulf account no 100 GB default quota B,C,H
H = helix, B = biowulf login node, C = biowulf compute nodes

/home
Each user has a home directory called /home/username which is accessible from every HPC system. The /home area has a quota of 8 GB which cannot be increased.
/lscratch (nodes)

Each Biowulf node has a directly attached disk containing a /lscratch filesystem. Note that this space is not backed up, and thus, users should use it only as temporary space while running a job. Once the job exits, you will no longer have access to /lscratch on the node.

To use /lscratch, see Using Local Disk in the Biowulf User Guide.

Please use /lscratch or /scratch instead of /tmp for storage of temporary files.

/scratch
There is a shared /scratch area that is accessible from Helix, the Biowulf login node, and the Biowulf computational nodes. Files on this filesystem which have not been accessed for 14 days are automatically deleted by the system.

/data

These are RAID-6 filesystems mounted over NFS or GPFS from one of the following, all of which are configured for high availability: eight NetApp FAS8040 controllers, a DataDirect Networks SFA10K storage system with eight fileservers and two DataDirect Networks SFA12K storage systems with eight fileservers each. This storage offers high performance NFS/GPFS access, and is exported to Biowulf over a dedicated high-speed network. /data is accessible from all computational nodes as well as Biowulf and Helix, and will be the filesystem of choice for most users to store their large datasets. Biowulf users are assigned an initial quota of 100 GB on /data; please contact the Biowulf staff if you need to increase your quota.

Note: your /data directory is actually physically located on filesystems named /spin1, /gs2, /gs3, /gs4 or /gs5. The /data directory consists of links to one of those filesystems. ALWAYS refer to your data directory through the /data links as opposed to the physical location because the physical location is subject to change based on administrator needs. In other words, use /data/yourname rather than (for example) /gs4/users/yourname in your scripts.

Snapshots are available that allow users to recover files that have been inadvertently deleted. For more information on backups and snapshots, please refer to the File Backups webpage.

If a user's data directory is located on the /spin1 storage area, it is possible that the size of snapshots in the user's data directory may cause a user to have less available storage space than he believes he should.

Each data directory on /spin1 has its own snapshot space that is separate from a user's regular file or data space. However, if a user deletes too much data within a short period of time (a week or less), it is possible that the snapshot space for that data directory can be exceeded. When this occurs, a user's regular file space (allocated quota space) will be used as additional snapshot space. Unfortunately there is no way for a user to either determine whether he has encountered this situation or to actually delete his snapshots. However, if a user does not see an expected increase in available space in the output of the 'checkquota' command following the cleanup of large files, the possibility exists that he has encountered this situation and should contact the Helix/Biowulf staff for assistance.

Sharing data

Information on methods for sharing data with collaborators both inside and outside NIH can be found on our sharing data webpage,

Checking your disk storage usage

Use the checkquota command to determine how much disk space you are using:

$ checkquota

Mount                   Used      Quota  Percent    Files    Limit 
/data:               70.4 GB   100.0 GB   70.41%   307411  6225917
/data(SharedDir):    11.4 GB    20.0 GB   56.95%     6273  1245180
/home:                2.0 GB     8.0 GB   25.19%    11125      n/a
mailbox:             74.7 MB     1.0 GB    7.29%         
Best practices
BAD GOOD
Submitting a swarm without knowing how much data it will generate Run a single job, sum up the output and tmp files, and figure out if you have enough space before submitting the swarm
Directory with 1 million files Directories with < 5,000 files
100 jobs all reading the same 50 GB file over and over from /data/$USER/ Use /lscratch instead, copy the file there, and have each job access the file on local disk
100 jobs all writing and deleting large numbers of small temporary files Use /lscratch instead, have all tmp files written to local disk
Each collaborator having a copy of data on Biowulf Ask for a shared area and keep shared files there to minimize duplication
Use Biowulf storage for archiving Move unused or old data back to you local system