File Backups and Snapshots on the HPC Systems

Storage Area Backup policy Snapshots
Home directories
(/home/username) 		Weekly backups, with daily incremental backups. These tapes are kept onsite for four weeks. Every four weeks a special full backup of home directories is done and sent offsite; these tapes are kept for eight weeks 6 hourly, 6 daily and 8 weekly
Data directories and shared data directories
(/data/username or /data/SharedDir) 		NOT BACKED UP. Users should back up any irreplaceable data to their own local systems. 2 nightly and 1 weekly. Very large /data directories may contain only one nightly snapshot.
Buy-in storage As agreed in the storage buy-in agreement. As agreed in the storage buy-in agreement.

Snapshots

A 'snapshot' is a read-only copy of the data at a particular point in time.

If you inadvertently deleted a file, you can try to restore it yourself from an existing snapshot, using the instructions below. If the file had been present in your directory when a snapshot was made, you will be able to see the file in the .snapshot and copy it back into place.

Note that .snapshot directories are 'hidden'. They won't show up in an 'ls'. .

Recover files in home directory from snapshot

Log into your helix account, go to your home directory:

 
  % cd

Assume that you are going to restore the files in your home directory, type:

 
% cd .snapshot

% ls
Hourly.2016-05-11_1700  Hourly.2016-05-12_1400   Nightly.2016-05-12_0010  Weekly.2016-04-24_0010
Hourly.2016-05-11_2000  Nightly.2016-05-06_0010  Weekly.2016-03-20_0010   Weekly.2016-05-01_0010
Hourly.2016-05-11_2300  Nightly.2016-05-07_0010  Weekly.2016-03-27_0010   Weekly.2016-05-08_0010
Hourly.2016-05-12_0500  Nightly.2016-05-09_0010  Weekly.2016-04-03_0010
Hourly.2016-05-12_0800  Nightly.2016-05-10_0010  Weekly.2016-04-10_0010
Hourly.2016-05-12_1100  Nightly.2016-05-11_0010  Weekly.2016-04-17_0010

Select the time frame when your files still existed. There are 3 time categories: hourly, nightly, and weekly. Each category has several snapshots with timestamps as part of the directory name (e.g. Nightly.2016-05-06_0010 is a snapshot taken on May 6, 2016, at 6 am.)

Go to selected directory:

% cd Nightly.2016-05-06_0010

Then find the file(s) you are interested in and copy them to your current home directory:

% cp lostfile /home/$USER/lostfile

If you have your directory mounted on your desktop Windows machine (see http://helix.nih.gov/Documentation/transfer.html), you can also recover files:

.snapshot directory

Recover files in data directory from snapshot

Snapshots for data directories are done less frequently than for home directories. Typically there are two daily and two weekly snapshot available for all data directories. The commands required for accessing the snapshots vary slightly depending on the storage system on which a data directory is located

To determine the storage system on which your data directory is located, run the following:

% ls -ld /data/$USER

A linked path starting with "/vf/users" indicates that the data directory is on the VAST storage system. A linked path starting with "/gs" indicates that the data directory is on a GPFS storage system and those starting with "/spin1" are on the NetApp storage system.

If the data directory is on a GPFS or NetApp storage system:

The snapshot directory names for GPFS filesystems have a format of daily.‹timestamp› and weekly.‹timestamp› where ‹timestamp› is the date/time when the snapshot was created. Those on the NetApp system have a format of _nightly.0 and _weekly.0

To recover data directory files from snapshots, go to the snapshot area of your data directory (or shared data directory):

% cd /data/$USER/.snapshot

Select the time frame when your files still existed based on the snapshot filename formats descibed above and go to selected directory. For example:

% cd weekly.2021-05-09_0751

Find the file(s) you wish to recover and copy them to your data directory:

% cp lostfile /data/$USER/lostfile

If the data directory is on the VAST storage system:

To recover data directory files on the VAST storage system, go to the global snapshot directory for the VAST system:

% cd /vf/users/.snapshot

The snapshot directory names for the VAST storage system have the following format: daily_‹timestamp› for daily snapshots and weekly_‹timestamp› for weekly snapshots where ‹timestamp› is the date/time when the snapshot was created.

Run a 'ls' to list the available snapshots.

Select the time frame when your files still existed based on the snapshot filename formats described above. For example, go to selected directory:

% cd weekly._2020-06-14T00_00_00.045003UTC

Go to the directory corresponding to your username (or the shared data directory):

% cd $USER

Find the file(s) you wish to recover and copy them to your data directory:

% cp lostfile /data/$USER/lostfile

Note that data directory snapshots are updated every weekday and may be temporarily unavailable while an update is in progress.