NIH HPC Core Facility Expansion

Aug 2016: Batch System Changes

A number of changes will be made to the Slurm batch system on the NIH Biowulf cluster in Aug 2016. These changes are being made as part of the expansion of the cluster and will allow the expanded system to more efficiently schedule jobs. They are based on our experience to date with SLURM as well as users' input.

Partition Current Starting late Aug 2016 Notes
norm (default) Max CPUs per user: 1024
Default walltime: 4 hrs
Max walltime: 10 days Max CPUs per user: 4096
Default walltime: 2 hrs
Max walltime: 10 days Only single-node jobs can be submitted to this partition.
multinode does not currently exist Max CPUS per user: 3072
Default walltime: 8 hrs
Max Walltime: 10 days Intended for parallel jobs that require 2 or more nodes. Single-node jobs will not be allowed to run on this partition. All nodes connected to FDR Infiniband network.
multinode + 'turbo' QoS does not currently exist Max CPUS per user: 8192
Default walltime: 8 hrs
Max Walltime: 8 hrs Add --qos=turbo to your sbatch command to use this QoS.
b1 Max CPUS per user: 4096
Default walltime: 6 hrs
Max Walltime: 10 days eliminated The b1 nodes will be merged into the quick partition.
ibfdr Max CPUS per user: 1280
Default walltime: 10 days
Max Walltime: 10 days eliminated The ibfdr nodes will be merged into the multinode partition.
ibqdr Max CPUS per user: 1280
Default walltime: 10 days
Max Walltime: 10 days eliminated The ibqdr nodes will be merged into the quick partition.
quick Max CPUS per user: 2048
Default walltime: 2 hrs
Max Walltime: 2 hrs Max CPUs per user: 6144
Default walltime: 1 hrs
Max walltime: 2 hrs
interactive Unchanged
Max CPUs per user: 64
Default walltime: 8 hrs
Max walltime: 36 hrs

largemem Unchanged
Max CPUs per user: 128
Default walltime: 4 hrs
Max walltime: 10 days

unlimited Unchanged
Max CPUs per user: 128
Default walltime: Unlimited
Max walltime: Unlimited

gpu Unchanged
Max CPUs per user: 128 (8 GPUs)
Default walltime: 10 days
Max walltime: 10 days

Buyin partitions
ccr, nimh, niddk Unchanged

The 'batchlim' command can be used at any time to see the current Max CPUs and walltime limits.

Partition	Current	Starting late Aug 2016	Notes
norm (default)	Max CPUs per user: 1024 Default walltime: 4 hrs Max walltime: 10 days	Max CPUs per user: 4096 Default walltime: 2 hrs Max walltime: 10 days	Only single-node jobs can be submitted to this partition.
multinode	does not currently exist	Max CPUS per user: 3072 Default walltime: 8 hrs Max Walltime: 10 days	Intended for parallel jobs that require 2 or more nodes. Single-node jobs will not be allowed to run on this partition. All nodes connected to FDR Infiniband network.
multinode + 'turbo' QoS	does not currently exist	Max CPUS per user: 8192 Default walltime: 8 hrs Max Walltime: 8 hrs	Add `--qos=turbo` to your sbatch command to use this QoS.
b1	Max CPUS per user: 4096 Default walltime: 6 hrs Max Walltime: 10 days	eliminated	The b1 nodes will be merged into the quick partition.
ibfdr	Max CPUS per user: 1280 Default walltime: 10 days Max Walltime: 10 days	eliminated	The ibfdr nodes will be merged into the multinode partition.
ibqdr	Max CPUS per user: 1280 Default walltime: 10 days Max Walltime: 10 days	eliminated	The ibqdr nodes will be merged into the quick partition.
quick	Max CPUS per user: 2048 Default walltime: 2 hrs Max Walltime: 2 hrs	Max CPUs per user: 6144 Default walltime: 1 hrs Max walltime: 2 hrs
interactive	Unchanged Max CPUs per user: 64 Default walltime: 8 hrs Max walltime: 36 hrs
largemem	Unchanged Max CPUs per user: 128 Default walltime: 4 hrs Max walltime: 10 days
unlimited	Unchanged Max CPUs per user: 128 Default walltime: Unlimited Max walltime: Unlimited
gpu	Unchanged Max CPUs per user: 128 (8 GPUs) Default walltime: 10 days Max walltime: 10 days
Buyin partitions ccr, nimh, niddk	Unchanged