High-Performance Computing at the NIH
GitHub YouTube @nih_hpc RSS Feed
Gromacs on Biowulf

GROMACS (www.gromacs.org) is a versatile package to perform molecular dynamics, i.e. simulate the Newtonian equations of motion for systems with hundreds to millions of particles. It is primarily designed for biochemical molecules like proteins and lipids that have a lot of complicated bonded interactions, but since GROMACS is extremely fast at calculating the nonbonded interactions (that usually dominate simulations) many groups are also using it for research on non-biological systems, e.g. polymers.

Important: Please read the webpage Making efficient use of Biowulf's Multinode Partition before running large parallel jobs.

On Helix

Gromacs is a compute-intensive parallel program, and is not suitable for Helix.

Batch job on Biowulf

Gromacs can multi-thread as well as use MPI. For small jobs, e.g. 8 cpus on a single node, multi-threading works almost as well as MPI. For larger jobs, it is best to use MPI. (see the Benchmarks page) for details.

Specifying a homogenous set of nodes

The 'multinode' partition, to which all jobs that require more than a single node must be submitted, is heterogenous. For efficient parallel jobs, you need to ensure that you request nodes of a single CPU type. For example, at the time of writing this webpage, the 'freen' command displays:

biowulf% freen
...
multinode   65/466       3640/26096        28    56    248g   400g   cpu56,core28,g256,ssd400,x2695,ibfdr
multinode   4/190        128/6080          16    32     60g   800g   cpu32,core16,g64,ssd800,x2650,ibfdr
multinode   312/539      17646/30184       28    56    250g   800g   cpu56,core28,g256,ssd800,x2680,ibfdr
...
These lines indicate that there are 3 kinds of nodes in the multinode partition. You should submit your job exclusively to one kind of node by specifying --constraint=x2695 or --constraint=x2650, as in the examples below.

Sample MPI batch script:

#!/bin/bash

module load gromacs/5.0.4

echo "Running Gromacs 5.0.4 with $SLURM_NTASKS MPI tasks"
echo "Nodelist: $SLURM_NODELIST"

mpirun -np $SLURM_NTASKS `which mdrun_mpi` -ntomp 1 -s ion_channel.tpr -maxh 0.50 -resethway -noconfout -nsteps 1000
Submit this job with:
sbatch --partition multinode --constraint=x2695 --job-name=gmx  --ntasks=# --ntasks-per-core=1  --time=168:00:00 --exclusive jobscript
where
--partition multinode Submit to the multinode partition where all nodes are Infiniband-connected
--constraint=x2695 All nodes should be x2695's.
--ntasks # the number of MPI processes you wish to run.
--ntasks-per-core=1 ensures that Gromacs will only run 1 MPI process per physical core, and not use hyperthreaded cores. This is recommended for parallel jobs.
-ntomp1 uses only one OMP thread per MPI thread. This means that Gromacs will run using only MPI, which provides the best performance.
--time=168:00:00 max walltime=168 hrs (1 week). See the section on chaining jobs below.
--exclusive Allocate the nodes exclusively to this job (recommended for parallel jobs)

Replica Exchange

Sample batch script for Gromacs 4.6.5 (thanks to Mingzhen Zhang)

#!/bin/bash

module load gromacs/4.6.5

cd $SLURM_SUBMIT_DIR

echo "Running Gromacs 5.0.4 with $SLURM_NTASKS MPI tasks"
echo "Nodelist: $SLURM_NODELIST"

mpirun -np $SLURM_NTASKS `which mdrun_mpi` -ntomp 1 -s cmd_.tpr -maxh 0.50 -resethway -noconfout -cpi state.cpt -noappend -multi 48 -replex 1000

Submit with:

sbatch --partition multinode --constraint=x2695 --job-name=MyJob --ntasks=64 --ntasks-per-core=1 --exclusive   myjobscript

GPU support is built in to Gromacs 5.*. Sample batch script with the adh_cubic job from the Gromacs GPU page. The files for this job are available in /usr/local/apps/gromacs/ADH_bench_systems.tar.gz (untar the file and look in the adh_cubic directory).

Sample batch script:

#!/bin/bash

module load gromacs/5.0.4

echo "Running Gromacs 5.0.4 with $SLURM_NTASKS MPI tasks"
echo "Nodelist: $SLURM_NODELIST"

grompp -f pme_verlet.mdp
mpirun -np $SLURM_NTASKS --mca btl_openib_if_exclude "mlx4_0:1" `which mdrun_mpi` -ntomp 1 -s topol.tpr

Submit this job with, for example:

sbatch --partition=gpu --gres=gpu:k20x:2 --ntasks=2  jobscript

The above command will submit the job to 2 GPUs.

-ntomp 1 sets 1 CPU thread per MPI (GPU) process. In our tests a 1:1 CPU:GPU ratio gave the best performance (see benchmarks).
--mca btl_openib_if_exclude "mlx4_0:1"prevents a warning about OpenFabrics from appearing in your output. You can also leave it out and live with the warning :-).
--partition=gpu submit to the GPU partition
--gres=gpu:k20x:2 Resource=gpu, Resource type=k20x, Count=2 (2 GPUs). Note that 'count' is required even if you are submitting to a single GPU.
--ntasks=2Number of MPI tasks to spawn.

Chaining jobs

The max walltime on the multinode partition is 10 days. (type 'batchlim' to see the CPU and walltime limits on all partitions). Thus, jobs should be designed to run for a week or so, save a checkpoint file, and submit a new job starting from that checkpoint.

A reasonable strategy would be to set up a job to run for a week or less by setting the number of steps appropriately, and then, at the end of the job, have it resubmit itself to continue the simulation. Below is a sample batch script:

#!/bin/bash
# this script is called Run.ib

module load gromacs/5.0.4

cd /path/to/my/dir

mpirun -np $SLURM_NTASKS `which mdrun_mpi` -ntomp 1 -s ion_channel.tpr -maxh 0.50 -resethway -noconfout -nsteps 1000

# use tpbconv to create a new topol.tpr file with an increased number of steps
tpbconv -s topol.tpr -extend 500 -o topol2.tpr

#move the newly created topol.tpr into place
mv topol.tpr topol.tpr.prev; mv topol2.tpr topol.tpr

#resubmit this script
sbatch --partition multinode --constraint=x2650 --job-name=gmx  --ntasks=# --ntasks-per-core=1  --time=168:00:00 --exclusive  Run.ib
More information at Extending Simulations on the Gromacs site.

If a Gromacs job is terminated unexpectedly (for example, the walltime limit was hit before the mdrun completed), it is simple to restart. The state.cpt file contains all the information necessary to continue the simulation. Use the '-cpi' and '-append' options to mdrun, which will append to existing energy, trajectory and log files. For example:

mpirun -n $np `which mdrun_mpi` -s topol.tpr -cpi state.cpt -append

More information at Doing Restarts on the Gromacs website.

Benchmarks

see the benchmarks page

Documentation

Gromacs website
mdrun documentation for v5.0.4