High-Performance Computing at the NIH
GitHub YouTube @nih_hpc RSS Feed
Acemd/HTMD on Biowulf & Helix

ACEMD is a high performance molecular dynamics code for biomolecular systems designed specifically for NVIDIA GPUs. Simple and fast, ACEMD uses very similar commands and input files of NAMD and output files as NAMD or Gromacs.

Acemd v321u1 is installed on the system. Use 'module load acemd' to load this version. This version will only run on the Biowulf K20x nodes.

The newer HTMD will run on both the K20x and K80 GPUs. Use 'module load htmd' to load this version.

ACEMD User Guide

HTMD User Guide

8 Acemd licenses are available on Biowulf.

On Helix

Acemd can only be run on the Biowulf GPU nodes.

Batch job on Biowulf

HTMD v1.7.35: This version will run on either the K80 or K20x GPUs.

#!/bin/bash

module load htmd/1.7.35
cd /data/$USER/mydir

acemd input > log.txt

With this batch script, ACEMD will run on one GPU on a single node. All the GPU nodes on Biowulf have 2 GPUs each. You should not use the '--device' flag in Acemd in this script, as you do not know in advance which GPU will be allocated to your job. Acemd will automatically run on whichever GPU is allocated.

Submit this job with:

sbatch  --partition gpu --gres=gpu:k80:1  --license=acemd  jobscript

where

--partition gpu job submitted to GPU partition
--gres=gpu:k80:1 requests 1 K80 GPU
--license=acemd requests an Acemd license. The job will only run when a license is available.

A sample job is available in /usr/local/apps/acemd/apoa1-test

Older Acemd 321u1: Note that this version will only run on the K20x GPUs. Set up a batch script like this:

#!/bin/bash

module laod acemd
cd /data/$USER/mydir

acemd input > log.txt

Submit with:

sbatch --partition=gpu --gres=gpu:k20x:1 --license=acemd jobscript
where
--partition gpu job submitted to GPU partition
--gres=gpu:k20x:1 requests 1 K20x GPU
--license=acemd requests an Acemd license. The job will only run when a license is available.
See the Acemd docs for more info.

Parallel Acemd job

If 2 GPUs are specified as a comma-separated list with the --device flag, Acemd will attempt to run a single simulation in parallel across them. For example:

#!/bin/bash

module load acemd
cd /data/$USER/mydir

acemd --device 0,1  input > log.txt
Submit with:
sbatch  -p gpu --gres=gpu:k20x:2 --nodes=1 --constraint=acemd  jobscript

where

-p gpu job submitted to GPU partition
--gres=gpu:k20x:2 requests 2 GPUs
--nodes=1 requests 1 node. This ensures that both allocated GPUs would be on a single node.
--constraint=acemd requests a node with an Acemd license

Note that requesting more GPUs does not necessarily double the performance. For example, with the test Apoa1 job (92K atoms):

1 GPU 54.473 ns/day
2 GPUs 69.794 ns/day

In this case, doubling the number of GPUs gave a 30% increase in performance. This is insufficient improvement to justify allocating 2 GPUs. The more resources that are requested, the longer it may take for the batch system to schedule the job. Other molecular systems may scale differently from Apoa1, so it is definitely worth running benchmarks of your own molecular system to determine the best number of GPUs to use.

See the Acemd documentation for more details.

Checkpoints

It is highly recommended that you checkpoint your Acemd jobs, since there is always a chance of hardware or software failure with longrunning jobs, and you don't want to waste the CPU you've already used. The following acemd flags are used for checkpointing:

restart on | off Enable the saving of checkpoint files, for restarting simulations. Default: off
restartfreq <+ve integer> Frequency with which ACEMD saves checkpoint files. Default: 0
restartname Prefix for checkpoint file filenames. Default: restart
restartsave on | off Enable the creation of checkpoint file backups. By default old checkpoints will be over-written. Default: off
Running a Replica Exchange job

ACEMD Pro supports ensemble simulations for replica exchange molecular dynamics. Ensemble mode is automatically enabled if ACEMD is run via MPI. For example, to run a 4 replica ensemble, set up a batch script as follows:

#!/bin/bash

module load acemd

cd /data/$USER/mydir

mpirun -np $SLURM_NTASKS  acemd input 

Note that you should not specify a GPU device number. ACEMD will run one replica per GPU.

Submit this job with:

sbatch -p gpu --gres=gpu:k20x:2  --ntasks=4  --constraint=acemd jobscript

Documentation

HTMD User Guide

ACEMD documentation

Acemd homepage at acellera.com