High-Performance Computing at the NIH
GitHub YouTube @nih_hpc RSS Feed
Schrödinger on Biowulf

A limited number of Schrödinger applications are available on the Biowulf cluster through the Molecular Modeling Interest Group. The use of Schrödinger applications is limited to intramural NIH users only.

Schrödinger applications can be run from the command line, through the batch system, and interactively through the Maestro GUI. See the documentation link below for more information, particularly the Job Control Guide.

Schrödinger Versions

There are multiple versions of Schrödinger available. An easy way of selecting the version is to use modules. To see the modules available, type

module avail schrodinger

To select a module, type

module load schrodinger/[ver]

where [ver] is the version of choice.

Please contact the Molecular Modeling Interest Group for current version status and any differences between versions.

Environment Variables

Schrödinger applications make use of two key environment variables. These handle where files are temporarily stored.

SCHRODINGER_TMPDIR is a scratch directory, and is set by default to ~/.schrodinger/tmp. SCHRODINGER_JOBDB2 is a directory containing information for the job database, and is set by default to ~/.schrodinger/.jobdb.

Because your /home directory is restricted to 8GB, it is strongly recommended to set these optional environment variables to your /data directory. Otherwise, you run the risk of running out of space during a run.

For example,

where [user] would be replaced by your username.

Local Scratch Disk Space

Schrödinger applications make extensive use of local scratch disk space. These are defined in two environment variables: SCHRODINGER_TMPDIR and TMPDIR. For the best I/O performance, allocate local scratch space to use, rather than shared space.

For both interactive sessions and batch jobs, including

--gres lscratch:N

will allocate N GB of local scratch space. Loading the schrodinger module will set the environment to use this space. The amount of space necessary depends on the jobs to be run; 20GB is a good minimal value.

NOTE: When Schrödinger jobs are submitted to the batch system, a jserver process is started and run in the background. This jserver process ensures that files created in the scratch directory are copied and incorporated into the job database. If the jserver process dies before the submitted job finishes, files in the scratch directory are not copied back and incorporated into the job database. If the scratch directory is deleted, as will happen when using dynamically allocated scratch space (/lscratch/$SLURM_JOB_ID), THE FILES ARE LOST.

Be aware that if a job is launched from an interactive Biowulf session, the remote jobs must finish within the time allocated to the interactive session, or the results of the jobs launched may be lost.

Schrödinger hosts file:

Job control is handled through the hosts file, schrodinger.hosts. When a Schrödinger application is started, it reads the schrodinger.hosts file to determine various attributes for job control.

Here are the keywords for schrodinger.hosts:

name The name of the entry. For a host this is usually the host name, but any name can be used. This name is displayed in Maestro by job control, and is used for the -HOST option. The name must not contain spaces. The value localhost is a special value that means the host on which the job is launched. host The host name. Required for batch queue entries; otherwise only needed if it is different from name.
nodelist List of entry names, used to define a multiple-host entry. Each name may be followed by a colon and a number of processors. Can be combined with a host setting.
schrodinger The path to the Schrödinger software installation on the host. You can include more than one of these settings. This value is superseded by the $SCHRODINGER environment variable.
user The user name to use on the host. This should never be set in the hosts file in the installation directory. It is required if the user has a different user name on the defined host than on the host on which the job is launched.
processors The number of processors available on the host. If the host is part of a cluster, this number should be the total number of processors available on the cluster. The default is 1.
env Environment variables to be set on the specified host. The syntax for the environment variables is variable=value, regardless of the shell used. List each environment variable on a separate env line. Not used on the submission host.
tmpdir Base directory for temporary or scratch files, also called the scratch directory. The actual directory created for scratch files (the job directory) is tmpdir/username/uniquename, where tmpdir is the directory defined here and username is the user name. Multiple tmpdir settings can be added for a given host and are used by Maestro, but the first setting is used otherwise. This value is superseded by the $SCHRODINGER_TMPDIR environment variable.
queue Queuing system name. PBS is the supported system on Biowulf. Must be set to the subdirectory of $SCHRODINGER/queues that contains the support files for the queuing system.
qargs Arguments to be used when submitting jobs to a batch queue. These arguments should specify any parameters that define the queue.
include The name of an auxiliary hosts file to be included in the current hosts file. The inclusion is done by replacing the include line with the contents of the specified file.
base The name of an entry (the base entry) that is the basis for the current entry. All the keywords from the base entry are inherited by the current entry, and new keywords may be added, in any order. May be used recursively.

Here is a generic schrodinger.hosts file that can be used under almost any circumstance. It can be spontaneous created with this command:

$ make_schrodinger_hostfile.sh
# ==============================================================================
# Run on the localhost
# ==============================================================================
Name: localhost
Host: localhost
Processors: 2

Name: localhost-large
Host: localhost
Processors: 16

# ==============================================================================
# Request a single CPU per subjob
# ==============================================================================

Name: singleproc
Queue: SLURM-biowulf
Qargs: --ntasks=1 --cpus-per-task=1
Host: localhost
Processors: 1024 

Name: singleproc-quick
Queue: SLURM-biowulf
Qargs: --ntasks=1 --cpus-per-task=1 --time=00:30:00 --partition=quick
Host: localhost
Processors: 1024 

Name: singleproc-long
Queue: SLURM-biowulf
Qargs: --ntasks=1 --cpus-per-task=1 --time=4-00:00:00
Host: localhost
Processors: 1024 

# ==============================================================================
# Request %NPROC% processors, one on each node
# ==============================================================================

Name: multiproc
Queue: SLURM-biowulf
Qargs: --nodes=1 --ntasks-per-node=%NPROC%
Host: localhost
Processors: 1024 

Name: multiproc-quick
Queue: SLURM-biowulf
Qargs: --partition=quick --nodes=1 --ntasks-per-node=%NPROC% --time=00:30:00
Host: localhost
Processors: 1024 

Name: multiproc-long
Queue: SLURM-biowulf
Qargs: --nodes=1 --ntasks-per-node=%NPROC% --time=4-00:00:00
Host: localhost
Processors: 1024 

# ==============================================================================
# Request %NPROC% processors, spread across nodes, 8 cpus per node, %NPROC%
# must be a multiple of 8
# ==============================================================================

Name: multiproc-multinode
Queue: SLURM-biowulf
Qargs: --nodes=%NPROC/8% --ntasks-per-node=8
Host: localhost
Processors: 1024

Name: multiproc-multinode-quick
Queue: SLURM-biowulf
Qargs: --nodes=%NPROC/8% --ntasks-per-node=8 --time=00:30:00 --partition=quick
Host: localhost
Processors: 1024

Name: multiproc-multinode-long
Queue: SLURM-biowulf
Qargs: --nodes=%NPROC/8% --ntasks-per-node=8 --time=4-00:00:00
Host: localhost
Processors: 1024

# ==============================================================================
# Parallel Jaguar: OpenMP.
# Request %TPP% threads per node.
# ==============================================================================

Name: multinode
Queue: SLURM-biowulf
Qargs: --nodes=1 --ntasks-per-node=%TPP%
Host: localhost
Processors: 1024

Name: multinode-quick
Queue: SLURM-biowulf
Qargs: --partition=quick --nodes=1 --ntasks-per-node=%TPP% --time=00:30:00
Host: localhost
Processors: 1024

Name: multinode-long
Queue: SLURM-biowulf
Qargs: --nodes=1 --ntasks-per-node=%TPP% --time=4-00:00:00
Host: localhost
Processors: 1024

This file can be copied to ~/.schrodinger/schrodinger.hosts for use under all circumstances.

There are four situations (single processor, multiprocessor, multinode-multiprocessor, and multinode) for three different time requirements (quick, normal, and long). Any modifications to the desired partition or time allocation necessitate changes to the schrodinger.hosts file. Custom versions of schrodinger.hosts must be copied to the working directory prior to running Schrödinger.

When running a Schrödinger application on a single node interactively (e.g., Glide), the localhost or localhost-large can be used. The value of Processors can be changed to match that available.

When running a Schrödinger application that requires multiple nodes (e.g., Desmond on Biowulf), the multinode entries can be used.

For more information about job control and the schrodinger.hosts file, please see the Job Control Guide.

Example for submission to the cluster:

First create a batch submission script:

[biowulf]$ cat bmintest.sh
#!/bin/bash
#SBATCH --job-name bmintest
module load schrodinger
bmin test2 -WAIT

Then submit to the cluster:

[biowulf]$ sbatch bmintest.sh
Using Maestro interactively:

Maestro is a GUI for running Schrödinger jobs interactively. It requires an X11 connection. The quality of graphics forwarding depends strongly on the X11 server and the graphics driver on your desktop machine. NX is highly recommended as the X11 server.

Maestro should be run on an interactive node, not on the login node. Make sure you have an X11 connection to Biowulf. Start an interactive session, then start maestro:

[biowulf]$ sinteractive

See here for more information on running interactive sessions on Biowulf.

Once an interactive session has been allocated, set the proper environment variables, then type

[node]$ maestro &

NOTE: Due to incompatibilities between the OpenGL/GLX commands given by the Maestro GUI and the X11 server running on your desktop client machine, the interface may hang or crash with errors. If this is the case, Maestro can be run with the -SGL option:

[node]$ maestro -SGL &

If running exclusively on an interactive node, always use the localhost submission entry.

Running Glide in parallel:

Glide runs can be submitted in parallel using the Biowlf cluster.

Running Desmond

Multiprocessor Desmond jobs SHOULD NOT BE SUBMTTED FROM MAESTRO, but instead encapsulated in a batch script and then submitted to the batch system.

Desmond jobs will not scale well beyond 8 CPUs, so it not advisable to use CPUs/subjob values larger than 8. Below is a plot of clocktimes versus the number of cores used for a 12348 atom system molecular dynamics run. The simulation time was 1.2 ns, using NPT and at 300 K. As can be seen, the simulation does not scale well beyond 4 cores (efficiency < 50%), and the clocktime essentially plateaus with 8 or more cores.

Desmond efficiency chart

For a more efficient molecular dynamics application, and a better explanation of efficiency, please see NAMD Benchmarks.

You can monitor the progress of the Desmond run from the monitor window:

Monitor Window

Schrödinger documentation and links