Biowulf High Performance Computing at the NIH
ORCA on Biowulf

ORCA is a multi-purpose quantum-chemical software package. It features a wide variety of methods ranging from semi-empirical methods to density functional theory to correlated single- and multi-reference wave-function based methods. Environmental as well as relativistic effects can be taken into account.

References:

Documentation

NOTE: ORCA is licensed software that requires registration.
It is free for academic use only. If you want to use ORCA on the NIH HPC systems you must send email to orca.license@kofo.mpg.de and accept the End User License Agreement (EULA). You will receive a confirmation email that you must forward to staff@hpc.nih.gov. An NIH HPC staff member will then add you to the appropriate group to use ORCA.

Important Notes

Interactive job
Interactive jobs should be used for debugging, graphics, or applications that cannot be run as batch jobs.

Allocate an interactive session and run the program.
Sample session (user input in bold):

[user@biowulf ~]$ sinteractive -c4 --mem=8g --gres=lscratch:10
salloc.exe: Pending job allocation 48899347
salloc.exe: job 48899347 queued and waiting for resources
salloc.exe: job 48899347 has been allocated resources
salloc.exe: Granted job allocation 48899347
srun: error: x11: no local DISPLAY defined, skipping

[user@cn0854 ~]$ cd /lscratch/$SLURM_JOB_ID

[user@cn0854 48899347]$ cat >inputfile.inp<<"EOF"
! B3LYP def2-SVP Opt
# My first ORCA calculation
*xyz 0 1
O        0.000000000      0.000000000      0.000000000
H        0.000000000      0.759337000      0.596043000
H        0.000000000     -0.759337000      0.596043000
*
EOF

[user@cn0854 48899347]$ module load ORCA
[+] Loading ORCA  4.2.1  on cn0854
[+] Loading openmpi 3.1.4  for Intel 2019.4.243
[+] Loading Intel 2019.4.243  Compilers ...

[user@cn0854 48899347]$ orca inputfile.inp > output.out

[user@cn0854 48899347]$ ls
inputfile.engrad  inputfile.inp  inputfile.prop          inputfile_trj.xyz  output.out
inputfile.gbw     inputfile.opt  inputfile_property.txt  inputfile.xyz

[user@cn0854 48899347]$ tail output.out

Timings for individual modules:

Sum of individual times         ...        6.073 sec (=   0.101 min)
GTO integral calculation        ...        1.165 sec (=   0.019 min)  19.2 %
SCF iterations                  ...        3.866 sec (=   0.064 min)  63.7 %
SCF Gradient evaluation         ...        0.859 sec (=   0.014 min)  14.1 %
Geometry relaxation             ...        0.183 sec (=   0.003 min)   3.0 %
                             ****ORCA TERMINATED NORMALLY****
TOTAL RUN TIME: 0 days 0 hours 0 minutes 6 seconds 294 msec

[user@cn0854 48899347]$ exit
exit
salloc.exe: Relinquishing job allocation 48899347

[user@biowulf ~]$ 

Batch job
Most jobs should be run as batch jobs.

ORCA is made to run in parallel with MPI. To run ORCA in parallel, you must add the nprocs directive to the .inp file using the %pal keyword.

The following .inp file is similar to the one used in the Interactive job section above, with changes in bold. This code block tells ORCA to run 4 MPI processes.
! B3LYP def2-SVP Opt
%pal
nprocs 4
end
# My first ORCA calculation
*xyz 0 1
O        0.000000000      0.000000000      0.000000000
H        0.000000000      0.759337000      0.596043000
H        0.000000000     -0.759337000      0.596043000
*

Create a batch input file (e.g. ORCA.sh). Note that ORCA will use the mpirun command internally. For example, the following will run 4 tasks on a single node. The relevant SBATCH parameters can be edited to run the same job on multiple nodes.

#!/bin/bash

#SBATCH -N 1
#SBATCH --tasks-per-node=4
#SBATCH --ntasks=4
#SBATCH --partition=norm
#SBATCH --cpus-per-task=1
#SBATCH --job-name=ORCA

job=${SLURM_JOB_NAME}
job=$(echo ${job%%.*})

module load ORCA

export scratchlocation=/scratch/$USER
tdir=$(mktemp -d ${scratchlocation}/orcajob__${SLURM_JOB_ID}-XXXX)

cp  ${SLURM_SUBMIT_DIR}/*.inp ${tdir}/

# Creating nodefile in tdir
echo $SLURM_NODELIST > ${tdir}/${job}.nodes

cd $tdir

mkdir -p ${SLURM_SUBMIT_DIR}/results
echo $SLURM_NODELIST >> ${SLURM_SUBMIT_DIR}/results/${job}.out

export OMPI_MCA_base_verbose=9 OMPI_MCA_btl_base_verbose=9 OMPI_MCA_mtl_base=9

${ORCA_HOME}/orca ${tdir}/*.inp >>  ${SLURM_SUBMIT_DIR}/results/${job}.out

cp ${tdir}/* ${SLURM_SUBMIT_DIR}/results

Submit this job using the Slurm sbatch command.

sbatch ORCA.sh
Swarm of Jobs
A swarm of jobs is an easy way to submit a set of independent commands requiring identical resources.

Create a swarmfile (e.g. ORCA.swarm). For example:

orca inputfile1.inp >output1.out
orca inputfile2.inp >output2.out
orca inputfile3.inp >output3.out
orca inputfile4.inp >output4.out

Submit this job using the swarm command.

swarm -f ORCA.swarm [-g #] [-t #] --module ORCA
where
-g # Number of Gigabytes of memory required for each process (1 line in the swarm command file)
-t # Number of threads/CPUs required for each process (1 line in the swarm command file).
--module ORCA Loads the ORCA module for each subjob in the swarm