Here are some simple examples of compiling and running an MPICH2 program on Biowulf. More information about running MPICH2 jobs under Slurm is available in the online Slurm documentation about MPI jobs. There is a version of MPICH2 built with default parameters (module mpich2/1.5/gnu-4.4.7) and a version built with Slurm and PMI2 support (module mpich2/1.5/gnu-4.4.7+slurm+pmi2). The examples on this page use the PMI2 version.

The examples used on this page are provided with the MPICH2 v 1.5, and are available in /usr/local/MPICH2/examples/.

The "PI" calculation example

The 'cpi' program is already compiled. The source code and executable are available in the examples directory. Sample batch script:

#!/bin/bash
# this file is called run.bat

cd /data/$USER/mpi_tests
module load mpich2/1.5/gnu-4.4.7+slurm+pmi2 

echo "Running $SLURM_NTASKS processes"
srun -n $SLURM_NTASKS --mpi=pmi2 /usr/local/MPICH2/examples/cpi

Submit with:

sbatch --ntasks=4 --ntasks-per-core=1 run.bat

where

--ntasks=4	number of MPI processes
--ntasks-per-core=1	specifies that only one MPI process should be run on each physical core, i.e. do not use hyperthreaded cores. This is recommended for most parallel programs (but of course, is unlikely to make any difference for the PI calculation!)

The output will appear in a file called slurm-######.out (#### = Jobnumber), and will look something like this:

[user@biowulf ~]$ cat  slurm-3472687.out
Running 4 processes
Process 0 of 4 is on cn0030
Process 1 of 4 is on cn0030
Process 2 of 4 is on cn0030
Process 3 of 4 is on cn0030
pi is approximately 3.1415926544231239, Error is 0.0000000008333307
wall clock time = 0.000074

Compiling and running Hello World

The hellow.c program is available in /usr/local/MPICH2/examples. The sample session below uses Slurm's implementation of the PMI library.

biowulf% cp /usr/local/MPICH2/exampleshellow.c .

biowulf% module load mpich2/1.5/gnu-4.4.7+slurm+pmi2

biowulf% mpicc  -o mpi_hello hellow.c

The module mpich2/1.5/gnu-4.4.7+slurm+pmi2 sets the environment variable

LDFLAGS="-L/usr/local/slurm/lib -lpmi"

Thus, when compiling, you do not need to specify these parameters.

Create a batch script along the following lines:

#!/bin/bash
# this file is called run.bat

cd /data/user/mpi
#module load mpich2/1.5/gnu-4.4.7+slurm+pmi2 

echo "Running $SLURM_NTASKS processes"
srun -n $SLURM_NTASKS --mpi=pmi2 /data/user/mpi/mpi_hello

Submit with:

sbatch --ntasks=4 --ntasks-per-core=1 run.bat

The output from this program will look something like:

[user@biowulf ~]$ cat slurm-3472699.out
Running 4 processes
Hello world from process 1 of 4, running on cn0286
Hello world from process 2 of 4, running on cn0286
Hello world from process 3 of 4, running on cn0286
Hello world from process 0 of 4, running on cn0286