Jun 2023: Cluster update from Centos7 to Rocky8

Coincident with the update of the Biowulf operating system there were several major changes to the python setup:

• Python 2.7 is no longer provided in any form. Python 2.7 must be run from user environments
• The python/2.7 and python/3.7 environments and modules were retired.
The operating system Python is now Python 3.6 at /usr/bin/python3
• /usr/local/bin/python is now the Python 3.9 interpter
• A python/3.10 module is now available and the default version
• User environments that were using symbolic links to the central package cache likely were broken during the transition and will have to be rebuilt.

Jan 2022

• A python/3.9 module is now available
• The python/3.8 module is now the default python module
• /usr/local/bin/python3 is now python 3.8
• The python/3.5 and 3.6 modules are deprecated. Please move to python>=3.7.
• Various package updates.
• mpi4py has been updated and migrated to a different MPI library. Use mpiexec instead of srun

Sep 2020: python/3.8 becomes available

Apr 2020: python/3.7 becomes the default module

The default python module changed to python/3.7 since python/2.7 is not supported any longer. python/2.7 continues to be available and /usr/local/bin/python still points to python 2.7.

Jul 2019: python/3.7 becomes available

Jun 2018: Cluster update from RHEL6 to RHEL7

Coincident with the update of the Biowulf operating system there were several major changes to the python setup:

• The python/2.7.X environments and modules were retired.
• The Anaconda/X environments and modules were retired.
• conda itself is not exposed to users any more. Please use a private installation of conda to build your own environments
• OMP_NUM_THREADS is set to 1. To make use of implicit multithreading in the numerical packages this environment variable has to be set explicitly to a value greater than 1.
• User environments that were using symbolic links to the central package cache likely were broken during the transition and will have to be rebuilt.

This is a list of common pitfalls for python users on the cluster. Some of these are discussed in other sections of this documentation in more detail.

Some commonly use command line tools are also included in the python environments

Some of the python packages in our environments depend on commonly used command line tools. Since our environments are created with conda, this means conda insists on installing these tools along with the python package. A common example is samtools/bcftools. This means loading a python module after loading a samtools module will result in samtools from the python environment appearing first in your PATH.

Implicit multithreading while using multiprocessing

Certain python libraries can use implicit multithreading to speed up some numerical algorithms. The number of threads used for this is determined by the environment variable OMP_NUM_THREADS. If this is done while also using multiprocessing then batch jobs can be overloaded. For example, if creating a multiprocessing jobs with 10 worker processes and setting OMP_NUM_THREADS also to 10, a total of 10 * 10 = 100 threads are created. To avoid this happening accidentally the python modules set OMP_NUM_THREADS to 1. It needs to be set to more than one explicitly to take advantage of parallelized math libraries.

Overloaded jobs due to nested parallelism

One example of this is passing a model that parallelize such as XGBClassifier (xgboost) to RandomizedSearchCV (scikit-learn) which parallelizes the search across a parameter space. Both will by default attempt to use all CPUs. So if there are N CPUs detected, RandomizedSearch will start N processes running XGBClassifier and each process will run N threads. This can be prevented by explicitly specifying the number of parallel threads/processes at each level. In this case, both components take `n_jobs` as an argument. The product of the two should equal the number of available CPUs.

Implicit multithreading without setting OMP_NUM_THREADS

If OMP_NUM_THREADS is not set, the numerical libraries will attempt use all CPUs on a compute node. Unless all CPUs have been allocated this will result in overloading the job. The general use python modules set OMP_NUM_THREADS to 1 requiring users to explicitly set the variable higher to take advantage of implicit multithreading. If you are using your own python environments, please be sure to set OMP_NUM_THREADS explicitly for all code that can potentially multithread.

multiprocessing.cpu_count()

The cpu_count() function of the multiprocessing package always reports the total CPU count for a node. This is often the cause for overloaded python batch jobs. Instead query the environment for SLURM_CPUS_PER_TASK and default to 2, the minimum allocation.

[user@biowulf ~]$ sinteractive --cpus-per-task=2
...
[user@cn3444 ~]$ module load python/3.9
[user@cn3444 ~]$ python
Python 3.9.9 | packaged by conda-forge | (main, Dec 20 2021, 02:41:03)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing
>>> multiprocessing.cpu_count()
56  # <-- wrong
>>> import os
>>> int(os.environ.get('SLURM_CPUS_PER_TASK', '2'))
2   # <-- right

Hung multiprocessing pools

If a worker in a multiprocessing pool gets killed as can happen if a job exceeds allocated memory, the whole pool will wait indefinitely (i.e. until the job times out) for the killed worker to return results. Make sure to allocate sufficient memory and test code.

Python startup problem

Python looks through a lot of paths for certain files during startup. Therefore, starting up many concurrent python processes can strain the shared file system. This is especially true if running large swarms where each subjob runs multiple short python processes. If you have very short running python processes, please modify the code such that each script does more work.

Packages installed in the home directory

Python includes ~/.local/lib/pythonX.X/site-packages in the package search path. If you have packages installed there they can override the centrally installed packages. This can lead to hard to diagnose problems. If you have problems that are resolved by adding the -s option a broken package installed in your home directory may be the cause.

matplotlib in non-interactive environments

By default matplotlib uses an interactive backend. This works if the backend can connect to a graphical user interface (e.g. if using X11 forwarding through an ssh tunnel or NX). However, if there is no GUI to connect to, this will fail with an error message that includes.

Could not connect to any X display

This can be solved by using a non-interactive backend for plotting. Which backend matplotlib uses can be changed in a couple of different ways:

matplotlib settings can be modified using a matplotlibrc file. This file can be placed either in the current working directory or in ~/.config/matplotlib. If placed in the former it will only affect jobs run from this directory. If placed in the latter, it will affect all your calls to matplotlib. For example, if you'd like to change the default backend, create the file ~/.config/matplotlib/matplotlibrc with the following content:

backend: agg

Alternatively, you can set the MPLBACKEND environment variable either ad hoc or in your .bashrc:

export MPLBACKEND=agg

And finally, it can also be set programatically:

>>> import matplotlib
>>> matplotlib.use("agg")

Available backends and the currently active backends can be queried interactively in python:

>>> import matplotlib
>>> matplotlib.rcsetup.interactive_bk
['GTK', 'GTKAgg', 'GTKCairo', 'GTK3Agg', 'GTK3Cairo', 'nbAgg', 'Qt4Agg', 'Qt4Cairo', 'Qt5Agg', 'Qt5Cairo', 'TkAgg', 'TkCairo', 'WebAgg', 'WX', 'WXAgg', 'WXCairo']
>>> matplotlib.rcsetup.non_interactive_bk
['agg', 'cairo', 'gdk', 'pdf', 'pgf', 'ps', 'svg', 'template']
>>> import matplotlib.pyplot as plt
>>> plt.get_backend()
'agg'

python environments and NoMachine problems

If you add the executables of the conda dbus package to your PATH in one of your startup files (e.g. ~/.bashrc), NoMachine may fail with a black screen. This can happen when you load a python module or activate your own conda installation. The solution is to remove any unnecessary initialization from your .bashrc or, if this is an environment under your control, remove the dbus package from the environment.

The general purpose Python environments are made available through the module system:

[user@cn3444 ~]$ module -t avail python
/usr/local/lmod/modulefiles:
python/2.7
python/3.6
python/3.7
python/3.8
python/3.9

[user@cn3444 ~]$ module load python/3.8
[+] Loading python 3.9  ...

[user@cn3444 ~]$ python
Python 3.9.9 | packaged by conda-forge | (main, Dec 20 2021, 02:41:03)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> 

These are fully featured (conda) environments with many installed packages including the usual modules that make up the scientific Python stack (see below). These environments best suited for development and for running code that is not dependent on specific versions of the installed packages. Packages in these environments are updated regularly. If you need more stability to ensure reproducibility or because your code depends on the presence of certain fixed versions of packages you can also build your own environments.

Python scientific stack

Because they are compiled against MKL, some mathematical operations (e.g. SVD) can make use of multithreading to accelerate computation. The number of threads such operations will create is determined by the environment variable OMP_NUM_THREADS. To avoid accidentally overloading jobs, especially when also doing multiprocessing, this variable is set to 1 by the python modules. If your code can take advantage of implicit parallelism you can set this variable to match the number of allocated CPUs for cluster jobs or adjust it such that the product of OMP_NUM_THREADS * number of processes equals the number of allocated CPUS. For example, if allocating 16 CPUs and planning on using multiprocessing with 4 workers, OMP_NUM_THREADS can be set as high as 4 without overloading the allocation.

Some common issues with using multiprocessing on biowulf were already pointed out in the common pitfalls section. Beyond those multiprocessing can be made more robust by setting workers up to ignore the SIGINT signal so that a multiprocessing script can be terminated cleanly with scancel or Ctrl-C. The following script also makes sure to correctly detect the number of available CPUs in batch jobs:

#! /usr/bin/env python

from multiprocessing import Pool
import signal
import os

def init_worker():
    """
    This is necessary to be able to interrupt the
    script with CTRL-C (or scancel for that matter).
    It makes sure that the workers ignore SIGINT so
    that any SIGINT sent goes to the master process
    """
    signal.signal(signal.SIGINT, signal.SIG_IGN)

def worker(i):
    return i*i

if __name__ == '__main__':
    # the the number of allocated cpus (or 2 if not run as a slurm job)
    nproc = int(os.environ.get("SLURM_CPUS_PER_TASK", "2"))
    print("Running on %d CPUs" % nproc)
    # set up 50 tasks
    tasks = range(0, 100)
    p = Pool(nproc, init_worker)
    try:
        # run the processing pool
        results = p.map(worker, tasks)
    except (KeyboardInterrupt, SystemExit):
        p.terminate()
        p.join()
        sys.exit(1)
    else:
        p.close()
        p.join()
        # result summary
        print("\n".join("%d * %d = %d" % (a, a, b) for a, b in zip(tasks, results)))

It is also important to benchmark scaling of your code. Many algorithms won't scale well beyond a certain number of parallel multiprocessing workers which can lead to very inefficient resource usage (e.g. allocating 56 CPUs when parallel efficiency drops below 50% at 24 CPUs).

In order to use the rpy2 package on biowulf it is necessary to load a separate rpy2 module which allows the package to find the correct R installation.

[user@cn3444 ~]$ module load python/3.7 rpy2
Python 3.7.5 (default, Oct 25 2019, 15:51:11)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import rpy2                                                       
>>> import rpy2.robjects as robjects                                  
>>> pi = robjects.r['pi']                                             
>>> pi[0]                                                             
3.141592653589793                                                     

Ray is a framework for distributed computing. It accomplishes this by

Ray can be used to parallelize work within a single cluster in which case it is run as a standard single node job. To run a ray cluster on biowulf and parallelize across several nodes, multinode jobs with one task per node are used. The tasks don't have to be exclusive but for real world use they often will be. If you allocate the nodes exclusively make sure to also allocate all resources available on the node.

An example script you can use to configure your own ray workloads is available on GitHub.

[user@biowulf ~]$ git clone https://github.com/NIH-HPC/biowulf_ray.git
[user@biowulf ~]$ sbatch submit-ray

Note: ray, like many other tools, will by default try to use all resources on a node even if they were not all allocated to ray. The sample script above specifically specifies resources to the workers and the cluster head process. Similarly, if you use `ray.init` in, for example, a single node job make sure to specify the number of cpus, gpus, and the memory

Parsl can be used to execute python code across multiple cores and nodes. Here is a brief tutorial on how to use parsl on Biowulf. It assumes that you already have a conda installation.

[user@biowulf ~]$ sinteractive --mem=8g --cpus-per-task=6
salloc: Pending job allocation 9569310
salloc: job 9569310 queued and waiting for resources
salloc: job 9569310 has been allocated resources
salloc: Granted job allocation 9569310
salloc: Waiting for resource configuration
salloc: Nodes cn3444 are ready for job

[user@cn3444]$ ## set up an environment
[user@cn3444]$ source myconda
[user@cn3444]$ mamba create -n parsl parsl=2025
...
[user@cn3444]$ mamba activate parsl


[user@cn3444]$ ## run a local test
[user@cn3444]$ cat > test_local.py <<__EOF__ 
import statistics
import sys
import parsl
from parsl.providers import LocalProvider
from parsl.executors import HighThroughputExecutor

### a parsl python app
@parsl.python_app
def estimate_pi(num_darts: int, seed: int) -> float:
    """Estimate pi using the dartboard method.
    Args:
        num_darts: The number of darts to throw.
    Returns:
        An estimation of pi.
    """
    import random
    random.seed(seed)

    inside_circle = 0
    for _ in range(num_darts):
        x = random.uniform(-1, 1)
        y = random.uniform(-1, 1)
        if x**2 + y**2 <= 1:
            inside_circle += 1
    return 4.0 * inside_circle / num_darts

local_procs = parsl.Config(
    executors=[
        HighThroughputExecutor(
            address = "",
            label="htex_local",
            worker_debug=True,
            cores_per_worker=1,
            max_workers_per_node=6,
            provider=LocalProvider(
                init_blocks=1,
                max_blocks=1,
            ),
        )
    ],
)

n = 12
with parsl.load(local_procs):
    estimates = []
    for i in range(n):
        estimates.append(estimate_pi(10**7, i))

    # IMPORTANT: this has to be in the 'with' block!
    results = [a.result() for a in estimates]
    for result in results:
        print(f"pi ≈ {result:12.10f}")
    print("------------------")
    print(f"     {statistics.mean(results):12.10f}")
__EOF__

[user@cn3444]$ python test_local.py
pi ≈ 3.1417104000
pi ≈ 3.1411884000
pi ≈ 3.1422124000
[...snip...]
pi ≈ 3.1413860000
------------------
     3.1415945667


[user@cn3444]$ ## run a test where work is processed in slurm jobs

[user@cn3444]$ cat <<__EOF__ > test_slurm.py
import statistics
import sys
import parsl
from parsl.providers import SlurmProvider
from parsl.executors import HighThroughputExecutor
from parsl.launchers import SrunLauncher
from parsl.addresses import address_by_interface

### a parsl python app
@parsl.python_app
def estimate_pi(num_darts: int, seed: int) -> float:
    """Estimate pi using the dartboard method.
    Args:
        num_darts: The number of darts to throw.
    Returns:
        An estimation of pi.
    """
    import random
    random.seed(seed)

    inside_circle = 0
    for _ in range(num_darts):
        x = random.uniform(-1, 1)
        y = random.uniform(-1, 1)
        if x**2 + y**2 <= 1:
            inside_circle += 1
    return 4.0 * inside_circle / num_darts

# slurm config
slurm = parsl.Config(
    executors=[
        HighThroughputExecutor(
            label="biowulf_htex",
            address=address_by_interface("ib0"),
            cores_per_worker=1,
            provider=SlurmProvider(
                nodes_per_block=1,
                cores_per_node=12,
                mem_per_node=24,
                exclusive=False,
                walltime="4:00:00",
                init_blocks=1,
                max_blocks=2,
                partition="norm",
                constraint="x2695",
                launcher=SrunLauncher(),
            ),
        )
    ],
)

n = 96
with parsl.load(slurm):
    estimates = []
    for i in range(n):
        estimates.append(estimate_pi(10**8, i))

    # IMPORTANT: this has to be in the 'with' block!
    results = [a.result() for a in estimates]
    for result in results:
        print(f"pi ≈ {result:12.10f}")
    print("------------------")
    print(f"     {statistics.mean(results):12.10f}")
__EOF__
[user@cn3444]$ export ORIG_SBATCH_OUTPUT=1  ### this is required
[user@cn3444]$ python test_slurm.py
pi ≈ 3.1414602000
pi ≈ 3.1415175600
[...snip...]
pi ≈ 3.1415930400
------------------
     3.1416134600

[user@cn3444]$ exit

In the slurm example above up to 2 jobs (nodes_per_block=1 and max_blocks=2) with 12 CPUs/24GB on x2695 nodes in the norm partition are allocated non-exclusively and one worker for each CPU is started to process tasks.

When using GPUs you generally want single node blocks with a single worker per GPU. This can be done by specifying the number of GPUs available in the configuration like in this example config

slurm = parsl.Config(
    executors=[
        HighThroughputExecutor(
            label="biowulf_htex",
            address=address_by_interface("ib0"),
            available_accelerators=4,
            provider=SlurmProvider(
                nodes_per_block=1,
                cores_per_node=56,
                mem_per_node=120,
                exclusive=True,
                walltime="4:00:00",
                init_blocks=1,
                max_blocks=2,
                partition="gpu",
                scheduler_options="#SBATCH --gres=gpu:k80:4",
                launcher=SrunLauncher(),
            ),
        )
    ],
)

Alternatively workers can also be bound to more than one GPU. For example to have 2 GPUs/worker change the available_accelerators to

available_accelerators=["0,1", "2,3"]

Spyder is a Python IDE focused on scientific python. It has the ability to connect to a remote spyder kernel running on a compute node using ssh tunnels. Currently the setup is not the most convenient but this may improve in the future.

Create an sinteractive session with the resources you need and 5 tunnels

[user@biowulf ~]$ sinteractive -TTTTT --mem=12g --cpus-per-task=2
salloc: Pending job allocation 9569310
salloc: job 9569310 queued and waiting for resources
salloc: job 9569310 has been allocated resources
salloc: Granted job allocation 9569310
salloc: Waiting for resource configuration
salloc: Nodes cn4270 are ready for job

Created 5 generic SSH tunnel(s) from this compute node to
biowulf for your use at port numbers defined
in the $PORTn ($PORT1, ...) environment variables.


Please create a SSH tunnel from your workstation to these ports on biowulf.
On Linux/MacOS, open a terminal and run:

    ssh  -L 41699:localhost:41699 -L 40387:localhost:40387 -L 42289:localhost:42289 [...snip...]

For Windows instructions, see https://hpc.nih.gov/docs/tunneling
[user@cne444]$

Follow our tunneling instructions to create the tunnels from your local to biowulf. Then start a spyder kernel in the sinteractive session. In the example below we are using one of the general purpose python modules but you can also install the spyder kernel module into your own environment

[user@cn4270]$ module load python/3.10
[user@cn4270]$ spyder_kernel start
starting kernel with python '/usr/local/Anaconda/envs/py3.10/bin/python' - please hold on
----------------------------------------------------
spyder kernel started successfully
----------------------------------------------------
Kernel connection file copied to '~/kernel-652313.json' for convenience
You can use this file via hpcdrive to establish a connection to this kernel
using a remote kernel with host biowulf.nih.gov
Delete the kernel file when you are done
[user@cn4270]$ spyder_kernel status
spyder kernel state file exits
spyder kernel is running

The spyder_kernel helper copies a connection file to your home directory. In the example above ~/kernel-652313.json. You can use this file to connect with your locally installed Spyder to the running kernel by either copying the file to your system or mounting your home directory as a hpcdrive. In the example below I am using the connection file directly via hpcdrive:

Since we are forwarding all the required ports it isn't necessary to check the 'remote connection' box. Alternatively you can skip setting up the local tunnels with the command generated by sinteractive and instead provide your login credentials in the remote kernel section of the dialog. Once the connection has been established verify that you are on a compute node

Stop the kernel when you are done by running exit() in the spyder session or exit from the sinteractive session with

[user@cn4270]$ spyder_kernel stop
spyder kernel has been stopped

• Python home
• Python 3.X documentation
• Python 2.X documentation
• Python 2 or 3?
• Transitioning to Python 3
• Porting Python 2 Code to Python 3