Quick Links Changelog Common pitfalls General purpose python environments Installed packages Multiprocessing Using mpi4py Using rpy2 Using ray Creating python environments Documentation

Jan 2022

• A python/3.9 module is now available
• The python/3.8 module is now the default python module
• /usr/local/bin/python3 is now python 3.8
• The python/3.5 and 3.6 modules are deprecated. Please move to python>=3.7.
• Various package updates.
• mpi4py has been updated and migrated to a different MPI library. Use mpiexec instead of srun

Sep 2020: python/3.8 becomes available

Apr 2020: python/3.7 becomes the default module

The default python module changed to python/3.7 since python/2.7 is not supported any longer. python/2.7 continues to be available and /usr/local/bin/python still points to python 2.7.

Jul 2019: python/3.7 becomes available

Jun 2018: Cluster update from RHEL6 to RHEL7

Coincident with the update of the Biowulf operating system there were several major changes to the python setup:

• The python/2.7.X environments and modules were retired.
• The Anaconda/X environments and modules were retired.
• conda itself is not exposed to users any more. Please use a private installation of conda to build your own environments
• OMP_NUM_THREADS is set to 1. To make use of implicit multithreading in the numerical packages this environment variable has to be set explicitly to a value greater than 1.
• User environments that were using symbolic links to the central package cache likely were broken during the transition and will have to be rebuilt.

This is a list of common pitfalls for python users on the cluster. Some of these are discussed in other sections of this documentation in more detail.

Status 403 error for conda defaults channels

If you have a miniconda or similar install and are getting an error similar to

RuntimeError: Multi-download failed. Reason: Transfer finalized, status: 403
    [https://repo.anaconda.com/pkgs/r/noarch/repodata.json] 4020 bytes

then you can fix it for most cases by disabling the default channels and adding alternative channels. This can be done with by adding the following lines to a .condarc file at the top level of the miniconda install (i.e. $CONDA_ROOT/.condarc). You might also want to set strict channel priority at the same time if you haven't already done so:

channels:
  - conda-forge
  - bioconda
defaults: []
channel_priority: strict

You should also ensure that your ~/.condarc does not explicitly set channels or defaults.

Some commonly use command line tools are also included in the python environments

Some of the python packages in our environments depend on commonly used command line tools. Since our environments are created with conda, this means conda insists on installing these tools along with the python package. A common example is samtools/bcftools. This means loading a python module after loading a samtools module will result in samtools from the python environment appearing first in your PATH.

Implicit multithreading while using multiprocessing

Certain python libraries can use implicit multithreading to speed up some numerical algorithms. The number of threads used for this is determined by the environment variable OMP_NUM_THREADS. If this is done while also using multiprocessing then batch jobs can be overloaded. For example, if creating a multiprocessing jobs with 10 worker processes and setting OMP_NUM_THREADS also to 10, a total of 10 * 10 = 100 threads are created. To avoid this happening accidentally the python modules set OMP_NUM_THREADS to 1. It needs to be set to more than one explicitly to take advantage of parallelized math libraries.

Overloaded jobs due to nested parallelism

One example of this is passing a model that parallelize such as XGBClassifier (xgboost) to RandomizedSearchCV (scikit-learn) which parallelizes the search across a parameter space. Both will by default attempt to use all CPUs. So if there are N CPUs detected, RandomizedSearch will start N processes running XGBClassifier and each process will run N threads. This can be prevented by explicitly specifying the number of parallel threads/processes at each level. In this case, both components take `n_jobs` as an argument. The product of the two should equal the number of available CPUs.

Implicit multithreading without setting OMP_NUM_THREADS

If OMP_NUM_THREADS is not set, the numerical libraries will attempt use all CPUs on a compute node. Unless all CPUs have been allocated this will result in overloading the job. The general use python modules set OMP_NUM_THREADS to 1 requiring users to explicitly set the variable higher to take advantage of implicit multithreading. If you are using your own python environments, please be sure to set OMP_NUM_THREADS explicitly for all code that can potentially multithread.

multiprocessing.cpu_count()

The cpu_count() function of the multiprocessing package always reports the total CPU count for a node. This is often the cause for overloaded python batch jobs. Instead query the environment for SLURM_CPUS_PER_TASK and default to 2, the minimum allocation.

[user@biowulf ~]$ sinteractive --cpus-per-task=2
...
[user@cn3444 ~]$ module load python/3.9
[user@cn3444 ~]$ python
Python 3.9.9 | packaged by conda-forge | (main, Dec 20 2021, 02:41:03)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing
>>> multiprocessing.cpu_count()
56  # <-- wrong
>>> import os
>>> int(os.environ.get('SLURM_CPUS_PER_TASK', '2'))
2   # <-- right

Hung multiprocessing pools

If a worker in a multiprocessing pool gets killed as can happen if a job exceeds allocated memory, the whole pool will wait indefinitely (i.e. until the job times out) for the killed worker to return results. Make sure to allocate sufficient memory and test code.

Python startup problem

Python looks through a lot of paths for certain files during startup. Therefore, starting up many concurrent python processes can strain the shared file system. This is especially true if running large swarms where each subjob runs multiple short python processes. If you have very short running python processes, please modify the code such that each script does more work.

Packages installed in the home directory

Python includes ~/.local/lib/pythonX.X/site-packages in the package search path. If you have packages installed there they can override the centrally installed packages. This can lead to hard to diagnose problems. If you have problems that are resolved by adding the -s option a broken package installed in your home directory may be the cause.

Conda install filled up /home

At this time we do not allow requests to increase the quota on home directories. A conda install with some moderately sized environments can fill up that space quickly. We therefore advise against installing conda in your home directory. /data is a better place.

Conda activate added to shell init

Adding conda initialization to your .bashrc/.cshrc/.zshrc can cause problems with NoMachine/VNC (and other issues). We recommend not allowing conda to initialize during shell startup.

Swarm and Python environments

If activating one of your own environment within a swarm some versions of conda may encounter a race condition that will lead to some swarm subjobs failing to activate the environment correctly. This can be avoided by either activating the conda environment before submitting the swarm (the environment gets exported to the job) or by calling python with the full path to your environment without activating the environment. Update: This should not be happening any longer but leaving note here in case there is a regression.

matplotlib in non-interactive environments

By default matplotlib uses an interactive backend. This works if the backend can connect to a graphical user interface (e.g. if using X11 forwarding through an ssh tunnel or NX). However, if there is no GUI to connect to, this will fail with an error message that includes.

Could not connect to any X display

This can be solved by using a non-interactive backend for plotting. Which backend matplotlib uses can be changed in a couple of different ways:

matplotlib settings can be modified using a matplotlibrc file. This file can be placed either in the current working directory or in ~/.config/matplotlib. If placed in the former it will only affect jobs run from this directory. If placed in the latter, it will affect all your calls to matplotlib. For example, if you'd like to change the default backend, create the file ~/.config/matplotlib/matplotlibrc with the following content:

backend: agg

Alternatively, you can set the MPLBACKEND environment variable either ad hoc or in your .bashrc:

export MPLBACKEND=agg

And finally, it can also be set programatically:

>>> import matplotlib
>>> matplotlib.use("agg")

Available backends and the currently active backends can be queried interactively in python:

>>> import matplotlib
>>> matplotlib.rcsetup.interactive_bk
['GTK', 'GTKAgg', 'GTKCairo', 'GTK3Agg', 'GTK3Cairo', 'nbAgg', 'Qt4Agg', 'Qt4Cairo', 'Qt5Agg', 'Qt5Cairo', 'TkAgg', 'TkCairo', 'WebAgg', 'WX', 'WXAgg', 'WXCairo']
>>> matplotlib.rcsetup.non_interactive_bk
['agg', 'cairo', 'gdk', 'pdf', 'pgf', 'ps', 'svg', 'template']
>>> import matplotlib.pyplot as plt
>>> plt.get_backend()
'agg'

python environments and NoMachine problems

If you add the executables of the conda dbus package to your PATH in one of your startup files (e.g. ~/.bashrc), NoMachine may fail with a black screen. This can happen when you load a python module or activate your own conda installation. The solution is to remove any unnecessary initialization from your .bashrc or, if this is an environment under your control, remove the dbus package from the environment.

The general purpose Python environments are made available through the module system:

[user@cn3444 ~]$ module -t avail python
/usr/local/lmod/modulefiles:
python/2.7
python/3.6
python/3.7
python/3.8
python/3.9

[user@cn3444 ~]$ module load python/3.8
[+] Loading python 3.9  ...

[user@cn3444 ~]$ python
Python 3.9.9 | packaged by conda-forge | (main, Dec 20 2021, 02:41:03)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> 

These are fully featured (conda) environments with many installed packages including the usual modules that make up the scientific Python stack (see below). These environments best suited for development and for running code that is not dependent on specific versions of the installed packages. Packages in these environments are updated regularly. If you need more stability to ensure reproducibility or because your code depends on the presence of certain fixed versions of packages you can also build your own environments.

Python scientific stack

The usual scientific python stack (numpy, scipy, scikit-learn, numba, pandas, ...) is included in all the general purpose environments. Numpy and scipy are compiled against the accelerated Intel MKL library.

Because they are compiled against MKL, some mathematical operations (e.g. SVD) can make use of multithreading to accelerate computation. The number of threads such operations will create is determined by the environment variable OMP_NUM_THREADS. To avoid accidentally overloading jobs, especially when also doing multiprocessing, this variable is set to 1 by the python modules. If your code can take advantage of implicit parallelism you can set this variable to match the number of allocated CPUs for cluster jobs or adjust it such that the product of OMP_NUM_THREADS * number of processes equals the number of allocated CPUS. For example, if allocating 16 CPUs and planning on using multiprocessing with 4 workers, OMP_NUM_THREADS can be set as high as 4 without overloading the allocation.

The following packages are available in the general purpose python environments

Some common issues with using multiprocessing on biowulf were already pointed out in the common pitfalls section. Beyond those multiprocessing can be made more robust by setting workers up to ignore the SIGINT signal so that a multiprocessing script can be terminated cleanly with scancel or Ctrl-C. The following script also makes sure to correctly detect the number of available CPUs in batch jobs:

#! /usr/bin/env python

from multiprocessing import Pool
import signal
import os

def init_worker():
    """
    This is necessary to be able to interrupt the
    script with CTRL-C (or scancel for that matter).
    It makes sure that the workers ignore SIGINT so
    that any SIGINT sent goes to the master process
    """
    signal.signal(signal.SIGINT, signal.SIG_IGN)

def worker(i):
    return i*i

if __name__ == '__main__':
    # the the number of allocated cpus (or 2 if not run as a slurm job)
    nproc = int(os.environ.get("SLURM_CPUS_PER_TASK", "2"))
    print("Running on %d CPUs" % nproc)
    # set up 50 tasks
    tasks = range(0, 100)
    p = Pool(nproc, init_worker)
    try:
        # run the processing pool
        results = p.map(worker, tasks)
    except (KeyboardInterrupt, SystemExit):
        p.terminate()
        p.join()
        sys.exit(1)
    else:
        p.close()
        p.join()
        # result summary
        print("\n".join("%d * %d = %d" % (a, a, b) for a, b in zip(tasks, results)))

It is also important to benchmark scaling of your code. Many algorithms won't scale well beyond a certain number of parallel multiprocessing workers which can lead to very inefficient resource usage (e.g. allocating 56 CPUs when parallel efficiency drops below 50% at 24 CPUs).

The python MPI library mpi4py is available in the python modules. In order to use this package it is necessary to load an additional module (mpi4py) to ensure that it can find the correct OpenMPI libraries. Here is an example for how to use this library:

The python MPI script:

#! /usr/bin/env python

from mpi4py import MPI

comm = MPI.COMM_WORLD

rank = comm.Get_rank()
size = comm.Get_size()
print("Hello! I'm rank {0} / {1} running in total...".format(rank, size))

comm.Barrier()

The batch script:

#!/bin/bash
#SBATCH --ntasks=8
#SBATCH --ntasks-per-core=1
#SBATCH --partition=multinode

module load mpi4py
module load python/3.9
mpiexec  ./test.py
# or srun --mpi=pmi2 ./test.py

Submit with

[user@biowulf ~]$ sbatch test.sh

In order to use the rpy2 package on biowulf it is necessary to load a separate rpy2 module which allows the package to find the correct R installation.

[user@cn3444 ~]$ module load python/3.7 rpy2
Python 3.7.5 (default, Oct 25 2019, 15:51:11)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import rpy2                                                       
>>> import rpy2.robjects as robjects                                  
>>> pi = robjects.r['pi']                                             
>>> pi[0]                                                             
3.141592653589793                                                     

Ray is a framework for distributed computing. It accomplishes this by

1. Providing simple primitives for building and running distributed applications.
2. Enabling end users to parallelize single machine code, with little to zero code changes.
3. Including a large ecosystem of applications, libraries, and tools on top of the core Ray to enable complex applications.

Ray can be used to parallelize work within a single cluster in which case it is run as a standard single node job. To run a ray cluster on biowulf and parallelize across several nodes, multinode jobs with one task per node are used. The tasks don't have to be exclusive but for real world use they often will be. If you allocate the nodes exclusively make sure to also allocate all resources available on the node.

An example script you can use to configure your own ray workloads is available on GitHub.

[user@biowulf ~]$ git clone https://github.com/NIH-HPC/biowulf_ray.git
[user@biowulf ~]$ sbatch submit-ray

Note: ray, like many other tools, will by default try to use all resources on a node even if they were not all allocated to ray. The sample script above specifically specifies resources to the workers and the cluster head process. Similarly, if you use `ray.init` in, for example, a single node job make sure to specify the number of cpus, gpus, and the memory

If you need common python packages at fixed versions or python packages that may not be suitable for installation in the general purpose python environments you can create your own environments using conda/mamba. We recommend installing mambaforge (a derivative of miniconda that includes mamba and uses the conda-forge channel in place of the defaults channel) into your data directory (or a shared data directory for collaborations) and creating environments within that installation.

In the following example we will cover some basics of using conda to create private environments. Much more detailed documentation as available on the conda documentation pages. Note that conda/mamba operations should be done in sinteractive allocations rather than on the biowulf login node because they can be computationally intensive.

Private python environments should be created in data directories, not the home directory since a conda/mamba installation can easily grow in space to exceed the home directory allocation.

Installing mambaforge on the biowulf login node or on helix will not work out of the box due to restrictions on /tmp. Please use an sinteractive session to set up and manage conde. If you do want to use helix you can do so by setting TMPDIR to a location in your data directory

[user@biowulf]$ sinteractive --mem=20g --gres=lscratch:20
...
[user@cn3444]$ cd /data/$USER
[user@cn3444]$ export TMPDIR=/lscratch/$SLURM_JOB_ID

Download the mambaforge installer and install to a path in a data or shared directory like /data/$USER/conda.

[user@cn3444]$ wget https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-x86_64.sh
--2022-04-01 11:13:41--  https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-x86_64.sh
[...snip...]
Length: 92971376 (89M) [application/octet-stream]
Saving to: ‘Mambaforge-Linux-x86_64.sh’

100%[===================================================>] 92,971,376   111MB/s   in 0.8s

2022-04-01 11:13:42 (111 MB/s) - ‘Mambaforge-Linux-x86_64.sh’ saved [92971376/92971376]


[user@cn3444]$ bash Mambaforge-Linux-x86_64.sh -p /data/$USER/conda -b
PREFIX=/data/$USER/conda
Unpacking payload ...
Extracting "python-3.9.10-h85951f9_2_cpython.tar.bz2"
Extracting "_libgcc_mutex-0.1-conda_forge.tar.bz2"
[...snip...]
installation finished.
[user@cn3444]$ rm Mambaforge-Linux-x86_64.sh

To use the newly installed conda you will have to source an init file. Do this each time you are going to work with your environments. Do not allow conda/mamba to add automatic initialization to your startup files (e.g. .bashrc) as environments can interfere with login or nomachine..

After sourcing the conda init file, activate the base environment and update the conda package manager which itself is just a package:

[user@cn3444]$ source /data/$USER/conda/etc/profile.d/conda.sh && source /data/$USER/conda/etc/profile.d/mamba.sh

### to make things easier you can create a file called `myconda` in a directory
### on your path such as ~/bin. This could be done like so (assuming the same
### paths as we used here).

[user@cn3444]$ mkdir -p ~/bin
[user@cn3444]$ cat <<'__EOF__' > ~/bin/myconda
__conda_setup="$('/data/$USER/conda/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/data/$USER/conda/etc/profile.d/conda.sh" ]; then
        . "/data/$USER/conda/etc/profile.d/conda.sh"
    else
        export PATH="/data/$USER/conda/bin:$PATH"
    fi
fi
unset __conda_setup

if [ -f "/data/$USER/conda/etc/profile.d/mamba.sh" ]; then
    . "/data/$USER/conda/etc/profile.d/mamba.sh"
fi
__EOF__

### then from *anywhere* the mambaforge install can be activated with
[user@cn3444]$ source myconda

### let's not show the large mamba banner all the time
[user@cn3444]$ export MAMBA_NO_BANNER=1
[user@cn3444]$ mamba activate base
(base) [user@cn3444]$ which python
/data/$USER/conda/bin/python
(base) [user@cn3444]$ mamba update --all
Looking for: ['_libgcc_mutex', 'ca-certificates', 'ld_impl_linux-64', 'libstdcxx-ng', 'libgomp', '_openmp_mutex', 'libgcc-ng', 'yaml-cpp', 'yaml', 'xz', 'reproc', 'openssl', 'ncurses', 'lzo', 'lz4-c', 'libzlib', 'libuuid', 'libnsl', 'libiconv', 'libffi', 'libev', 'keyutils', 'icu', 'c-ares', 'bzip2', 'reproc-cpp', 'libedit', 'readline', 'zstd', 'zlib', 'tk', 'krb5', 'sqlite', 'libxml2', 'libssh2', 'libsolv', 'libnghttp2', 'libarchive', 'libcurl', 'libmamba', 'pybind11-abi', 'tzdata', 'python', 'python_abi', 'setuptools', 'wheel', 'pip', 'six', 'pycparser', 'idna', 'colorama', 'charset-normalizer', 'tqdm', 'ruamel_yaml', 'pysocks', 'pycosat', 'libmambapy', 'certifi', 'cffi', 'conda-package-handling', 'cryptography', 'brotlipy', 'pyopenssl', 'urllib3', 'requests', 'conda', 'mamba']

conda-forge/noarch                                   7.8MB @   3.8MB/s  2.2s
conda-forge/linux-64                                21.9MB @   3.6MB/s  6.6s

Pinned packages:
  - python 3.9.*

[...snip...]
  Change: 4 packages
  Upgrade: 5 packages

  Total download: 47MB

───────────────────────────────────────────────────────────────────────────────

Confirm changes: [Y/n] Y

[...snip...]


Preparing transaction: done
Verifying transaction: done
Executing transaction: done

(base) [user@cn3444]$ mamba clean --all --yes
Cache location: /data/$USER/conda/pkgs
Will remove the following tarballs:

/data/$USER/conda/pkgs
------------------------
python-3.9.5-h12debd9_4.tar.bz2             22.6 MB
[...snip...]
idna-3.2-pyhd3eb1b0_0.conda                   48 KB

---------------------------------------------------
Total:                                      59.4 MB

Removed python-3.9.5-h12debd9_4.tar.bz2
[...snip...]

Now let's create a new environment called project1 with an older version of pysam from the bioconda channel and python 3.7. For this we use mamba.

(base) [user@cn3444]$ mamba deactivate
[user@cn3444]$ mamba create -n project1 python=3.7 numpy scipy bioconda::pysam==0.15.3 samtools==1.9
Looking for: ['python=3.7', 'numpy', 'scipy', 'bioconda::pysam==0.15.3', 'samtools==1.9']

bioconda/linux-64                                    4.1MB @   3.8MB/s  1.2s
bioconda/noarch                                      3.5MB @   2.8MB/s  1.3s
conda-forge/noarch                                   7.8MB @   3.9MB/s  2.2s
conda-forge/linux-64                                21.9MB @   3.9MB/s  6.2s
Transaction

  Prefix: /data/$USER/conda/envs/project1

  Updating specs:

   - python=3.7
   - numpy
   - scipy
   - bioconda::pysam==0.15.3
   - samtools==1.9

[...snip...]

Confirm changes: [Y/n] Y
[...snip...]
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ mamba activate project1
#
# To deactivate an active environment, use
#
#     $ mamba deactivate

[user@cn3444]$ mamba activate project1
(project1) [user@cn3444]$ which python
/data/$USER/conda/envs/project1/bin/python
(project1) [user@cn3444]$ samtools --version
samtools 1.9
Using htslib 1.9
Copyright (C) 2018 Genome Research Ltd.
(project1) [user@cn3444]$ mamba deactivate
[user@cn3444]$

Now an environment for a different project with current pysam, some other tools, and numpy using the OpenBlas numerical libraries. This time we add the bioconda channel to the channels for the environment so we don't have to use the bioconda:: prefix. A common pattern for environments used for bioinformatic software is to set up bioconda and conda-forge channels on a per-environment basis. This allows conda-forge packages to override packages from the defaults channel. We also specify MKL for the numerical libraries and pin that so it won't change accidentally.

Note that at this point mamba does not yet include the config command.

[user@cn3444 temp]$ mamba create -n project2 python=3.8
[...snip...]
[user@cn3444]$ mamba activate project2
(project2) [user@cn3444]$ conda config --env --add channels bioconda
(project2) [user@cn3444]$ conda config --env --add channels conda-forge
Warning: 'conda-forge' already in 'channels' list, moving to the top
(project2) [user@cn3444]$ conda config --env --set channel_priority strict
(project2) [user@cn3444]$ conda config --env --add pinned_packages blas=*=mkl
(project2) [user@cn3444]$ conda config --show-sources
==> /data/$USER/conda/.condarc <==
channels:
  - conda-forge

==> /data/$USER/envs/project2/.condarc <==
pinned_packages:
  - libblas=*=*_mkl
  - python=3.8
channel_priority: strict
channels:
  - conda-forge
  - bioconda

(project2) [user@cn3444]$ mamba install -q pysam bedtools hisat2 blas numpy scipy
[...snip...]
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

(project2) [user@cn3444]$ which python
/data/$USER/conda/envs/project2/bin/python
(project2) [user@cn3444]$ mamba install tensorflow=*=cuda*
[...snip...]

Note that, pip can be used to install packages into conda environments as well. However, this can sometimes cause problems when pip overwrites existing conda-installed packages.

List environments in the current conda install, then deactivate environments

(project2) [user@cn3444]$ mamba info --env
# conda environments:
#
base                     /data/$USER/conda
project1                 /data/$USER/conda/envs/project1
project2              *  /data/$USER/conda/envs/project2
(project2) [user@cn3444]$ mamba deactivate
[user@cn3444]$ 

• Python home
• Python 3.X documentation
• Python 2.X documentation
• Python 2 or 3?
• Transitioning to Python 3
• Porting Python 2 Code to Python 3
• Conda documentation
• mamba