Quick Links Changelog Common pitfalls General purpose python environments Installed packages Multiprocessing Using mpi4py Using rpy2 Creating python environments Documentation

June 2018: Cluster update from RHEL6 to RHEL7

Coincident with the update of the Biowulf operating system there were several major changes to the python setup:

• The python/2.7.X environments and modules were retired.
• The Anaconda/X environments and modules were retired.
• conda itself is not exposed to users any more. Please use a private installation of conda to build your own environments
• OMP_NUM_THREADS is set to 1. To make use of implicit multithreading in the numerical packages this environment variable has to be set explicitly to a value greater than 1.
• User environments that were using symbolic links to the central package cache likely were broken during the transition and will have to be rebuilt.

July 2019: Installation of python/3.7

Installation of python/3.7

This is a list of common pitfalls for python users on the cluster. Some of these are discussed in other sections of this documentation in more detail.

Some commonly use command line tools are also included in the python environments

Some of the python packages in our environments depend on commonly used command line tools. Since our environments are created with conda, this means conda insists on installing these tools along with the python package. A common example is samtools/bcftools. This means loading a python module after loading a samtools module will result in samtools from the python environment appearing first in your PATH.

Implicit multithreading while using multiprocessing

Certain python libraries can use implicit multithreading to speed up some numerical algorithms. The number of threads used for this is determined by the environment variable OMP_NUM_THREADS. If this is done while also using multiprocessing batch jobs can be overloaded. For example, if creating a multiprocessing jobs with 10 worker processes and setting OMP_NUM_THREADS also to 10, a total of 10 * 10 = 100 threads is created. To avoid this happening accidentally the python modules set OMP_NUM_THREADS to 1. It needs to be set to more than one explicitly to take advantage of parallelized math libraries.

Implicit multithreading without setting OMP_NUM_THREADS

If OMP_NUM_THREADS is not set, the numerical libraries will attempt use all CPUs on a compute node. Unless all CPUs have been allocated this will result in overloading the job. The general use python modules set OMP_NUM_THREADS to 1 requiring users to explicitly set the variable higher to take advantage of implicit multithreading. If you are using your own python environments, please be sure to set OMP_NUM_THREADS explicitly for all code that can potentially multithread.

multiprocessing.cpu_count()

The cpu_count() function of the multiprocessing package always reports the total CPU count for a node. This is often the cause for overloaded python batch jobs. Instead query the environment for SLURM_CPUS_PER_TASK and default to 2, the minimum allocation.

[user@biowulf ~]$ sinteractive --cpus-per-task=2
...
[user@cn3444 ~]$ module load python/2.7
[user@cn3444 ~]$ python
Python 2.7.14 |Anaconda custom (64-bit)| (default, Mar 27 2018, 17:29:31)
>>> import multiprocessing
>>> multiprocessing.cpu_count()
56  # <-- wrong
>>> import os
>>> int(os.environ.get('SLURM_CPUS_PER_TASK', '2'))
2   # <-- right

Hung multiprocessing pools

If a worker in a multiprocessing pool gets killed as can happen if a job exceeds allocated memory, the whole pool will wait indefinately (i.e. until the job times out) for the killed worker to return results. Make sure to allocate sufficient memory and test code.

Python startup problem

Python looks through a lot of paths for certain files during startup. Therefore, starting up many concurrent python processes can strain the shared file system. This is especially true if running large swarms where each subjob runs multiple short python proceses. If you have very short running python processes, please modify the code such that each script does more work.

Packages installed in the home directory

Python includes ~/.local/lib/pythonX.X/site-packages in the package search path. If you have packages installed there they can override the centrally installed packages. This can lead to hard to diagnose problems. If you have problems that are resolved by adding the -s option a broken package installed in your home directory may be the cause.

Swarm and Python environments

If activating one of your own environment within a swarm some versions of conda may encounter a race condition that will lead to some swarm subjobs failing to activate the environment correctly. This can be avoided by either activating the conda environment before submitting the swarm (the environment gets exported to the job) or by calling python with the full path to your environment without activating the environment.

matplotlib in non-interactive environments

By default matplotlib uses an interactive backend. This works if the backend can connect to a graphical user interface (e.g. if using X11 forwarding through an ssh tunnel or NX). However, if there is no GUI to connect to, this will fail with an error message that includes.

Could not connect to any X display

This can be solved by using a non-interactive backend for plotting. Which backend matplotlib uses can be changed in a couple of different ways:

matplotlib settings can be modified using a matplotlibrc file. This file can be placed either in the current working directory or in ~/.config/matplotlib. If placed in the former it will only affect jobs run from this directory. If placed in the latter, it will affect all your calls to matplotlib. For example, if you'd like to change the default backend, create the file ~/.config/matplotlib/matplotlibrc with the following content:

backend: agg

Alternatively, you can set the MPLBACKEND environment variable either ad hoc or in your .bashrc:

export MPLBACKEND=agg

And finally, it can also be set programatically:

>>> import matplotlib
>>> matplotlib.use("agg")

Available backends and the currently active backends can be queried interactively in python:

>>> import matplotlib
>>> matplotlib.rcsetup.interactive_bk
['GTK', 'GTKAgg', 'GTKCairo', 'GTK3Agg', 'GTK3Cairo', 'nbAgg', 'Qt4Agg', 'Qt4Cairo', 'Qt5Agg', 'Qt5Cairo', 'TkAgg', 'TkCairo', 'WebAgg', 'WX', 'WXAgg', 'WXCairo']
>>> matplotlib.rcsetup.non_interactive_bk
['agg', 'cairo', 'gdk', 'pdf', 'pgf', 'ps', 'svg', 'template']
>>> import matplotlib.pyplot as plt
>>> plt.get_backend()
'agg'

python environments and NoMachine problems

If you add the executables of the conda dbus package to your PATH in one of your startup files (e.g. ~/.bashrc), NoMachine may fail with a black screen. This can happen when you load a python module or activate your own conda installation. The solution is to remove any unnecessary initialization from your .bashrc or, if this is an environment under your control, remove the dbus package from the environment.

The general purpose Python environments are made available through the module system:

[user@cn3444 ~]$ module avail python
--------------- /usr/local/lmod/modulefiles ---------------
   python/2.7 (D)    python/3.5    python/3.6    python/3.7 

  Where:
   D:  Default Module

[user@cn3444 ~]$ module load python/3.7
[+] Loading python 3.7  ...

[user@cn3444 ~]$ python
Python 3.7.5 (default, Oct 25 2019, 15:51:11)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> 

These are fully featured environments with many installed packages including the usual modules that make up the scientific Python stack (see below). These environments best suited for development and for running code that is not dependent on specific versions of the installed packages. Packages in these environments are updated regularly. If you need more stability to ensure reproducibility or because your code depends on the presence of certain fixed versions of packages you can also build your own environments.

Python scientific stack

The usual scientific python stack (numpy, scipy, scikit-learn, numba, pandas, ...) is included in all the general purpose environments. Numpy and scipy are compiled against the accelerated Intel MKL library.

[user@cn3444 ~]$ python
Python 3.7.5 (default, Oct 25 2019, 15:51:11)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.__config__.show()
blas_mkl_info:
    libraries = ['blas', 'cblas', 'lapack', 'pthread', 'blas', 'cblas', 'lapack']
    library_dirs = ['/usr/local/Anaconda/envs/py3.7/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/usr/local/Anaconda/envs/py3.7/include']
[...snip...]

Because they are compiled against MKL, some mathematical operations (e.g. SVD) can make use of multithreading to accelerate computation. The number of threads such operations will create is determined by the environment variable OMP_NUM_THREADS. To avoid accidentally overloading jobs, especially when also doing multiprocessing, this variable is set to 1 by the python modules. If your code can take advantage of implicit parallelism you can set this variable to match the number of allocated CPUs for cluster jobs or adjust it such that the product of OMP_NUM_THREADS * number of processes equals the number of allocated CPUS. For example, if allocating 16 CPUs and planning on using multiprocessing with 4 workers, OMP_NUM_THREADS can be set as high as 4 without overloading the allocation.

The following packages are available in the general purpose python environments

Some common issues with using multiprocessing on biowulf were already pointed out in the common pitfalls section. Beyond those multiprocessing can be made more robust by setting workers up to ignore the SIGINT signal so that a multiprocessing script can be terminated cleanly with scancel or Ctrl-C. The following script also makes sure to correctly detect the number of available CPUs in batch jobs:

#! /usr/bin/env python

from multiprocessing import Pool
import signal
import os

def init_worker():
    """
    This is necessary to be able to interrupt the
    script with CTRL-C (or scancel for that matter).
    It makes sure that the workers ignore SIGINT so
    that any SIGINT sent goes to the master process
    """
    signal.signal(signal.SIGINT, signal.SIG_IGN)

def worker(i):
    return i*i

if __name__ == '__main__':
    # the the number of allocated cpus (or 2 if not run as a slurm job)
    nproc = int(os.environ.get("SLURM_CPUS_PER_TASK", "2"))
    print("Running on %d CPUs" % nproc)
    # set up 50 tasks
    tasks = range(0, 100)
    p = Pool(nproc, init_worker)
    try:
        # run the processing pool
        results = p.map(worker, tasks)
    except (KeyboardInterrupt, SystemExit):
        p.terminate()
        p.join()
        sys.exit(1)
    else:
        p.close()
        p.join()
        # result summary
        print("\n".join("%d * %d = %d" % (a, a, b) for a, b in zip(tasks, results)))

It is also important to benchmark scaling of your code. Many algorithms won't scale well beyond a certain number of parallel multiprocessing workers which can lead to very inefficient resource usage (e.g. allocating 56 CPUs when parallel efficiency drops below 50% at 24 CPUs).

The python MPI library mpi4py is available in the python modules. In order to use this package it is necessary to load an additional module (mpi4py) to ensure that it can find the correct OpenMPI libraries. Here is an example for how to use this library:

The python MPI script:

#! /usr/bin/env python

from mpi4py import MPI

comm = MPI.COMM_WORLD

rank = comm.Get_rank()
size = comm.Get_size()
print("Hello! I'm rank {0} / {1} running in total...".format(rank, size))

comm.Barrier()

The batch script:

#!/bin/bash
#SBATCH --ntasks=8
#SBATCH --ntasks-per-core=1
#SBATCH --partition=multinode

module load mpi4py
module load python/3.7
#mpiexec  ./test.py
srun --mpi=pmix ./test.py

Submit with

[user@biowulf ~]$ sbatch test.sh

In order to use the rpy2 package on biowulf it is necessary to load a separate rpy2 module which allows the package to find the correct R installation.

[user@cn3444 ~]$ module load python/3.7 rpy2
Python 3.7.5 (default, Oct 25 2019, 15:51:11)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import rpy2                                                       
>>> import rpy2.robjects as robjects                                  
>>> pi = robjects.r['pi']                                             
>>> pi[0]                                                             
3.141592653589793                                                     

If you need common python packages at fixed versions or python packages that may not be suitable for installation in the general purpose python environments you can create your own environments using conda. This involves installing miniconda into your data directory (or a shared data directory for collaborations) and creating environments within that installation. This is a fairly quick and straight forward process. In the following example we will cover some basics of using conda to create private environments. Much more detailed documentation as available on the conda documentation pages.

Private python environments should be created in data directories, not the home directory since a conda installation can easily grow in space to exceed the home directory allocation.

[user@biowulf ~]$ cd /data/$USER

Download the miniconda installer and install to /data/$USER/conda. Don't install to your home directory since the quota there limits space.

[user@biowulf temp]$ wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
--2018-04-23 15:04:10--  https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
Resolving dtn04-e0... 10.1.200.240
Connecting to dtn04-e0|10.1.200.240|:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 58304693 (56M) [application/x-sh]
Saving to: “Miniconda3-latest-Linux-x86_64.sh”
2018-04-23 15:04:11 (118 MB/s) - “Miniconda3-latest-Linux-x86_64.sh” saved [58304693/58304693]

[user@biowulf temp]$ mkdir -p /scratch/$USER/temp
[user@biowulf temp]$ TMPDIR=/scratch/$USER/temp bash Miniconda3-latest-Linux-x86_64.sh -p /data/$USER/conda -b
PREFIX=/data/$USER/conda
Unpacking payload ...
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##
[...snip...]
installation finished.
[user@biowulf temp]$ rm Miniconda3-latest-Linux-x86_64.sh

To use the newly installed conda you will have to source an init file. Do this each time you are going to work with your environments. We then activate the base environment (previously called the root environment) and update the conda package manager wich itself is just a package:

[user@biowulf temp]$ source /data/$USER/conda/etc/profile.d/conda.sh
[user@biowulf temp]$ conda activate base
(base) [user@biowulf temp]$ which python
/data/$USER/conda/bin/python
(base) [user@biowulf temp]$ conda update conda
Solving environment: done

## Package Plan ##
[...snip...]

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(base) [user@biowulf temp]$ conda clean --all --yes
Cache location: /data/$USER/conda/pkgs
Will remove the following tarballs:

/data/$USER/conda/pkgs
------------------------
python-3.6.4-hc3d631a_1.tar.bz2             29.1 MB
[...snip...]
conda-4.5.1-py36_0.tar.bz2                   1.0 MB

---------------------------------------------------
Total:                                      59.4 MB

Removed python-3.6.4-hc3d631a_1.tar.bz2
[...snip...]

Now create a new environment called project1 with an older version of pysam from the bioconda channel and python 2.7:

(base) [user@biowulf temp]$ conda create -n project1 python=3.7 numpy scipy bioconda::pysam==0.15.3

Solving environment: done

## Package Plan ##
[...snip...]

Proceed ([y]/n)? y
[...snip...]
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate project1
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(base) [user@biowulf temp]$ conda activate project1
(project1) [user@biowulf temp]$ which python
/data/$USER/conda/envs/project1/bin/python
(project1) [user@biowulf temp]$ samtools --version
samtools 1.9
Using htslib 1.9
Copyright (C) 2018 Genome Research Ltd.
(project1) [user@biowulf temp]$ conda deactivate

Now an environment for a different project with current pysam plus some other tools. This time we add the bioconda channel to the channels for the environment so we don't have to use the bioconda:: prefix. A common pattern for environments used for bioinformatic software is to set up bioconda and conda-forge channels in the following order on a per-environment basis. This allows conda-forge packages to override packages from the defaults channel.

(base) [user@biowulf temp]$ conda create -n project2 python=3.6
[...snip...]
(base) [user@biowulf temp]$ conda activate project2
(project2) [user@biowulf temp]$ conda config --env --add channels defaults
(project2) [user@biowulf temp]$ conda config --env --add channels conda-forge
(project2) [user@biowulf temp]$ conda config --env --add channels bioconda
(project2) [user@biowulf temp]$ conda config --show-sources
==> /data/$USER/conda/envs/project2/.condarc <==
channels:
  - bioconda
  - conda-forge
  - defaults

(project2) [user@biowulf temp]$ conda install pysam bedtools hisat2
[...snip...]
(project2) [user@biowulf temp]$ which pip
/data/$USER/conda/envs/project2/bin/pip
(project2) [user@biowulf temp]$ pip install tensorflow-gpu
[...snip...]

Note that, as show above, pip can be used to install packages into conda environments as well.

List environments in the current conda install, then deactivate environments

(project2) [user@biowulf temp]$ conda info --env
# conda environments:
#
base                     /data/$USER/conda
project1                 /data/$USER/conda/envs/project1
project2              *  /data/$USER/conda/envs/project2
(project2) [user@biowulf temp]$ conda deactivate
(base) [user@biowulf temp]$ conda deactivate
[user@biowulf temp]$ 

• Python home
• Python 3.X documentation
• Python 2.X documentation
• Python 2 or 3?
• Transitioning to Python 3
• Porting Python 2 Code to Python 3
• Conda documentation