High-Performance Computing at the NIH
GitHub YouTube @nih_hpc RSS Feed
Singularity

      


Extreme Mobility of Compute

Singularity containers let users run applications in a Linux environment of their choosing.

Possible uses for Singularity on Biowulf:

Web sites

References
Example definition files written by the NIH HPC staff

Please Note:
Singularity gives you the ability to install and run applications in your own Linux environment with your own customized software stack. With this ability comes the added responsibility of managing your own Linux environment. While the NIH HPC staff can provide guidance on how to create and use singularity containers, we do not have the resources to manage containers for individual users. If you decide to use Singularity, it is your responsibility to build and manage your own containers.

Creating Singularity containers
back to top

To use Singularity on Biowulf, you either need to create your own Singularity container, or use one created by someone else. To build a Singularity container, you need root access to the build system. Thus, you cannot build a Singularity container on Helix or Biowulf. Possible options are:

Depending on your environment and the type of Singularity container you want to build, you may need to install some dependencies before installing and/or using Singularity. For instance, the following may need to be installed on Ubuntu for Singularity to build and run properly. (user input in bold)

[user@someUbuntu ~]$ sudo apt-get update && sudo apt-get install build-essential debootstrap yum
On Centos, these commands will provide some needed dependencies for Singularity:
[user@someCentos ~]$ sudo yum update
[user@someCentos ~]$ sudo yum groupinstall 'Development Tools'
[user@someCentos ~]$ sudo yum install wget epel-release
[user@someCentos ~]$ sudo yum install debootstrap.noarch

You can find more information about installing Singularity on your Linux build system here.

In addition to your own Linux environment, you will also need a definition file to build a Singularity container from scratch. You can find some simple definition files for a variety of Linux distributions in the /example directory of the source code. You can also find a small list of definition files containing popular applications at the top of this page. Detailed documentation about building Singularity container images is available at the Singularity website.

Binding external directories
back to top

Binding a directory to your Singularity container allows you to access files in a host system directory from within your container. By default, Singularity will bind your $HOME directory (along with a few other directories such as /tmp and /dev). You can also bind other directories into your Singularity container yourself. To do this, you must do two things:

Let's say you want to bind /lscratch into a singularity container. You can either create bind points in your singularity container during the build procedure within your .def file, or you can use the pre-existing empty directory /mnt as your mount point.

For instance, if you decided to create the bind point in your .def file it would look something like this:

BootStrap: debootstrap
OSVersion: xenial
MirrorURL: http://us.archive.ubuntu.com/ubuntu/

%post
    # create mount points for NIH HPC environment
    mkdir /scratch

Or, if you you would like to make your container more generic, you can bind the system's /scratch/$USER directory to /mnt (despite them having different names) as follows:

[user@cn1234 ~]$ singularity shell --bind /scratch/$USER:/mnt my_container.simg

The --bind option also works with the run and exec Singularity commands. And you can bind multiple directories in a single command with this syntax:

[user@cn1234 ~]$ singularity shell --bind /us:/them,/me:/you,/black:/blue my_container.simg 
Finally, you can use the environment variable $SINGULARITY_BINDPATH to bind host directories to container directories:
[user@cn1234 ~]$ export SINGULARITY_BINDPATH="/scratch,/fdb:/myfdb"
This means "bind the host /scratch to /scratch in my container, and bind the host /fdb to the directory called /myfdb in my containter". Using environment variables, you can bind directories even when you are running your container as an executable file with a runscript. If you bind many directories into your singularity container and they don't change, you could even put this variable in your .bashrc file.

This process is further documented at the LBL Singularity website.

Binding NIH HPC Directories in a Portable Way

Newer systems can take advantage of Singularity's support for the overlay filesystem, which allows binding directories without having to manually manage creation of mount points. Unfortunately, Biowulf and other systems do not support this, so we cannot rely on this feature. However, rather than mirroring the NIH HPC directory tree in your container (by making mount points /fdb, /lscratch, and so on), we suggest creating a generic set of mount points in your container that can be used analogously on other systems.

With the following definition file, we create two additional mount points to be used alongside /mnt, which we reserve for some auxiliary use (like /scratch, /lscratch, or an additional data directory).

BootStrap: debootstrap
OSVersion: xenial
MirrorURL: http://us.archive.ubuntu.com/ubuntu/

%post
    # create generic mount points
    mkdir \
    /data      `# for your personal or group data directory` \
    /resources `# for reference data-- corresponds to /fdb on NIH HPC`
Now, after we have built a container using the above definition file, we can bind our directories as follows, assuming we have allocated local scratch space, which would make the /lscratch/$SLURM_JOB_ID directory available:
[user@cn1234 ~]$ singularity shell --bind /data/$USER:/data,/fdb:/resources,/lscratch/$SLURM_JOB_ID:/tmp my-container.simg
or, using the environment variable:
[user@cn1234 ~]$ export SINGULARITY_BINDPATH="/data/$USER:/data,/fdb:/resources,/lscratch/$SLURM_JOB_ID:/tmp"
[user@cn1234 ~]$ singularity shell my-container.simg
If you would like to store this in ~/.bashrc, you should make sure to test for the existence of the lscratch subdirectory before adding it to the bind path:
SINGULARITY_BINDPATH=/data/$USER:/data,/fdb:/resources
if [ -n "$SLURM_JOB_ID" ] && [ -d "/lscratch/$SLURM_JOB_ID" ]
then
    SINGULARITY_BINDPATH="${SINGULARITY_BINDPATH},/lscratch/$SLURM_JOB_ID:/tmp"
fi
export SINGULARITY_BINDPATH
When you share this container, your colleagues here and elsewhere can bind their own corresponding directories to these same mountpoints.

Interactive Singularity containers
back to top

Singularity cannot be run on Helix or the Biowulf login node.

To run a Singularity container image on Biowulf interactively, you need to allocate an interactive session, and load the Singularity module. In this sample session (user input in bold), an Ubuntu 16.04 Singularity container is downloaded and run from Docker Hub. If you want to run a local Singularity container instead of downloading one, just replace the DockerHub URL with the path to your container image file.

[user@biowulf ~]$ sinteractive --cpus-per-task=4 --mem=10g
salloc.exe: Pending job allocation 43131269
salloc.exe: job 43131269 queued and waiting for resources
salloc.exe: job 43131269 has been allocated resources
salloc.exe: Granted job allocation 43131269
salloc.exe: Waiting for resource configuration
salloc.exe: Nodes cn0123 are ready for job
srun: error: x11: no local DISPLAY defined, skipping

[user@cn0123 ~]$ module load singularity
[+] Loading singularity 2.4 on cn3160

[user@cn0123 ~]$ singularity shell docker://ubuntu
Docker image path: index.docker.io/library/ubuntu:latest
Cache folder set to /spin1/home/linux/godlovedc/.singularity/docker
Creating container runtime...
WARNING: Bind file source does not exist on host: /etc/resolv.conf
Singularity: Invoking an interactive shell within container...

Singularity ubuntu:~> cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

Singularity ubuntu:~> exit

[user@cn0123 ~]$ exit
exit

salloc.exe: Relinquishing job allocation 23562157
[user@biowulf ~]$
Note that you need to exit your Singularity container as well as your allocated interactive Slurm session when you are done.

Expand the tab below to view a demo of interactive Singularity usage.

Singularity interactive container demo
  • space - play / pause
  • f - toggle fullscreen mode
  • arrow keys(←/→) - rewind 5 seconds / fast-forward 5 seconds
  • 0, 1, 2 ... 9 - jump to 0%, 10%, 20% ... 90%
  • copy and paste text from movie

Singularity containers in batch
back to top

In this example, singularity will be used to run a TensorFlow example in an Ubuntu 16.04 container. (User input in bold).

First, create a container image on a machine where you have root privileges. These commands were run on a Google Cloud VM instance running an Ubuntu 16.04 image, and the Singularity container was created using this definition file that includes a TensorFlow installation.

[user@someCloud ~]$ sudo singularity build ubuntu_w_TFlow.simg ubuntu_w_TFlow.def

Next, copy the TensorFlow script that you want to run into your home directory, or another directory that will be visible from within the container at runtime. (See Binding external directories' above). In this case, this example script from the TensorFlow website was copied to /home/$USER, and the container was moved to the user's data directory

[user@someCloud ~]$ scp TFlow_example.py user@biowulf.nih.gov: 

[user@someCloud ~]$ scp ubuntu_w_Tflow.simg user@biowulf.nih.gov:/data/user 

Then ssh to Biowulf and write a batch script to run the singularity command similar to this:

#!/bin/sh
# file called myjob.batch
set -e
module load singularity
cd /data/user
singularity exec ubuntu_w_TFlow.simg python ~/TFlow_example.py

Submit the job like so:

[user@biowulf ~]$ sbatch myjob.batch

After the job finishes executing you should see the following output in the slurm*.out file.

[+] Loading singularity 2.4 on cn2725
(0, array([-0.39398459], dtype=float32), array([ 0.78525567], dtype=float32))
(20, array([-0.05549375], dtype=float32), array([ 0.38339305], dtype=float32))
(40, array([ 0.05872268], dtype=float32), array([ 0.3221375], dtype=float32))
(60, array([ 0.08904253], dtype=float32), array([ 0.30587664], dtype=float32))
(80, array([ 0.09709124], dtype=float32), array([ 0.30156001], dtype=float32))
(100, array([ 0.09922785], dtype=float32), array([ 0.30041414], dtype=float32))
(120, array([ 0.09979502], dtype=float32), array([ 0.30010995], dtype=float32))
(140, array([ 0.09994559], dtype=float32), array([ 0.30002919], dtype=float32))
(160, array([ 0.09998555], dtype=float32), array([ 0.30000776], dtype=float32))
(180, array([ 0.09999616], dtype=float32), array([ 0.30000207], dtype=float32))
(200, array([ 0.09999899], dtype=float32), array([ 0.30000055], dtype=float32))

Expand the tab below to watch a quick demo of Singularity in batch mode.

Singularity containers in batch demo
  • space - play / pause
  • f - toggle fullscreen mode
  • arrow keys(←/→) - rewind 5 seconds / fast-forward 5 seconds
  • 0, 1, 2 ... 9 - jump to 0%, 10%, 20% ... 90%
  • copy and paste text from movie

Singularity containers on GPU nodes
back to top

With the release of Singularity v2.3 it is no longer necessary to install NVIDIA drivers into your Singularity container to access the GPU on a host node. If you still want the deprecated gpu4singularity script that was used to install NVIDIA drivers within containers for use on our GPU nodes you can find it on GitHub.

Now, you can simply use the --nv option to grant your containers GPU support at runtime. Consider the following example in which we will download some TensorFlow models to the user's home directory and then run the latest TensorFlow container from DockerHub to train a model on the MNIST handwritten digit data set using a GPU node.

[user@biowulf ~]$ git clone https://github.com/tensorflow/models.git
Initialized empty Git repository in /home/user/models/.git/
remote: Counting objects: 4971, done.
remote: Compressing objects: 100% (26/26), done.
remote: Total 4971 (delta 14), reused 11 (delta 2), pack-reused 4943
Receiving objects: 100% (4971/4971), 153.50 MiB | 12.21 MiB/s, done.
Resolving deltas: 100% (2540/2540), done.

[user@biowulf ~]$ sinteractive --constraint=gpuk80 --gres=gpu:k80:1
salloc.exe: Pending job allocation 39836528
salloc.exe: job 39836528 queued and waiting for resources
salloc.exe: job 39836528 has been allocated resources
salloc.exe: Granted job allocation 39836528
salloc.exe: Waiting for resource configuration
salloc.exe: Nodes cn4178 are ready for job
srun: error: x11: no local DISPLAY defined, skipping

[user@cn4178 ~]$ module load singularity
[+] Loading singularity 2.4 on cn4178

[user@cn4178 ~]$ singularity exec --nv docker://tensorflow/tensorflow:latest-gpu \
                        python ~/models/tutorials/image/mnist/convolutional.py
Docker image path: index.docker.io/tensorflow/tensorflow:latest-gpu
Cache folder set to /spin1/home/linux/godlovedc/.singularity/docker
[19/19] |===================================| 100.0%
Creating container runtime...
WARNING: Bind file source does not exist on host: /etc/resolv.conf
Extracting data/train-images-idx3-ubyte.gz
Extracting data/train-labels-idx1-ubyte.gz
Extracting data/t10k-images-idx3-ubyte.gz
Extracting data/t10k-labels-idx1-ubyte.gz
2017-06-14 18:54:40.157855: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-14 18:54:40.157887: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-14 18:54:40.157896: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-06-14 18:54:40.157903: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-14 18:54:40.157911: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-06-14 18:54:40.737822: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 0 with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:84:00.0
Total memory: 11.92GiB
Free memory: 11.86GiB
2017-06-14 18:54:40.737858: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0
2017-06-14 18:54:40.737867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0:   Y
2017-06-14 18:54:40.737881: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K80, pci bus id: 0000:84:00.0)
Initialized!
Step 0 (epoch 0.00), 17.1 ms
Minibatch loss: 8.334, learning rate: 0.010000
Minibatch error: 85.9%
Validation error: 84.6%
Step 100 (epoch 0.12), 13.4 ms
Minibatch loss: 3.254, learning rate: 0.010000
Minibatch error: 3.1%
Validation error: 7.8%
Step 200 (epoch 0.23), 11.6 ms
Minibatch loss: 3.354, learning rate: 0.010000
Minibatch error: 10.9%
Validation error: 4.5%
Step 300 (epoch 0.35), 11.5 ms
[...snip...]

Expand the tab below to see a demo of installing and using GPU support in a Singularity container.

Using the GPU demo
  • space - play / pause
  • f - toggle fullscreen mode
  • arrow keys(←/→) - rewind 5 seconds / fast-forward 5 seconds
  • 0, 1, 2 ... 9 - jump to 0%, 10%, 20% ... 90%
  • copy and paste text from movie

Using Docker containers with Singularity
back to top

Singularity can import, bootstrap, and even run Docker images directly from Docker Hub. For instance, the following commands will start an Ubuntu container running on a compute node with no need for a definition file or container image!

[user@cn0123 ~]$ module load singularity
[+] Loading singularity 2.4 on cn0123

[user@cn0123 ~]$ singularity shell docker://ubuntu:latest
Docker image path: index.docker.io/library/ubuntu:latest
Cache folder set to /spin1/home/linux/godlovedc/.singularity/docker
[5/5] |===================================| 100.0%
Creating container runtime...
WARNING: Bind file source does not exist on host: /etc/resolv.conf
Singularity: Invoking an interactive shell within container...

Singularity.ubuntu:latest> cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
In this instance the container is ephemeral. It will disappear as soon as you exit the shell. If you wanted to actually download the container from Docker Hub, you could use the pull command like so:
[user@cn0123 ~]$ singularity pull docker://ubuntu:latest
WARNING: pull for Docker Hub is not guaranteed to produce the
WARNING: same image on repeated pull. Use Singularity Registry
WARNING: (shub://) to pull exactly equivalent images.
Docker image path: index.docker.io/library/ubuntu:latest
Cache folder set to /spin1/home/linux/user/.singularity/docker
Importing: base Singularity environment
Importing: /spin1/home/linux/user/.singularity/docker/sha256:ae79f251470513c2a0ec750117a81f2d58a50727901ca416efecf297b8a03913.tar.gz
Importing: /spin1/home/linux/user/.singularity/docker/sha256:c59d01a7e4caf1aba785eb33192fec3f96e4ab01975962bcec10f4989a6edcc6.tar.gz
Importing: /spin1/home/linux/user/.singularity/docker/sha256:41ba73a9054d231e1f555c40a74762276900cc6487f5c6cf13b89c7606635c67.tar.gz
Importing: /spin1/home/linux/user/.singularity/docker/sha256:f1bbfd495cc1112b503350686641ee4fa2cea8ccd13fb8a8a302c81dae61d418.tar.gz
Importing: /spin1/home/linux/user/.singularity/docker/sha256:0c346f7223e24b517358f52c4a3f5f9af1c86e5ddeaee5659c8889846e46d1e2.tar.gz
Importing: /spin1/home/linux/user/.singularity/metadata/sha256:f6be9f4f6905406c1e7fd6031ee3104d25ad6a31d10d5e9192e7abf7a21e519a.tar.gz
WARNING: Building container as an unprivileged user. If you run this container as root
WARNING: it may be missing some functionality.
Building Singularity image...
Singularity container built: ./ubuntu-latest.img
Cleaning up...

[user@cn0123 ~]$ singularity shell ubuntu.img
WARNING: Bind file source does not exist on host: /etc/resolv.conf
Singularity: Invoking an interactive shell within container...

Singularity ubuntu.img:~> cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
This feature gives you instant access to 100,000+ pre-built container images. You can even use a Docker Hub container as a starting point in a definition file.

In this example, we will create a Singularity container image starting from the official continuumio miniconda container on Docker Hub. Then we'll install a number of RNASeq tools. This would allow us to write a pipeline with, for example, Snakemake and distribute it along with the image to create an easily shared, reproducible workflow. This definition file also installs a runscript enabling us to treat our container like an executable.

BootStrap: docker
From: continuumio/miniconda:latest
IncludeCmd: yes

%post
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# this will install all necessary packages and prepare the container
    apt-get -y update
    apt-get -y install make gcc zlib1g-dev libncurses5-dev
    wget https://github.com/samtools/samtools/releases/download/1.3.1/samtools-1.3.1.tar.bz2 \
        && tar -xjf samtools-1.3.1.tar.bz2 \
        && cd samtools-1.3.1 \
        && make \
        && make prefix=/usr/local install
    export PATH=/opt/conda/bin:$PATH
    conda install --yes -c bioconda \
        star=2.5.2b \
        sailfish=0.10.1 \
        fastqc=0.11.5 \
        kallisto=0.43.0 \
        subread=1.5.0.post3
    conda clean --index-cache --tarballs --packages --yes
    mkdir /data /resources

%runscript
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# this text code will run whenever the container
# is called as an executable or with `singularity run`
function usage() {
    cat <<EOF
NAME
    rnaseq - rnaseq pipeline tools 0.1
SYNOPSIS
    rnaseq tool [tool options]
    rnaseq list
    rnaseq help
DESCRIPTION
    Singularity container with tools to build rnaseq pipeline. 
EOF
}

function tools() {
    echo "conda: $(which conda)"
    echo "---------------------------------------------------------------"
    conda list
    echo "---------------------------------------------------------------"
    echo "samtools: $(samtools --version | head -n1)"
}

arg="${1:-none}"

case "$arg" in
    none) usage; exit 1;;
    help) usage; exit 0;;
    list) tools; exit 0;;
    # just try to execute it then
    *)    $@;;
esac

%environment
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# This sets global environment variables for anything run within the container
export PATH="/opt/conda/bin:/usr/local/bin:/usr/bin:/bin:"
unset CONDA_DEFAULT_ENV
export ANACONDA_HOME=/opt/conda

Assuming this file is called rnaseq.def, we can create a Singularity container called rnaseq on our build system with the following commands:

[user@some_build_system ~]$ sudo singularity build rnaseq rnaseq.def

This image contains miniconda and our rnaseq tools and can be called directly as an executable like so:

[user@some_build_system ~]$ ./rnaseq help
NAME
    rnaseq - rnaseq pipeline tools 0.1
SYNOPSIS
    rnaseq snakemake [snakemake options]
    rnaseq list
    rnaseq help
DESCRIPTION
    Singularity container with tools to build rnaseq pipeline. 

[user@some_build_system ~]$ ./rnaseq list
conda: /opt/conda/bin/conda
---------------------------------------------------------------
# packages in environment at /opt/conda:
#
fastqc                    0.11.5                        1    bioconda
java-jdk                  8.0.92                        1    bioconda
kallisto                  0.43.0                        1    bioconda
sailfish                  0.10.1              boost1.60_1    bioconda
[...snip...]

[user@some_build_system ~]$ ./rnaseq samtools --version
samtools 1.3.1
Using htslib 1.3.1
Copyright (C) 2016 Genome Research Ltd.

After copying the image to the NIH HPC systems, allocate an sinteractive session and test it there

[user@cn1234 ~]$ module load singularity
[user@cn1234 ~]$ ./rnaseq list
conda: /opt/conda/bin/conda
---------------------------------------------------------------
# packages in environment at /opt/conda:
#
fastqc                    0.11.5                        1    bioconda
java-jdk                  8.0.92                        1    bioconda
kallisto                  0.43.0                        1    bioconda
sailfish                  0.10.1              boost1.60_1    bioconda
[...snip...]

This could be used with a Snakemake file like this

rule fastqc:
    input: "{sample}.fq.gz"
    output: "{sample}.fastqc.html"
    shell: 
        """
        module load singularity
        ./rnaseq fastqc ... {input}
        """

rule align:
    input: "{sample}.fq.gz"
    output: "{sample}.bam"
    shell: 
        """
        module load singularity
        ./rnaseq STAR ....
        """

Expand the tab below to see an example of creating a Singularity container to be used as an executable from a Docker image on DockerHub.

Singularity with Docker demo
  • space - play / pause
  • f - toggle fullscreen mode
  • arrow keys(←/→) - rewind 5 seconds / fast-forward 5 seconds
  • 0, 1, 2 ... 9 - jump to 0%, 10%, 20% ... 90%
  • copy and paste text from movie

Documentation
back to top