Quick Links Important Notes Creating Singularity containers Binding external directories Singularity as an Installation Medium Interactive Singularity containers Singularity containers in batch Singularity containers on GPU nodes Using Docker containers with Singularity Troubleshooting containers that hang

Singularity containers let users run applications in a Linux environment of their choosing.

Possible uses for Singularity on Biowulf:

• Run an application that was built for a different distribution of Linux than the host OS.
• Reproduce an environment to run a workflow created by someone else.
• Run a series of applications (a 'pipeline') that includes applications built on different platforms.
• Run an application from Docker Hub on Biowulf without actually installing anything.

• Singularity home
• Singularity Documentation
• Singularity on GitHub
• Docker Hub
• Singularity Container Services

• Class taught by NIH HPC staff (2020):
  • Materials
  • Day 1 recording
  • Day 2 recording
• Singularity Basics videos by Sylabs
• NIH HPC Singularity Tutorial

These definition files can all be found on GitHub:

• DIGITS (Using an NVIDIA Docker Hub image with drivers for NIH HPC GPU nodes)
• Keras/tensorflow
• RStudio
• Theano (GPU/CPU support but requires a GPU to be present either way, running under Ubuntu 16.04)

Additionally, a large number of staff maintained definition files and associated helper scripts can be found at this GitHub repo. These are files that staff members use to install containerized apps on the NIH HPC systems.

• Module Name: singularity (see the modules page for more information). Do NOT load a specific version of singularity because we generally only keep one version on the system for security reasons
• Singularity caches containers in your home directory by default. This can quickly fill your home directory, so the NIH HPC staff suggests you configure Singularity to cache containers in your data directory by executing the following or similar command. (If you use Singularity a lot, you might want to write this command into your ~/.bashrc file.)
  

  export SINGULARITY_CACHEDIR=/data/${USER}/.singularity

• You are encouraged to source the file /usr/local/current/singularity/app_conf/sing_binds to bind all appropriate host system directories in your container at runtime.
• Bind-mounting the /scratch directory will result in an error because access to /scratch (different from /lscratch) is disabled from compute nodes (since September 2021).
• Singularity gives you the ability to install and run applications in your own Linux environment with your own customized software stack. With this ability comes the added responsibility of managing your own Linux environment. While the NIH HPC staff can provide guidance on how to create and use singularity containers, we do not have the resources to manage containers for individual users. If you decide to use Singularity, it is your responsibility to build and manage your own containers.

To use Singularity on Biowulf, you either need to create your own Singularity container, or use one created by someone else. You have several options to build Singularity containers:

• Singularity allows building some containers using proot which is available with the Singularity module on Biowulf (as of Septmeber 2025). Only certain definitions files work. Make sure to read the documentation before building containers on Biowulf. There is nothing special you have to do to enable this; singularity will automatically find proot and build with it. See example below.
• You can build small and medium sized containers from Helix or Biowulf compute nodes using the --remote option. In a nutshell, you must log in and generate a token on Singularity Container Services and use the command line to copy that token into your environment (using the command singularity remote login SylabsCloud). Once you've done that, the --remote option will allow you to build containers using the Sylabs remote builder.
• If you have a Linux system to which you have root (admin) access, you can install Singularity and build your Singularity container there.
• If you have a very recent Linux system (like Ubuntu >=18) you can build Singularity containers without root access using the --fakeroot option.
• If you don't have a Linux system you could install one in a virtual machine using software like VirtualBox, Vagrant, VMware, or Parallels. (If you use a virtual machine be sure to allocate at least 2GB of memory or some of your builds may fail with out of memory errors.)
• You can allocate a cloud instance, to which you will have root access. Install Singularity and build your Singularity container there.

You can find information about installing Singularity on Linux here.

In addition to your own Linux environment, you will also need a definition file to build a Singularity container from scratch. You can find some simple definition files for a variety of Linux distributions in the /example directory of the source code. You can also find a small list of definition files containing popular applications at the top of this page. Detailed documentation about building Singularity container images is available at the Singularity website.

Binding a directory to your Singularity container allows you to access files in a host system directory from within your container. By default, Singularity will bind your $HOME directory (along with a few other directories such as /tmp and /dev). You can also bind other directories into your Singularity container yourself. The process is described in detail in the Singularity documentation.

While $HOME is bind-mounted to the container by default, there are several filesystems on the NIH HPC systems that you may also want to include. Furthermore, if you are running a job and have allocated local scratch space, you might like to bind mount your /lscratch directory to /tmp in the container.

The following command opens a shell in a container while bind-mounting your data directory, /fdb, and /lscratch into the same path inside the container If you have access to shared data directories, you'll want to add them to the list as well (for example, /data/$USER,/data/mygroup1,/data/mygroup2,/fdb,...).

[user@cn1234 ~]$ singularity shell --bind /data/$USER,/fdb,/lscratch my-container.sif

[user@cn1234 ~]$ export SINGULARITY_BINDPATH="/data/$USER,/fdb,/lscratch"
[user@cn1234 ~]$ singularity shell my-container.sif

SINGULARITY_BINDPATH="/data/$USER,/fdb"
[ -d /lscratch ] && SINGULARITY_BINDPATH="${SINGULARITY_BINDPATH},/lscratch/${SLURM_JOB_ID}:/tmp"
export SINGULARITY_BINDPATH

[user@cn1234 ~]$ . /usr/local/current/singularity/app_conf/sing_binds

One use case of Singularity is to transparently use software in a container as though it were directly installed on the host system. To accomplish this on our systems, you need to be aware of the shared filesystem locations and bind mount the corresponding directories inside the container, which is more complicated than it seems because we use symbolic links to refer to some of our network storage systems. As a result, you will need to specify some directories in addition to the ones you use directly to ensure that the symbolic link destinations are also bound into the container.

If you wanted to take advantage of a Debian package this way and use it to install software into your home directory, for example samtools and bcftools, you would use a definition file, Singularity, with these contents:

Bootstrap: docker
From: debian:9-slim

%post
    # install the desired software
    apt-get update
    apt-get install -y samtools bcftools
    apt-get clean

After finalizing the definition file, you can proceed to build the container (of course, on a system where you have sudo or root access):

sudo singularity build hts.simg Singularity

You can then set up your installation prefix (here, it's $HOME/opt/hts) as follows, making use of symbolic links and a wrapper script:

$HOME/opt
└── hts
    ├── bin
    │   ├── samtools -> ../libexec/wrap
    │   └── bcftools -> ../libexec/wrap
    └── libexec
        ├── wrap
        └── hts.simg

#!/bin/bash

. /usr/local/current/singularity/app_conf/sing_binds
selfdir="$(dirname $(readlink -f ${BASH_SOURCE[0]}))"
instdir="$(dirname ${selfdir})"
cmd="$(basename $0)"
singularity exec -B "${instdir}" "${selfdir}/hts.simg" "$cmd" "$@"

So if you have added the installation prefix $HOME/opt/hts/bin to your PATH, then calling samtools or bcftools will run those programs from within your container. And because we have arranged to bind mount all the necessary filesystems into the container, the path names you provide for input and output into the programs will be available to the container in the same way.

Singularity cannot be run on the Biowulf login node.

To run a Singularity container image on Biowulf interactively, you need to allocate an interactive session, and load the Singularity module. In this sample session (user input in bold), an Ubuntu Singularity container is downloaded and run from Docker Hub. If you want to run a local Singularity container instead of downloading one, just replace the DockerHub URL with the path to your container image file.

[user@biowulf ~]$ sinteractive --cpus-per-task=4 --mem=10g
salloc.exe: Pending job allocation 43131269
salloc.exe: job 43131269 queued and waiting for resources
salloc.exe: job 43131269 has been allocated resources
salloc.exe: Granted job allocation 43131269
salloc.exe: Waiting for resource configuration
salloc.exe: Nodes cn0123 are ready for job

[user@cn0123 ~]$ module load singularity
[+] Loading singularity 4.2.2 on cn0123

[user@cn0123 ~]$ singularity shell docker://ubuntu
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
INFO:    Fetching OCI image...
28.3MiB / 28.3MiB [=====================================] 100 % 14.6 MiB/s 0s
INFO:    Extracting OCI image...
INFO:    Inserting Singularity configuration...
INFO:    Creating SIF file...
Singularity> cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
Singularity ubuntu:~> exit

[user@cn0123 ~]$ exit
exit

salloc.exe: Relinquishing job allocation 23562157
[user@biowulf ~]$

Expand the tab below to view a demo of interactive Singularity usage.

In this example, singularity will be used to run a TensorFlow example in an Ubuntu container. (User input in bold).

First, let's build a container from a simple definition file. The build process will automatically use proot, and hence not require root privileges on Biowulf. We will start an interactive session, write the definition file and then run singularity build.

[user@biowulf ~]$ sinteractive --mem=8G
salloc: Pending job allocation 43131269
salloc: job 43131269 queued and waiting for resources
salloc: job 43131269 has been allocated resources
salloc: Granted job allocation 43131269
salloc: Waiting for resource configuration
salloc: Nodes cn0123 are ready for job

[user@cn0123 ~]$ module load singularity
[+] Loading singularity  4.2.2  on cn0123

[user@cn0123 ~]$ cd /data/$USER

[user@cn0123 user]$ cat > tf.def < EOF
BootStrap: docker
From: tensorflow/tensorflow:latest

%environment
export LC_ALL=C
export TF_CPP_MIN_LOG_LEVEL=3
EOF

[user@cn0123 user]$ singularity build ubuntu_w_TFlow.sif tf.def
INFO:    Using proot to build unprivileged. Not all builds are supported. If build fails, use --remote or --fakeroot.
INFO:    Starting build...
INFO:    Fetching OCI image...
36.5MiB / 36.5MiB [===========================================================================] 100 % 53.1 MiB/s 0s
343.7MiB / 343.7MiB [=========================================================================] 100 % 53.1 MiB/s 0s
28.2MiB / 28.2MiB [===========================================================================] 100 % 53.1 MiB/s 0s
90.2MiB / 90.2MiB [===========================================================================] 100 % 53.1 MiB/s 0s
88.6MiB / 88.6MiB [===========================================================================] 100 % 53.1 MiB/s 0s
INFO:    Extracting OCI image...
INFO:    Inserting Singularity configuration...
INFO:    Adding environment to container
INFO:    Creating SIF file...
INFO:    Build complete: ubuntu_w_TFlow.sif

Next, we write a simple tensorflow python script:

[user@cn0123 user]$ cat > TFlow_example.py < EOF
import tensorflow as tf
mnist = tf.keras.datasets.mnist

(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation='softmax')
])

model.compile(optimizer='adam',
  loss='sparse_categorical_crossentropy',
  metrics=['accuracy'])

model.fit(x_train, y_train, epochs=5)
model.evaluate(x_test, y_test)
EOF

Finally we write a batch script to run the singularity command similar to this:

[user@cn0123 user]$ cat > myjob.batch < EOF
#!/bin/sh
# file called myjob.batch
set -e
module load singularity
cd /data/$USER
. /usr/local/current/singularity/app_conf/sing_binds
singularity exec ubuntu_w_TFlow.sif python TFlow_example.py
EOF

And submit the job like so:

[user@cn0123 user]$ sbatch myjob.batch

After the job finishes executing you should see the following output in the slurm*.out file.

[+] Loading singularity 4.2.2 on cn0123
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11490434/11490434 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step
Epoch 1/5
1875/1875 ━━━━━━━━━━━━━━━━━━━━ 3s 1ms/step - accuracy: 0.9153 - loss: 0.2901
Epoch 2/5
1875/1875 ━━━━━━━━━━━━━━━━━━━━ 2s 1ms/step - accuracy: 0.9577 - loss: 0.1416
Epoch 3/5
1875/1875 ━━━━━━━━━━━━━━━━━━━━ 3s 1ms/step - accuracy: 0.9681 - loss: 0.1041
Epoch 4/5
1875/1875 ━━━━━━━━━━━━━━━━━━━━ 2s 1ms/step - accuracy: 0.9734 - loss: 0.0859
Epoch 5/5
1875/1875 ━━━━━━━━━━━━━━━━━━━━ 3s 1ms/step - accuracy: 0.9767 - loss: 0.0732
313/313 ━━━━━━━━━━━━━━━━━━━━ 0s 706us/step - accuracy: 0.9781 - loss: 0.0685

Here' another example. Expand the tab below to watch a quick demo of Singularity in batch mode.

Since Singularity v2.3 it is no longer necessary to install NVIDIA drivers into your Singularity container to access the GPU on a host node. If you still want the deprecated gpu4singularity script that was used to install NVIDIA drivers within containers for use on our GPU nodes you can find it on GitHub.

Now, you can simply use the --nv option to grant your containers GPU support at runtime. Consider the following example in which we will download some TensorFlow models to the user's home directory and then run the latest TensorFlow container from DockerHub to train a model on the MNIST handwritten digit data set using a GPU node. This example uses the k80 GPUs which are now outdated; we recommend using new GPU types like p100, v100x, or a100.

[user@biowulf ~]$ git clone https://github.com/tensorflow/models.git
Initialized empty Git repository in /home/user/models/.git/
remote: Counting objects: 4971, done.
remote: Compressing objects: 100% (26/26), done.
remote: Total 4971 (delta 14), reused 11 (delta 2), pack-reused 4943
Receiving objects: 100% (4971/4971), 153.50 MiB | 12.21 MiB/s, done.
Resolving deltas: 100% (2540/2540), done.

[user@biowulf ~]$ sinteractive --gres=gpu:k80:1
salloc.exe: Pending job allocation 39836528
salloc.exe: job 39836528 queued and waiting for resources
salloc.exe: job 39836528 has been allocated resources
salloc.exe: Granted job allocation 39836528
salloc.exe: Waiting for resource configuration
salloc.exe: Nodes cn4178 are ready for job

[user@cn4178 ~]$ module load singularity
[+] Loading singularity 2.4 on cn4178

[user@cn4178 ~]$ singularity exec --nv docker://tensorflow/tensorflow:latest-gpu \
                        python ~/models/tutorials/image/mnist/convolutional.py
Docker image path: index.docker.io/tensorflow/tensorflow:latest-gpu
Cache folder set to /spin1/home/linux/user/.singularity/docker
[19/19] |===================================| 100.0%
Creating container runtime...
WARNING: Bind file source does not exist on host: /etc/resolv.conf
Extracting data/train-images-idx3-ubyte.gz
Extracting data/train-labels-idx1-ubyte.gz
Extracting data/t10k-images-idx3-ubyte.gz
Extracting data/t10k-labels-idx1-ubyte.gz
2017-06-14 18:54:40.157855: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-14 18:54:40.157887: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-14 18:54:40.157896: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-06-14 18:54:40.157903: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-14 18:54:40.157911: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-06-14 18:54:40.737822: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 0 with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:84:00.0
Total memory: 11.92GiB
Free memory: 11.86GiB
2017-06-14 18:54:40.737858: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0
2017-06-14 18:54:40.737867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0:   Y
2017-06-14 18:54:40.737881: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K80, pci bus id: 0000:84:00.0)
Initialized!
Step 0 (epoch 0.00), 17.1 ms
Minibatch loss: 8.334, learning rate: 0.010000
Minibatch error: 85.9%
Validation error: 84.6%
Step 100 (epoch 0.12), 13.4 ms
Minibatch loss: 3.254, learning rate: 0.010000
Minibatch error: 3.1%
Validation error: 7.8%
Step 200 (epoch 0.23), 11.6 ms
Minibatch loss: 3.354, learning rate: 0.010000
Minibatch error: 10.9%
Validation error: 4.5%
Step 300 (epoch 0.35), 11.5 ms
[...snip...]

Expand the tab below to see a demo of installing and using GPU support in a Singularity container.

Singularity can import, bootstrap, and even run Docker images directly from Docker Hub. For instance, the following commands will start an Ubuntu container running on a compute node with no need for a definition file or container image! And, of course, we remember to set SINGULARITY_BINDPATH appropriately to be able to access all our files.

[user@cn0123 ~]$ module load singularity
[+] Loading singularity 4.2.2 on cn0123
[user@cn0123 ~]$ . /usr/local/current/singularity/app_conf/sing_binds
[user@cn0123 ~]$ singularity shell docker://ubuntu:latest
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
INFO:    Fetching OCI image...
28.3MiB / 28.3MiB [=====================================] 100 % 14.6 MiB/s 0s
INFO:    Extracting OCI image...
INFO:    Inserting Singularity configuration...
INFO:    Creating SIF file...
Singularity> cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.3 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

[user@cn0123 ~]$ singularity pull docker://ubuntu:latest
INFO:    Using cached SIF image
[user@cn0123 ~]$ . /usr/local/current/singularity/app_conf/sing_binds
[user@cn0123 ~]$ singularity shell ubuntu_latest.sif
WARNING: Bind file source does not exist on host: /etc/resolv.conf
Singularity: Invoking an interactive shell within container...

Singularity ubuntu.img:~> cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.3 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

In the next example, we will create a Singularity container image starting from the official continuumio miniconda container on Docker Hub. Then we'll install a number of RNASeq tools. This would allow us to write a pipeline with, for example, Snakemake and distribute it along with the image to create an easily shared, reproducible workflow. This definition file also installs a runscript enabling us to treat our container like an executable.

BootStrap: docker
From: continuumio/miniconda3:latest
IncludeCmd: yes

%post
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# this will install all necessary packages and prepare the container
    apt-get -y update
    export PATH=/opt/conda/bin:$PATH
    conda update --yes conda
    conda install --yes -c conda-forge -c bioconda \
          star samtools salmon fastqc kallisto subread
    conda clean --all --yes
    mkdir /data /resources

%runscript
#!/bin/bash
unset which
function usage() {
    cat << EOF
NAME
    rnaseq - rnaseq pipeline tools 0.1
SYNOPSIS
    rnaseq tool [tool options]
    rnaseq list
    rnaseq help
DESCRIPTION
    Singularity container with tools to build rnaseq pipeline.
EOF
}

function tools() {
    echo "conda: $(which conda)"
    echo "---------------------------------------------------------------"
    conda list | grep bioconda
    echo "---------------------------------------------------------------"
    echo "samtools: $(samtools --version | head -n1)"
}

arg="${1:-none}"

case "$arg" in
    none) usage; exit 1;;
    help) usage; exit 0;;
    list) tools; exit 0;;
    # just try to execute it then
    *)    $@;;
esac

%environment
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# This sets global environment variables for anything run within the container
export PATH="/opt/conda/bin:/usr/local/bin:/usr/bin:/bin:"
unset CONDA_DEFAULT_ENV
export ANACONDA_HOME=/opt/conda

Assuming this file is called rnaseq.def, we can create a Singularity container called rnaseq on Biowulf itself (which use proot automatically):

[user@biowulf ~]$ sinteractive --mem=4G
[user@cn0123 ~]$ module load singularity
[user@cn0123 ~]$ singularity build rnaseq rnaseq.def

This image contains miniconda and our rnaseq tools and can be called directly as an executable like so:

[user@scn0123 ~]$ ./rnaseq help
NAME
    rnaseq - rnaseq pipeline tools 0.1
SYNOPSIS
    rnaseq snakemake [snakemake options]
    rnaseq list
    rnaseq help
DESCRIPTION
    Singularity container with tools to build rnaseq pipeline. 

[user@cn0123 ~]$ ./rnaseq list
conda: /opt/conda/bin/conda
---------------------------------------------------------------
fastqc                       0.12.1           hdfd78af_0             bioconda
htslib                       1.22.1           h566b1c6_0             bioconda
kallisto                     0.51.1           h2b92561_2             bioconda
libgff                       2.0.0            h077b44d_3             bioconda
salmon                       1.10.3           h45fbf2d_5             bioconda
samtools                     1.22.1           h96c455f_0             bioconda
staden_io_lib                1.15.1           hfc9290b_0             bioconda
star                         2.7.11b          h5ca1c30_7             bioconda
subread                      2.1.1            h577a1d6_0             bioconda
---------------------------------------------------------------
samtools: samtools 1.22.1

[user@cn0123 ~]$ ./rnaseq samtools --version
samtools 1.22.1
Using htslib 1.22.1
Copyright (C) 2025 Genome Research Ltd.

Samtools compilation details:
...

This could be used with a Snakemake file like this

rule fastqc:
    input: "{sample}.fq.gz"
    output: "{sample}.fastqc.html"
    shell: 
        """
        module load singularity
        ./rnaseq fastqc ... {input}
        """

rule align:
    input: "{sample}.fq.gz"
    output: "{sample}.bam"
    shell: 
        """
        module load singularity
        ./rnaseq STAR ....
        """

Expand the tab below to see an example of creating a Singularity container to be used as an executable from a Docker image on DockerHub.

A few containers have caused issues on Biowulf by triggering a kernel level bug described in detail here and here. These include fmriprep and nanodisco. The problems follow a predictable pattern:

• The containerized application uses more memory than has been allocated for the job.
• The Out Of Memory (OOM) killer is invoked by the kernel to kill the offending process.
• Other processes that presumably depend on the first process fail to cancel and may enter the D state (awaiting IO).
• Node responses become sluggish as it bogged down with D state processes.
• File system performance may suffer as the container tries repeatedly to communicate with the singularity starter process. Depending on the number of jobs running, this may produce noticeable degragation of file system performance.