Quick Links
|
Quick Links
|
Apptainer, previously known as Singularity, is a tool allowing you to build and run Linux containers. Linux containers can be thought of as small, lightweight virtual machines encapsulating an entire operating system. Containers let users run applications in a Linux environment of their choosing.
Possible uses for Apptainer on Biowulf:
Please note, Apptainer gives you the ability to install and run applications in your own Linux environment with your own customized software stack. With this ability comes the added responsibility of managing your own Linux environment. While the NIH HPC staff can provide guidance on how to create and use Apptainer containers, we do not have the resources to manage containers for individual users. If you decide to use Apptainer, it is your responsibility to manage your own containers.
export APPTAINER_CACHEDIR=/data/${USER}/.apptainer
mv ~/.apptainer /data/${USER}/.apptainer
ln -s /data/${USER}/.apptainer ~/.apptainer
To use Apptainer on Biowulf, you either need to use a pre-built container created by someone else, or build your own container. Unfortunately, building containers using definition files is currently not supported on the NIH HPC systems. You have several options to build Apptainer containers:
You can find information about installing Apptainer on Linux here.
In addition to your own Linux environment, you will also need a definition file to build an Apptainer container from scratch. You can find some simple definition files for a variety of Linux distributions in the /example directory of the source code. You can also find links to definition files containing popular applications in the Documentation section above. Detailed documentation about building Apptainer container images is available at the Apptainer website.
Binding a directory to your Apptainer container allows you to read and write files on the host system from within your container. By default, Apptainer binds your $HOME directory (along with a few other directories such as /tmp and /dev). You can also bind other directories into your container yourself. The process is described in detail in the Apptainer documentation.
There are several filesystems on the NIH HPC systems that you may want to access from within your container. If you are running a job and have allocated local scratch space, you might like to bind your /lscratch into the container as well. You can bind directories into your container at runtime using either the --bind option or by setting the $APPTAINER_BINDPATH environment variable.
The following command opens a shell in a container while bind-mounting your /data directory, /fdb, and /lscratch into the container. If you have access to shared data directories, you could add them to the list as well (for example, /data/$USER,/data/mygroup1,/data/mygroup2,/fdb,...).
[user@cn1234 ~]$ apptainer shell --bind /data/$USER,/fdb,/lscratch my-container.sifor, using the environment variable:
[user@cn1234 ~]$ export APPTAINER_BINDPATH="/data/$USER,/fdb,/lscratch" [user@cn1234 ~]$ apptainer shell my-container.sif
The NIH HPC staff maintains a file that will set the $APPTAINER_BINDPATH environment variable appropriately for a wide variety of situations. It is considered a Biowulf best practice to source this file since it will be updated in the case of additions or deletions to the shared file system. You can source this file from the command prompt or from within a script like so:
[user@cn1234 ~]$ . /usr/local/current/apptainer/app_conf/sing_binds
Sourcing this file instead of setting bind paths yourself can help to "future-proof" your workflow since it is maintained by NIH HPC staff.
Remember that you may need to change the directories that you bind into the container if you use your container on a different system or share it with a colleague.
One can use Apptainer to "install" software and use it transparently as though it were installed directly on the host system. In fact, NIH HPC staff members use Apptainer to install a large number of scientific applications. This method can make it easier to install software and renders the final product portable. For example, if you wanted to use the default Debian package manager (APT) to "install" software on Biowulf you could do something like this. Here we install samtools and bcftools, with the following definition file:
Bootstrap: docker
From: debian:9-slim
%post
# install the desired software
apt-get update
apt-get install -y samtools bcftools
apt-get clean
This defines a container based on the space-efficient "slim" Debian images from Docker Hub and installs the samtools and bcftools packages via APT.
After finalizing the definition file, you can proceed to build the container (of course, on a system where you have sudo or root access):
[user@some_build_host ~]$ sudo apptainer build hts.sif hts.def
You can then set up your installation prefix (here, it's $HOME/opt/hts) as follows, making use of symbolic links and a wrapper script:
$HOME/opt
└── hts
├── bin
│ ├── samtools -> ../libexec/wrap
│ └── bcftools -> ../libexec/wrap
└── libexec
├── wrap
└── hts.simg
where the wrapper script wrap looks like:
#!/bin/bash
. /usr/local/current/apptainer/app_conf/sing_binds
selfdir="$(dirname $(readlink -f ${BASH_SOURCE[0]}))"
cmd="$(basename $0)"
apptainer exec "${selfdir}/hts.simg" "$cmd" "$@"
wrap checks to see how it was called, then passes that same command to the container after appropriately setting APPTAINER_BINDPATH by calling the staff maintained sing_binds script.
So if you have added the installation prefix $HOME/opt/hts/bin to your PATH, then calling samtools or bcftools will run those programs from within your container. And because we have arranged to bind mount all the necessary filesystems into the container, the path names you provide for input and output into the programs will be available to the container in the same way.
Like most scientific applications, Apptainer cannot be run on the Biowulf login node.
To run an Apptainer container image on Biowulf interactively, you need to allocate an interactive session, and load the Apptainer module (user input in bold).
[user@biowulf ~]$ sinteractive -c4 --mem=8g --gres=lscratch:10 salloc: Pending job allocation 35492498 salloc: job 35492498 queued and waiting for resources salloc: job 35492498 has been allocated resources salloc: Granted job allocation 35492498 salloc: Waiting for resource configuration salloc: Nodes cn0991 are ready for job [user@cn0991 ~]$ module load apptainer [+] Loading apptainer 1.4.5 on cn0991 [user@cn0991 ~]$ cd /lscratch/$SLURM_JOB_ID
You can download containers using the pull command. Apptainer containers are saved on disk as image files. In this example, we use the library:// URI to download containers from the Sylabs cloud library.
[user@cn0991 35492498]$ apptainer pull docker://godlovedc/lolcow
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
INFO: Fetching OCI image...
45.3MiB / 45.3MiB [=====================================] 100 % 20.9 MiB/s 0s
53.7MiB / 53.7MiB [=====================================] 100 % 20.9 MiB/s 0s
INFO: Extracting OCI image...
2026/10/08 10:40:25 warn rootless{dev/agpgart} creating empty file in place of device 10:175
2026/10/08 10:40:25 warn rootless{dev/audio} creating empty file in place of device 14:4
[...]
INFO: Inserting Apptainer configuration...
INFO: Creating SIF file...
INFO: To see mksquashfs output with progress bar enable verbose logging
[user@cn0991 35492498]$ ls -lh
total 90M
-rwxr-xr-x 1 user user 88M Apr 4 12:27 lolcow_latest.sif
[user@cn0991 35492498]$
You can "run" your newly downloaded container either using the run command, or by treating the container as an executable file and supplying its path.
[user@cn0991 35492498]$ apptainer run lolcow_latest.sif
INFO: squashfuse not found, will not be able to use gocryptfs
INFO: gocryptfs not found, will not be able to use gocryptfs
___________________________________
< You will triumph over your enemy. >
-----------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
[user@cn0991 35492498]$ ./lolcow_latest.sif
_________________________________________
/ It is so very hard to be an \
| on-your-own-take-care-of-yourself-becau |
| se-there-is-no-one-else-to-do-it-for-yo |
\ u grown-up. /
-----------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
[user@cn0991 35492498]$
You can start a new shell session within the container you downloaded using the shell command. This is useful if you want to look around and find things inside the container. We will use -s option to suppress the INFO messages. Note how the operating system in the container has changed to Ubuntu 16.04 (codenamed "Xenial Xerus"). Also note that you must execute the exit command to quit the shell session within the container when you are finished.
[user@cn0991 35492498]$ apptainer -s shell lolcow_latest.sif
Apptainer> cowsay moo
_____
< moo >
-----
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
Apptainer> cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
Apptainer> exit
exit
[user@cn0991 35492498]$
The exec command is useful to execute a single command inside the container.
[user@cn0991 35492498]$ apptainer -s exec lolcow_latest.sif fortune Be security conscious -- National defense is at stake. [user@cn0991 35492498]$
Good advice.
Finally, you may find it useful to use the run, exec, or shell commands on containers you have not downloaded. In this example we use the shell command to start a shell session within the official Docker Hub alpine container, thereby switching to the Alpine operating system with a single command.
[user@cn0991 35492498]$ apptainer shell docker://alpine INFO: Converting OCI blobs to SIF format INFO: Starting build... INFO: Fetching OCI image... 3.7MiB / 3.7MiB [===========================================================] 100 %0s INFO: Extracting OCI image... INFO: Inserting Apptainer configuration... INFO: Creating SIF file... INFO: To see mksquashfs output with progress bar enable verbose logging INFO: squashfuse not found, will not be able to use gocryptfs INFO: gocryptfs not found, will not be able to use gocryptfs Apptainer> cat /etc/os-release NAME="Alpine Linux" ID=alpine VERSION_ID=3.24.2 PRETTY_NAME="Alpine Linux v3.24" HOME_URL="https://alpinelinux.org/" BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues" Apptainer> exit [user@cn0991 35492498]$
This is a very small sample of Apptainer's vast capabilities. See the official user documentation for a thorough introduction.
To leverage the hardware on a GPU node, you must pass the --nv option with your Apptainer command. Consider the following example in which a toy deep learning model is trained on a GPU using a tensorflow container:
[user@biowulf ~]$ sinteractive -c8 --mem=32g --gres=gpu:p100:1,lscratch:20
salloc: Pending job allocation 35509918
salloc: job 35509918 queued and waiting for resources
salloc: job 35509918 has been allocated resources
salloc: Granted job allocation 35509918
salloc: Waiting for resource configuration
salloc: Nodes cn4194 are ready for job
[user@cn4194 ~]$ cd /lscratch/$SLURM_JOB_ID
[user@cn4194 35509918]$ cat >hello-tflow.py<<'EOF'
import tensorflow as tf
mnist = tf.keras.datasets.mnist
(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10, activation='softmax')
])
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(x_train, y_train, epochs=5)
model.evaluate(x_test, y_test)
EOF
[user@cn4194 35509918]$ module load apptainer
[+] Loading apptainer 1.0.1 on cn4194
[user@cn4194 35509918]$ . /usr/local/current/apptainer/app_conf/sing_binds
[user@cn4194 35509918]$ apptainer exec --nv docker://tensorflow/tensorflow:latest-gpu python hello-tflow.py
INFO: Using cached SIF image
2022-04-04 16:39:07.555227: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-04-04 16:39:09.893898: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 11560 MB memory: -> device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:84:00.0, compute capability: 3.7
Epoch 1/5
1875/1875 [==============================] - 6s 2ms/step - loss: 0.2960 - accuracy: 0.9143
Epoch 2/5
1875/1875 [==============================] - 3s 2ms/step - loss: 0.1387 - accuracy: 0.9583
Epoch 3/5
1875/1875 [==============================] - 3s 2ms/step - loss: 0.1043 - accuracy: 0.9683
Epoch 4/5
1875/1875 [==============================] - 3s 2ms/step - loss: 0.0861 - accuracy: 0.9724
Epoch 5/5
1875/1875 [==============================] - 3s 2ms/step - loss: 0.0752 - accuracy: 0.9763
313/313 [==============================] - 1s 2ms/step - loss: 0.0735 - accuracy: 0.9766
[user@cn4194 35509918]$
More detailed information about GPU support in Apptainer can be obtained from the official docs.
In this example, we assume that you have the script from the GPU example above saved in your /data directory. To do so, execute the following commands:
[user@biowulf ~]$ cd /data/$USER
[user@biowulf user]$ cat >hello-tflow.py<<'EOF'
import tensorflow as tf
mnist = tf.keras.datasets.mnist
(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10, activation='softmax')
])
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(x_train, y_train, epochs=5)
model.evaluate(x_test, y_test)
EOF
[user@biowulf user]$
Then write a batch script to run the apptainer command similar to this:
#!/bin/bash
# file called myjob.batch
set -e
module load apptainer
cd /data/$USER
. /usr/local/current/apptainer/app_conf/sing_binds
apptainer exec --nv docker://tensorflow/tensorflow:latest-gpu \
python /data/${USER}/hello-tflow.py
Submit the job like so:
[user@biowulf user]$ sbatch --time=10 --cpus-per-task=8 --partition=gpu --mem=32g --gres=gpu:p100:1 myjob.batch 35686040 [user@biowulf user]$
After the job finishes executing you should see the following output in the slurm*.out file.
[user@biowulf user]$ cat slurm-35686040.out [+] Loading apptainer 1.0.1 on cn4181 INFO: Using cached SIF image 2022-04-06 12:27:30.092624: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2022-04-06 12:27:37.983259: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 11560 MB memory: -> device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:8b:00.0, compute capability: 3.7 Epoch 1/5 1875/1875 [==============================] - 12s 2ms/step - loss: 0.3009 - accuracy: 0.9131 Epoch 2/5 1875/1875 [==============================] - 3s 2ms/step - loss: 0.1458 - accuracy: 0.9569 Epoch 3/5 1875/1875 [==============================] - 3s 2ms/step - loss: 0.1093 - accuracy: 0.9670 Epoch 4/5 1875/1875 [==============================] - 3s 2ms/step - loss: 0.0871 - accuracy: 0.9732 Epoch 5/5 1875/1875 [==============================] - 3s 2ms/step - loss: 0.0750 - accuracy: 0.9768 313/313 [==============================] - 1s 2ms/step - loss: 0.0802 - accuracy: 0.9764 [user@biowulf user]$
The example above shows how to run a script within a containerized environment on Biowulf. This example can be easily extended to a swarm file with a series of commands. Let's assume you have a list of python scripts that you want to run in a custom python environment.
First, source the sing_binds file to set the $APPTAINER_BINDPATH variable appropriately in your environment. This variable will propagate to your jobs so that your Apptainer container will have access to files on the host system.
[user@biowulf ~]$ . /usr/local/current/apptainer/app_conf/sing_binds [user@biowulf ~]$
Create a swarmfile (e.g. apptainer.swarm). For example:
apptainer exec docker://python python /data/${USER}/script1.py
apptainer exec docker://python python /data/${USER}/script2.py
apptainer exec docker://python python /data/${USER}/script3.py
apptainer exec docker://python python /data/${USER}/script4.py
Submit this job using the swarm command.
[user@biowulf ~]$ swarm -f apptainer.swarm [-g #] [-t #] --module apptainerwhere
| -g # | Number of Gigabytes of memory required for each process (1 line in the swarm command file) |
| -t # | Number of threads/CPUs required for each process (1 line in the swarm command file). |
| --module apptainer | Loads the apptainer module for each subjob in the swarm |
Prior to 2023, a few containers caused issues on Biowulf by triggering a kernel level bug described in detail here and here. These included fmriprep and nanodisco container. The problems followed a predictable pattern:
Since the OS upgrade in 2023, this is generally not a problem. However, if one of your containers displays these symptoms, you will need to convert the container from sif format to ext3 format. This a non-trivial operation and requires you have access to a system with root privileges. Please contact staff@hpc.nih.gov and we can give you recommendations.
In this example, we will create an Apptainer container image starting from the official continuumio miniconda container on Docker Hub. Then we'll install a number of RNASeq tools. This would allow us to write a pipeline with, for example, Snakemake and distribute it along with the image to create an easily shared, reproducible workflow. This definition file also installs a runscript enabling us to treat our container like an executable.
BootStrap: docker
From: continuumio/miniconda3:latest
IncludeCmd: yes
%post
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# this will install all necessary packages and prepare the container
apt-get -y update
export PATH=/opt/conda/bin:$PATH
conda update --yes conda
conda install --yes -c conda-forge -c bioconda \
star samtools salmon fastqc kallisto subread
conda clean --all --yes
mkdir /data /resources
%runscript
#!/bin/bash
unset which
function usage() {
cat << EOF
NAME
rnaseq - rnaseq pipeline tools 0.1
SYNOPSIS
rnaseq tool [tool options]
rnaseq list
rnaseq help
DESCRIPTION
Singularity container with tools to build rnaseq pipeline.
EOF
}
function tools() {
echo "conda: $(which conda)"
echo "---------------------------------------------------------------"
conda list | grep bioconda
echo "---------------------------------------------------------------"
echo "samtools: $(samtools --version | head -n1)"
}
arg="${1:-none}"
case "$arg" in
none) usage; exit 1;;
help) usage; exit 0;;
list) tools; exit 0;;
# just try to execute it then
*) $@;;
esac
%environment
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# This sets global environment variables for anything run within the container
export PATH="/opt/conda/bin:/usr/local/bin:/usr/bin:/bin:"
unset CONDA_DEFAULT_ENV
export ANACONDA_HOME=/opt/conda
Assuming this file is called rnaseq.def, we can create an Apptainer container called rnaseq on our build system with the following commands. If your build system is new enough, you should not need to run the build command with sudo:
[user@some_build_system ~]$ sudo apptainer build rnaseq rnaseq.def
Once your image is built successfully, transfer the file to the NIH HPC systems. This image file contains miniconda and our rnaseq tools and can be called directly as an executable in an interactive session. For example:
[user@cn1234 ~]$ module load apptainer
[user@cn1234 ~]$ ./rnaseq help
NAME
rnaseq - rnaseq pipeline tools 0.1
SYNOPSIS
rnaseq snakemake [snakemake options]
rnaseq list
rnaseq help
DESCRIPTION
Singularity container with tools to build rnaseq pipeline.
[user@cn0123 ~]$ ./rnaseq list
conda: /opt/conda/bin/conda
---------------------------------------------------------------
fastqc 0.12.1 hdfd78af_0 bioconda
htslib 1.22.1 h566b1c6_0 bioconda
kallisto 0.51.1 h2b92561_2 bioconda
libgff 2.0.0 h077b44d_3 bioconda
salmon 1.10.3 h45fbf2d_5 bioconda
samtools 1.22.1 h96c455f_0 bioconda
staden_io_lib 1.15.1 hfc9290b_0 bioconda
star 2.7.11b h5ca1c30_7 bioconda
subread 2.1.1 h577a1d6_0 bioconda
---------------------------------------------------------------
[user@cn0123 ~]$ ./rnaseq samtools --version
samtools 1.22.1
Using htslib 1.22.1
Copyright (C) 2025 Genome Research Ltd.
Samtools compilation details:
...
This could be used with a Snakemake file like this
rule fastqc:
input: "{sample}.fq.gz"
output: "{sample}.fastqc.html"
shell:
"""
module load apptainer
./rnaseq fastqc ... {input}
"""
rule align:
input: "{sample}.fq.gz"
output: "{sample}.bam"
shell:
"""
module load apptainer
./rnaseq STAR ....
"""