Pigz is a multi-core implementation of gzip.

• pigz Main Site: GitHub

• Module Name: pigz (see the modules page for more information)
• Environment variables set: PIGZ_HOME.
• Ensure that the number of CPUs allocated is equal or greater than the threads used in the pigz command.
• Note that to be able to use the environment variable SLURM_MEM_PER_NODE below, you will need to explicitly allocate memory resources.

Allocate an interactive session and run the program. Sample session:

[user@biowulf]$ sinteractive --cpus-per-task=4 --mem=4g
salloc.exe: Pending job allocation 46116226
salloc.exe: job 46116226 queued and waiting for resources
salloc.exe: job 46116226 has been allocated resources
salloc.exe: Granted job allocation 46116226
salloc.exe: Waiting for resource configuration
salloc.exe: Nodes cn4224 are ready for job

[user@cn4224 ~]$ module load pigz
[+] Loading pigz 2.7 on cn4224 

[user@cn4224 ~]$ pigz --best -p 4 -v longshot_output.vcf
longshot_output.vcf to longshot_output.vcf.gz 

[user@cn4224 ~]$ exit
salloc.exe: Relinquishing job allocation 46116226

[user@biowulf ~]$

Create a batch input file (e.g. pigz.sh) similar to the following.

#! /bin/bash

set -e

module load pigz

pigz --best -p 4 longshot_output.vcf

Submit these jobs using the Slurm sbatch command.

sbatch --cpus-per-task=4 --mem=4g pigz.sh

Create a swarmfile for the first step of the pipeline (e.g. pigz.swarm). For example:

pigz --best -p 4 longshot_output_1.vcf
pigz --best -p 4 longshot_output_2.vcf
pigz --best -p 4 longshot_output_3.vcf

Submit this job using the swarm command.

swarm -f pigz.swarm -g 4 -t 4 --module pigz