Biowulf High Performance Computing at the NIH
BOLT-LMM on Biowulf

The BOLT-LMM software package currently consists of two main algorithms, the BOLT-LMM algorithm for mixed model association testing, and the BOLT-REML algorithm for variance components analysis (i.e., partitioning of SNP-heritability and estimation of genetic correlations).

References:

Documentation
Important Notes

Interactive job
Interactive jobs should be used for debugging, graphics, or applications that cannot be run as batch jobs.

Allocate an interactive session and run the program. Sample session:

[user@biowulf]$ sinteractive
salloc.exe: Pending job allocation 46116226
salloc.exe: job 46116226 queued and waiting for resources
salloc.exe: job 46116226 has been allocated resources
salloc.exe: Granted job allocation 46116226
salloc.exe: Waiting for resource configuration
salloc.exe: Nodes cn3144 are ready for job

[user@cn3144 ~]$ module load BOLT-LMM
[user@cn3144 ~]$ cp -r /usr/local/apps/BOLT-LMM/v2.3/example /data/$USER/bolt
[user@cn3144 ~]$ cp -r /usr/local/apps/BOLT-LMM/v2.3/tables /data/$USER/bolt
[user@cn3144 ~]$ cd /data/$USER/bolt/examples
[user@cn3144 ~]$ bolt \
[user@cn3144 ~]$     --bfile=EUR_subset \
[user@cn3144 ~]$     --remove=EUR_subset.remove \
[user@cn3144 ~]$     --exclude=EUR_subset.exclude \
[user@cn3144 ~]$     --phenoFile=EUR_subset.pheno.covars \
[user@cn3144 ~]$     --phenoCol=PHENO \
[user@cn3144 ~]$     --covarFile=EUR_subset.pheno.covars \
[user@cn3144 ~]$     --covarCol=CAT_COV \
[user@cn3144 ~]$     --qCovarCol=QCOV{1:2} \
[user@cn3144 ~]$     --modelSnps=EUR_subset.modelSnps \
[user@cn3144 ~]$     --lmm \
[user@cn3144 ~]$     --LDscoresFile=../tables/LDSCORE.1000G_EUR.tab.gz \
[user@cn3144 ~]$     --numThreads=2 \
[user@cn3144 ~]$     --statsFile=example.stats \
[user@cn3144 ~]$     --dosageFile=EUR_subset.dosage.chr17first100 \
[user@cn3144 ~]$     --dosageFile=EUR_subset.dosage.chr22last100.gz \
[user@cn3144 ~]$     --dosageFidIidFile=EUR_subset.dosage.indivs \
[user@cn3144 ~]$     --statsFileDosageSnps=example.dosageSnps.stats \
[user@cn3144 ~]$     --impute2FileList=EUR_subset.impute2FileList.txt \
[user@cn3144 ~]$     --impute2FidIidFile=EUR_subset.impute2.indivs \
[user@cn3144 ~]$     --statsFileImpute2Snps=example.impute2Snps.stats \
[user@cn3144 ~]$     --dosage2FileList=EUR_subset.dosage2FileList.txt \ 
[user@cn3144 ~]$     --statsFileDosage2Snps=example.dosage2Snps.stats \
[user@cn3144 ~]$     2>&1 | tee example.log # log output written to stdout and stderr
[user@cn3144 ~]$ exit
salloc.exe: Relinquishing job allocation 46116226
[user@biowulf ~]$

Batch job
Most jobs should be run as batch jobs.

Create a batch input file (e.g., run_example.sh in the example directory). For example:

#!/bin/bash
module load BOLT-LMM
bolt \
    --bfile=EUR_subset \
    --remove=EUR_subset.remove \
    --exclude=EUR_subset.exclude \
    --phenoFile=EUR_subset.pheno.covars \
    --phenoCol=PHENO \
    --covarFile=EUR_subset.pheno.covars \
    --covarCol=CAT_COV \
    --qCovarCol=QCOV{1:2} \
    --modelSnps=EUR_subset.modelSnps \
    --lmm \
    --LDscoresFile=../tables/LDSCORE.1000G_EUR.tab.gz \
    --numThreads=2 \
    --statsFile=example.stats \
    --dosageFile=EUR_subset.dosage.chr17first100 \
    --dosageFile=EUR_subset.dosage.chr22last100.gz \
    --dosageFidIidFile=EUR_subset.dosage.indivs \
    --statsFileDosageSnps=example.dosageSnps.stats \
    --impute2FileList=EUR_subset.impute2FileList.txt \
    --impute2FidIidFile=EUR_subset.impute2.indivs \
    --statsFileImpute2Snps=example.impute2Snps.stats \
    --dosage2FileList=EUR_subset.dosage2FileList.txt \
    --statsFileDosage2Snps=example.dosage2Snps.stats \
    2>&1 | tee example.log # log output written to stdout and stderr

Submit this job using the Slurm sbatch command.

sbatch [--cpus-per-task=#] [--mem=#] run_example.sh

Note: For test runs: Copy /usr/local/apps/BOLT-LMM/v2.3/example and /usr/local/apps/BOLT-LMM/v2.3/tables to your local directory. Under the example directory, run_example.sh and run_example_reml2.sh are batch scripts that can be used in slurm

Swarm of Jobs
A swarm of jobs is an easy way to submit a set of independent commands requiring identical resources.

Create a swarmfile (e.g. BOLT-LMM.swarm). For example:

bolt --bfile=AFR_subset ...
bolt --bfile=AMR_subset ...
bolt --bfile=EAS_subset ...
bolt --bfile=EUR_subset ...
bolt --bfile=SAS_subset ...

Submit this job using the swarm command.

swarm -f BOLT-LMM.swarm [-g #] [-t #] --module BOLT-LMM
where
-g # Number of Gigabytes of memory required for each process (1 line in the swarm command file)
-t # Number of threads/CPUs required for each process (1 line in the swarm command file).
--module BOLT-LMM Loads the BOLT-LMM module for each subjob in the swarm