QSIRecon on Biowulf

Quick Links

QSIRecon is a post-processing workflow for diffusion MRI data. It takes data that have already been preprocessed, usually with QSIPrep, and applies reconstruction and analysis methods from tools such as MRtrix3, DIPY, DSI Studio, and PyAFQ. Depending on the selected workflow, it can fit diffusion models, estimate fiber orientations, generate tractography, measure properties along white-matter pathways, and create regional connectivity tables. It provides standardized, reproducible workflows so researchers do not have to assemble every analysis step themselves.

Documentation

QSIRecon Documentation

Important Notes

Module Name: qsirecon (see the modules page for more information)

Interactive job

Interactive jobs should be used for debugging, graphics, or applications that cannot be run as ba tch jobs.

Below we use datalad on Helix to download a sample dataset, then test qsirecon on an sinteractive session (user input in bold):

[user@biowulf ~]$ ssh helix.nih.gov

[user@helix ~] cd /data/$USER
[user@helix ~] mkdir -pv QSIRECON_TEST
[user@helix ~] cd QSIRECON_TEST

[user@helix ~]$ module load datalad
[+] Loading datalad  1.1.4  on helix.nih.gov 

[user@helix ~]$ datalad install https://github.com/OpenNeuroDatasets/ds005134.git
[INFO   ] git-annex: http proxy settings not used due to annex.security.allowed-ip-addresses configuration                                                                                        
[INFO   ] Remote origin not usable by git-annex; setting annex-ignore 
[INFO   ] https://github.com/OpenNeuroDatasets/ds005134.git/config download failed: ConnectionFailure Network.Socket.getAddrInfo (called with preferred socket type/protocol: AddrInfo {addrFlags = [AI_ADDRCONFIG], addrFamily = AF_UNSPEC, addrSocketType = Stream, addrProtocol = 6, addrAddress = , addrCanonName = }, host name: Just "github.com", service name: Just "443"): does not exist (Name or service not known) 
install(ok): QSIRECON_TEST/ds005134 (dataset)

[user@helix ~]$ datalad get \
     -d ds005134 \
     -r ds005134/derivatives/qsiprep_0.21.4/qsiprep

[INFO   ] Ensuring presence of Dataset(QSIRECON_TEST/ds005134) to get QSIRECON_TEST/ds005134/derivatives/qsiprep_0.21.4/qsiprep 
get(ok): derivatives/qsiprep_0.21.4/qsiprep/logs/CITATION.html (file)                                      
get(ok): derivatives/qsiprep_0.21.4/qsiprep/logs/CITATION.md (file)                                        
get(ok): derivatives/qsiprep_0.21.4/qsiprep/logs/CITATION.tex (file)
get(ok): derivatives/qsiprep_0.21.4/qsiprep/sub-01.html (file)
get(ok): derivatives/qsiprep_0.21.4/qsiprep/sub-01/anat/sub-01_desc-aseg_dseg.nii.gz (file)
get(ok): derivatives/qsiprep_0.21.4/qsiprep/sub-01/anat/sub-01_desc-brain_mask.nii.gz (file)
get(ok): derivatives/qsiprep_0.21.4/qsiprep/sub-01/anat/sub-01_desc-preproc_T1w.nii.gz (file)
get(ok): derivatives/qsiprep_0.21.4/qsiprep/sub-01/anat/sub-01_dseg.nii.gz (file)
get(ok): derivatives/qsiprep_0.21.4/qsiprep/sub-01/anat/sub-01_from-MNI152NLin2009cAsym_to-T1w_mode-image_xfm.h5 (file)
get(ok): derivatives/qsiprep_0.21.4/qsiprep/sub-01/anat/sub-01_from-T1wACPC_to-T1wNative_mode-image_xfm.mat (file)
  [24 similar messages have been suppressed; disable with datalad.ui.suppress-similar-results=off]
get(ok): derivatives/qsiprep_0.21.4/qsiprep (directory)

[user@helix ~]$ exit

[user@biowulf ~] sinteractive --mem=50g --cpus-per-task=32
salloc: Pending job allocation 28020319
salloc: job 28020319 queued and waiting for resources
salloc: job 28020319 has been allocated resources
salloc: Granted job allocation 28020319
salloc: Waiting for resource configuration
salloc: Nodes cn1234 are ready for job

[user@cn1234 ~] cd /data/$USER/QSIRECON_TEST

[user@cn1234 ~] qsirecon ds005134/derivatives/qsiprep_0.21.4/qsiprep \
	           output \
	           participant \
	           --participant-label 01 \
	           --input-type qsiprep \
	           --recon-spec amico_noddi \
	           --output-resolution 2 \
	           --nprocs 32 \
	           --omp-nthreads 16 \
	           --mem 16000 \
	           -w work \
	           --notrack
      
260821-09:25:19,448 nipype.workflow IMPORTANT:
	 Running QSIRecon version 26.0.0

         License NOTICE ##################################################
         QSIRecon 26.0.0
         Copyright The PennLINC Developers.
         
         This product includes software developed by
         the NiPreps Community (https://nipreps.org/).
         
         Portions of this software were developed at the Department of
         Psychology at Stanford University, Stanford, CA, US.
         
         This software is also distributed as a Docker container image.
         The bootstrapping file for the image ("Dockerfile") is licensed
         under the MIT License.
         
         This software may be distributed through an add-on package called
         "Docker Wrapper" that is under the BSD 3-clause License.
         #################################################################

[...]
      
260821-09:39:10,987 nipype.workflow IMPORTANT:
	 QSIRecon finished successfully!
IMPORTANT:nipype.workflow:QSIRecon finished successfully!

Batch job

Most jobs should be run as batch jobs.

Create a batch input file (e.g. qsirecon.sh). For example:

#!/bin/bash

module load qsirecon

cd /data/${USER}/QSIRECON_TEST

qsirecon ds005134/derivatives/qsiprep_0.21.4/qsiprep \
	           output \
	           participant \
	           --participant-label 01 \
	           --input-type qsiprep \
	           --recon-spec amico_noddi \
	           --output-resolution 2 \
	           --nprocs 32 \
	           --omp-nthreads 16 \
	           --mem 16000 \
	           -w work \
	           --notrack

Submit this job using the Slurm sbatch command.

sbatch --cpus-per-task=32 --mem=50g qsirecon.sh

Swarm of Jobs

A swarm of jobs is an easy way to submit a set of independent commands requiring identical resources.

Create a swarmfile (e.g. qsirecon.swarm). For example:

qsirecon dataset1/derivatives/qsiprep output/dataset1 \
  participant \
  --participant-label 01 \
  --input-type qsiprep \
  --recon-spec amico_noddi \
  --output-resolution 2 \
  --nprocs 32 \
  --omp-nthreads 16 \
  --mem 16000 \
  -w /lscratch/${SLURM_JOB_ID} \
  --notrack
qsirecon dataset2/derivatives/qsiprep output/dataset2 \
  participant \
  --participant-label 01 \
  --input-type qsiprep \
  --recon-spec amico_noddi \
  --output-resolution 2 \
  --nprocs 32 \
  --omp-nthreads 16 \
  --mem 16000 \
  -w /lscratch/${SLURM_JOB_ID} \
  --notrack
qsirecon dataset3/derivatives/qsiprep output/dataset3 \
  participant \
  --participant-label 01 \
  --input-type qsiprep \
  --recon-spec amico_noddi \
  --output-resolution 2 \
  --nprocs 32 \
  --omp-nthreads 16 \
  --mem 16000 \
  -w /lscratch/${SLURM_JOB_ID} \
  --notrack

Submit this job using the swarm command.

swarm --gres=lscratch:50] -g 50 -t 32 --module qsirecon qsirecon.swarm

where

`-gres=lscratch:#`	Number of Gigabytes of local disk space allocated per process (1 line in the swarm command file)
`-g #`	Number of Gigabytes of memory required for each process (1 line in the swarm command file)
`-t #`	Number of threads/CPUs required for each process (1 line in the swarm command file).
`--module qsirecon`	Loads the qsirecon module for each subjob in the swarm