Biowulf High Performance Computing at the NIH
xcpEngine on Biowulf

xcpEngine is a pipeline application that performs denoising of fMRI datasets and computes functional connectivity. This pipeline can take as input a dataset that has been preprocessed with fmriprep

Web site

Important Notes

Interactive job
Interactive jobs should be used for debugging, graphics, or applications that cannot be run as batch jobs.

Allocate an interactive session and run the program.
Sample session to display usage (user input in bold):

[user@biowulf]$ sinteractive
salloc.exe: Pending job allocation 46116226
salloc.exe: job 46116226 queued and waiting for resources
salloc.exe: job 46116226 has been allocated resources
salloc.exe: Granted job allocation 46116226
salloc.exe: Waiting for resource configuration
salloc.exe: Nodes cn1234 are ready for job

[user@cn1234 ~]$ module load xcpengine

[user@cn1234 ~]$ xcpEngine
   Usage: xcpEngine -d  

   Compulsory arguments:
    -d : Primary design file for pipeline:
         The design file specifies the pipeline modules to
         be included in the current analysis and configures
         any parameters necessary to run the modules.
    -c : Cohort file for pipeline input:
         A comma-separated catalogue of the analytic sample.
         Each row corresponds to a subject, and each column
         corresponds either to an identifier or to an input.
    -o : Parent directory for pipeline output:
         A valid path on the current filesystem specifying
         the directory wherein all output from the current
         analysis will be written.

   Optional arguments:
    -m : Execution mode:
         Input can either be 's' (for serial execution on a
         single machine)[default], 'c' (for execution on a
         computing cluster) or a path to a file (for execution
         on a computing cluster, subject to the specifications
         defined in the file).
    -i : Scratch space for pipeline intermediates:
         Some systems operate more quickly when temporary
         files are written in a dedicated scratch space. This
         argument enables a scratch space for intermediates.
    -r : Root directory for inputs:
         If all paths defined in the cohort file are defined
         relative to a root directory, then this argument will
         define the root directory. Otherwise, all paths will
         be treated as absolute.
    -t : Trace:
         Integer value ( 0 - 3 ) that indicates the level
         of verbosity during module execution. Higher
         levels reduce readability but provide useful
         information for troubleshooting.
         0  [default]: Human-readable explanations of
            processing steps and error traces only.
         1: Explicitly trace module-level computations.
            Print a workflow map when execution completes.
         2: Explicitly trace module- and utility-level
         3: All commands called by the module and all
            children are traced and explicitly replicated
            in a log file.

[user@cn1234 ~]$ exit
salloc.exe: Relinquishing job allocation 46116226
[user@biowulf ~]$

Batch job
Most jobs should be run as batch jobs.

Create a batch input file (e.g. For example:

# sbatch --gres=lscratch:100 --mem=32g --cpus-per-task=48 --time=72:00:00

set -e

function fail {
    echo "FAIL: $@" >&2
    exit 1  # signal failure

module load xcpengine; \
cd /lscratch/$SLURM_JOB_ID/
cp /data/user/XCPENGINE_TEST/fmriprep.tar.gz /lscratch/$SLURM_JOB_ID/.; \
tar -xzf /lscratch/$SLURM_JOB_ID/fmriprep.tar.gz -C /lscratch/$SLURM_JOB_ID; \
cp /data/user/XCPENGINE_TEST/anat-antsct.dsn /lscratch/$SLURM_JOB_ID/.; \
cp /data/user/XCPENGINE_TEST/anat_cohort.csv /lscratch/$SLURM_JOB_ID/.; \
xcpEngine -d /lscratch/$SLURM_JOB_ID/anat-antsct.dsn -c /lscratch/$SLURM_JOB_ID/anat_cohort.csv -o /lscratch/$SLURM_JOB_ID/xcp_output -t 1 -i /lscratch/$SLURM_JOB_ID -r /lscratch/$SLURM_JOB_ID; \
tar -czf xcp_output.tar.gz xcp_output; \
cp /lscratch/$SLURM_JOB_ID/xcp_output.tar.gz /data/user/XCPENGINE_TEST/. 

Submit this job using the Slurm sbatch command.

sbatch [--gres=lscratch:#] [--cpus-per-task=#] [--mem=#]