HALFpipe on Biowulf

HALFpipe is a workflow for analyzing resting-state and task-based fMRI data. It supports preprocessing, individual participant analysis, and group analysis, and it can work with datasets that have not already been converted to BIDS format. HALFpipe combines methods from established neuroimaging tools, including fMRIPrep, ANTs, FreeSurfer, FSL, AFNI, and Nipype, to provide a consistent and reproducible analysis process.

Documentation
Important Notes

Interactive job
Interactive jobs should be used for debugging, graphics, or applications that cannot be run as ba tch jobs.
Below we use datalad on Helix to download a sample dataset, then test halfpipe on an sinteractive session (user input in bold):

[user@biowulf ~]$ ssh helix.nih.gov

[user@helix ~] cd /data/$USER
[user@helix ~] mkdir -pv HALFPIPE_TEST
[user@helix ~] cd HALFPIPE_TEST

[user@helix ~]$ module load datalad
[+] Loading datalad  1.1.4  on helix.nih.gov 

[user@helix ~]$ datalad install \
                   https://github.com/OpenNeuroDatasets/ds000005.git
                
[...]
install(ok): HALFPIPE_TEST/ds000005 (dataset)

[user@helix ~]$ cd ds000005

[user@helix ~]$ datalad get \
    dataset_description.json \
    task-mixedgamblestask_bold.json \
    sub-01/anat/sub-01_T1w.nii.gz \
    sub-01/func/sub-01_task-mixedgamblestask_run-01_bold.nii.gz \
    sub-01/func/sub-01_task-mixedgamblestask_run-01_events.tsv

get(ok): sub-01/func/sub-01_task-mixedgamblestask_run-01_bold.nii.gz (file) [from s3-PUBLIC...]        
get(ok): sub-01/anat/sub-01_T1w.nii.gz (file) [from s3-PUBLIC...]                                      
action summary:
  get (notneeded: 3, ok: 2)

[user@helix ~]mkdir -p ../bids/sub-01/anat

[user@helix ~]mkdir -p ../bids/sub-01/func

[user@helix ~]cp dataset_description.json task-mixedgamblestask_bold.json ../bids/

[user@helix ~]cp -L sub-01/anat/sub-01_T1w.nii.gz ../bids/sub-01/anat/

[user@helix ~]cp -L \
    sub-01/func/sub-01_task-mixedgamblestask_run-01_bold.nii.gz \
    sub-01/func/sub-01_task-mixedgamblestask_run-01_events.tsv \
    ../bids/sub-01/func/

[user@helix ~]$ exit

[user@biowulf ~]$ sinteractive --mem=32g --cpus-per-task=16 --gres=lscratch:100
salloc: Pending job allocation 28020319
salloc: job 28020319 queued and waiting for resources
salloc: job 28020319 has been allocated resources
salloc: Granted job allocation 28020319
salloc: Waiting for resource configuration
salloc: Nodes cn1234 are ready for job

[user@cn1234 ~]$ cd /data/$USER/HALFPIPE_TEST
[user@cn1234 ~]$ module load halfpipe
[+] Loading halfpipe  1.3.2  on cn4271 
[+] Loading singularity  4.3.7  on cn4271 

[user@cn1234 ~]$ mkdir -p /data/${USER}/HALFPIPE_TEST/config

[user@cn1234 ~]$ 
cat > /data/${USER}/HALFPIPE_TEST/config/spec.json <<EOF
{
  "halfpipe_version": "1.3.2",
  "schema_version": "3.0",
  "timestamp": "$(date +%Y-%m-%d_%H-%M)",
  "global_settings": {
    "dummy_scans": 0,
    "slice_timing": false,
    "use_bbr": null,
    "skull_strip_algorithm": "ants",
    "run_mriqc": false,
    "run_fmriprep": true,
    "run_halfpipe": true,
    "fd_thres": 0.5,
    "anat_only": false,
    "write_graph": false,
    "hires": false,
    "run_reconall": false,
    "t2s_coreg": false,
    "medial_surface_nan": false,
    "bold2t1w_dof": 9,
    "fmap_bspline": true,
    "force_syn": false,
    "longitudinal": false,
    "regressors_all_comps": false,
    "regressors_dvars_th": 1.5,
    "regressors_fd_th": 0.5,
    "skull_strip_fixed_seed": false,
    "skull_strip_template": "OASIS30ANTs",
    "run_aroma": false,
    "aroma_err_on_warn": false,
    "aroma_melodic_dim": -200,
    "sloppy": false
  },
  "files": [
    {
      "datatype": "anat",
      "tags": {},
      "extension": ".nii.gz",
      "path": "/data/${USER}/HALFPIPE_TEST/bids/sub-{sub}/anat/sub-{sub}_T1w.nii.gz",
      "suffix": "T1w"
    },
    {
      "datatype": "func",
      "tags": {
        "task": "mixedgamblestask",
        "run": "01"
      },
      "metadata": {
        "repetition_time": 2.0
      },
      "extension": ".nii.gz",
      "path": "/data/${USER}/HALFPIPE_TEST/bids/sub-{sub}/func/sub-{sub}_task-mixedgamblestask_run-01_bold.nii.gz",
      "suffix": "bold"
    }
  ],
  "settings": [
    {
      "name": "preprocessed",
      "space": "standard",
      "ica_aroma": false,
      "output_image": true
    }
  ],
  "features": [],
  "models": []
}
EOF
[user@cn1234 ~]$ halfpipe \ --workdir ${SLURM_JOB_ID} \ --spec-path /data/${USER}/HALFPIPE_TEST/config/spec.json \ --skip-spec-ui \ --nipype-n-procs 32 \ --nipype-omp-nthreads 16 \ --nipype-memory-gb 30 \ --keep none \ --verbose [2026-08-26 15:59:19,0893] [halfpipe ] [INFO ] HALFpipe version 1.3.2 [2026-08-26 15:59:19,0893] [halfpipe ] [INFO ] Loading existing spec [2026-08-26 15:59:19,0895] [halfpipe ] [INFO ] Stage: workflow [2026-08-26 15:59:19,0895] [halfpipe ] [INFO ] config.nipype.omp_nthreads=16 (command line arguments) [2026-08-26 15:59:35,0819] [halfpipe ] [INFO ] Loading spec file "HALFPIPE_TEST/config/spec.json" [2026-08-26 15:59:35,0823] [halfpipe ] [INFO ] Initializing file database [2026-08-26 15:59:35,0832] [halfpipe ] [INFO ] Initializing new workflow 8a53cdc8 [2026-08-26 15:59:37,0315] [nipype.workflow ] [IMPORTANT] Running fMRIPrep version 25.2.3 │ License NOTICE ################################################## │ fMRIPrep 25.2.3 │ Copyright The NiPreps Developers. │ This product includes software developed by │ the NiPreps Community (https://nipreps.org/). │ Portions of this software were developed at the Department of │ Psychology at Stanford University, Stanford, CA, US. │ This software is also distributed as a Docker container image. │ The bootstrapping file for the image ("Dockerfile") is licensed │ under the MIT License. │ This software may be distributed through an add-on package called │ "Docker Wrapper" that is under the BSD 3-clause License. └─ ################################################################# [...] [user@cn1234 ~]$ exit

Batch job
Most jobs should be run as batch jobs.

Create a batch input file (e.g. halfpipe.sh). For example:

#!/bin/bash
#SBATCH --job-name=halfpipe
#SBATCH --cpus-per-task=32
#SBATCH --mem=32g
#SBATCH --time=24:00:00
#SBATCH --gres=lscratch:100
module load halfpipe

cd /data/$USER/HALFPIPE_TEST

halfpipe \
    --workdir /lscratch/${SLURM_JOB_ID} \
    --spec-path config/spec.json \
    --skip-spec-ui \
    --nipype-n-procs 32 \
    --nipype-omp-nthreads 16 \
    --nipype-memory-gb 30 \
    --keep none \
    --verbose

Submit this job using the Slurm sbatch command.

sbatch halfpipe.sh
Swarm of Jobs
A swarm of jobs is an easy way to submit a set of independent commands requiring identical resources.

Create a swarmfile (e.g. halfpipe.swarm). For example:

halfpipe \
    --workdir /lscratch/${SLURM_JOB_ID} \
    --spec-path /data/${USER}/HALFPIPE_TEST/config/spec.json \
    --skip-spec-ui \
    --subject-include 01 \
    --nipype-n-procs 32 \
    --nipype-omp-nthreads 16 \
    --nipype-memory-gb 30 \
    --keep none \
halfpipe \
    --workdir /lscratch/${SLURM_JOB_ID} \
    --spec-path /data/${USER}/HALFPIPE_TEST/config/spec.json \
    --skip-spec-ui \
    --subject-include 02 \
    --nipype-n-procs 32 \
    --nipype-omp-nthreads 16 \
    --nipype-memory-gb 30 \
    --keep none \
halfpipe \
    --workdir /lscratch/${SLURM_JOB_ID} \
    --spec-path /data/${USER}/HALFPIPE_TEST/config/spec.json \
    --skip-spec-ui \
    --subject-include 03 \
    --nipype-n-procs 32 \
    --nipype-omp-nthreads 16 \
    --nipype-memory-gb 30 \
    --keep none \
    --verbose

Submit this job using the swarm command.

swarm --gres=lscratch:100 -g 32 -t 32 --module halfpipe halfpipe.swarm
where
-gres=lscratch:# Number of Gigabytes of local disk space allocated per process (1 line in the swarm command file)
-g # Number of Gigabytes of memory required for each process (1 line in the swarm command file)
-t # Number of threads/CPUs required for each process (1 line in the swarm command file).
--module halfpipe Loads the halfpipe module for each subjob in the swarm