High-Performance Computing at the NIH
GitHub YouTube @nih_hpc RSS Feed
AFNI on Biowulf & Helix

AFNI (Analysis of Functional NeuroImages) is a set of C programs for processing, analyzing, and displaying functional MRI (FMRI) data - a technique for mapping human brain activity. AFNI is developed by the Scientific and Statistical Computing Core, NIMH. [AFNI website]

AFNI on Biowulf is installed in /usr/local/apps/afni. Users should set up their environment for AFNI using the 'module' commands as below:

[user@biowulf ~]$ module avail afni
------------------------------------- /usr/local/lmod/modulefiles ----------------------------
   afni/current-openmp  (D)  afni/Dec-2016    afni/Jun-2016    afni/Mar-2017    afni/Sep-2016

  Where:
   (D):  Default Module

Typing 'module load afni' without a version will load the default (latest) version. AFNI binaries on Biowulf are updated once a day at 4 am, so loading the 'current' or 'current-openmp' version will get you the latest binaries. Loading the module will give you the timestamp of the last update, e.g.

[user@biowulf ~]$ module load afni/current-openmp
[+] Loading AFNI current-openmp ...
AFNI/current-openmp last updated  2017-05-22

A copy of AFNI is retained every 3 months, for users who want to complete a run using the same version of the executables. These 3-month snapshots are stored for a year. In the example above, a snapshot of AFNI from June 2016, Sep 2016, Dec 2016 and Mar 2017 is available on the system.

Running AFNI on Helix

It is possible to run AFNI on Helix, but since most users will be running large numbers of AFNI jobs, it is best to run on Biowulf.

To run a test or interactive job on Helix, simply type

helix% module load afni

helix% ...any AFNI command...
Running a single AFNI batch job on Biowulf

Set up a batch script along the following lines:

#!/bin/bash
# this file is called myjob.bat

cd /data/$USER/my_afni_dir
module load afni

....any AFNI command...

Submit this job with:

sbatch  myjob.bat
Most AFNI programs are single-threaded. The 'sbatch' command above will allocate you 2 CPUs and 4 GB of memory. If your job requires more than 4 GB of memory, submit with:
sbatch --mem=#g  myjob.bat
where '#' is the number of GB of memory required. (e.g. '--mem=4g')

If you are using the 'current-openmp' version of AFNI, that version can multi-thread. In that case, you will want to specify the number of threads and CPUs desired with

sbatch --cpus-per-task=#  --mem=#g  myjob.bat

Running a swarm of AFNI batch jobs on Biowulf

Typically, AFNI on Biowulf is used to simultaneously process a large number of independent datasets via the swarm utility.

A small sample set of data for 3 subjects can be copied from /usr/local/apps/afni/AFNI_data2_helix.tar.gz, courtesy of Rick Reynolds (NIMH). This tar file includes a script called proc.subj to process data for a single subject.

You can unpack this dataset into your own area with
cd /data/$USER
tar xvzf /usr/local/apps/afni/AFNI_data2_helix.tar.gz

A swarm command file to process these 3 datasets would look like

#-------- this file is called swarm.cmd -----------------
cd /data/$USER/AFNI_data2_helix; tcsh ./proc.subj ED
cd /data/$USER/AFNI_data2_helix; tcsh ./proc.subj EE
cd /data/$USER/AFNI_data2_helix; tcsh ./proc.subj EF
#--------------------------------------------------------

This swarm command file would be submitted to the batch system with:

swarm -f swarm.cmd --module afni

If each individual process requires more than 4 GB of RAM, you can specify the required memory with

swarm -g # -f swarm.cmd --module afni
where # is the number of GB required. See the swarm webpage for more information, or contact the Biowulf staff at staff@hpc.nih.gov

Running an interactive job on Biowulf

Allocate an interactive node with 'sinteractive', load the module and then run the desired AFNI command. Sample session.

[user@biowulf ~]$ sinteractive
salloc.exe: Pending job allocation 41583225
salloc.exe: job 41583225 queued and waiting for resources
salloc.exe: job 41583225 has been allocated resources
salloc.exe: Granted job allocation 41583225
salloc.exe: Waiting for resource configuration
salloc.exe: Nodes cn3102 are ready for job

[user@cn3102 ~]$ module load afni
[+] Loading AFNI current-openmp ...
AFNI/current-openmp last updated  2017-05-22

[user@cn3102 ~]$ cd /data/user/afni/AFNI_data2_helix/

[user@cn3102 AFNI_data2_helix]$ time tcsh proc.subj ED
(version 1.24, Jun 4, 2007)
++ 3dcopy: AFNI version=AFNI_2011_12_21_1014 (Jan  8 2015) [64-bit]
++ 3dTcat: AFNI version=AFNI_2011_12_21_1014 (Jan  8 2015) [64-bit]
++ 3dTcat: AFNI version=AFNI_2011_12_21_1014 (Jan  8 2015) [64-bit]
++ 3dTcat: AFNI version=AFNI_2011_12_21_1014 (Jan  8 2015) [64-bit]
++ 3dTcat: AFNI version=AFNI_2011_12_21_1014 (Jan  8 2015) [64-bit]

[....etc...]
[user@cn3102 AFNI_data2_helix]$ exit

[user@biowulf ~]$

Documentation

AFNI documentation at the AFNI website.