High-Performance Computing at the NIH
GitHub YouTube @nih_hpc RSS Feed
FSL on Biowulf & Helix

fsl logo FSL is a comprehensive library of image analysis and statistical tools for FMRI, MRI and DTI brain imaging data. FSL is written mainly by members of the Analysis Group, FMRIB, Oxford, UK. FSL website.

FSL can analyze the following:
Functional MRI: FEAT, MELODIC, FABBER, BASIL, VERBENA
Structural MRI: BET, FAST, FIRST, FLIRT & FNIRT, FSLVBM, SIENA & SIENAX, fsl_anat
Diffusion MRI: FDT, TBSS, EDDY, TOPUP
GLM / Stats: GLM general advice, Randomise, Cluster, FDR, Dual Regression, Mm, FLOBS
Other: FSLView, Fslutils, Atlases, Atlasquery, SUSAN, FUGUE, MCFLIRT, Miscvis, POSSUM, BayCEST
Detailed Overview of FSL tools

Multiple versions of FSL are maintained on Biowulf and Helix. The easiest way to access them is via the Modules commands. For example:

[susanc@biowulf]$ module avail fsl

----------------------------------- /usr/local/lmod/modulefiles ---------------------------
   fsl/5.0
   
[susanc@biowulf]$ module load fsl

[susanc@biowulf]$ module list

Currently Loaded Modules:
  1) fsl/5.0

Running FSL on Helix

The FSL GUI can be used on Helix. However, Helix should only be used for short jobs that are not computationally or memory-intensive. (< 1 hr, < 4 cores, < 100 GB memory).

Sample session:


helix$ module load fsl

helix$ fsl

You should now see the FSL GUI appear on your desktop as above. Once you are finished using the GUI, please exit your interactive session by typing 'quit'.

You can also start up the GUI for an individual FSL program by typing the name of the program with an uppercase first character. The programs available with GUIs are

ApplyXFM			Bet			ConcatXFM			Fast
Fdt				Feat			Featquery			Flirt
Fmrib_prepare_fieldmap		fsl			fsl_gui				Glm
InvertXFM			Make_flobs		Melodic				Nudge
Possum				Renderhighres		Renderstats			Runtcl
Susan

Running a single FSL batch job on Biowulf

FSL batch jobs are most easily submitted via the built-in fsl_sub command which will automatically parallelize the job. However, you may want to run a series of FSL commands in a batch file. Set up a batch script along the following lines:

#!/bin/bash
# this file is called fsl.bat

module load fsl
cd /data/$user/my_fsl_dir
mcflirt -in /data/user/fmri1 -out mcf1 -mats -plots -refvol 90 -rmsrel -rmsabs; betfunc mcf1 bet1

Submit this job with:

biowulf$  sbatch  fsl.bat
This will submit the job to the default 2 CPUs with 4 GB of memory. If you need additional memory, e.g. 6 GB, you can specify it with the --mem flag. For example:
biowulf$  sbatch  --mem=6g fsl.bat

Running a swarm of FSL batch jobs on Biowulf

Below is a sample swarm command file which runs mcflirt and bedfunc in succession on each image.

Example FSL runs courtesy of Adam Thomas, NIMH
# this file is called myswarmfile
mcflirt -in /data/user/fmri1 -out mcf1 -mats -plots -refvol 90 -rmsrel -rmsabs; betfunc mcf1 bet1
mcflirt -in /data/user/fmri2 -out mcf2 -mats -plots -refvol 90 -rmsrel -rmsabs; betfunc mcf2 bet2
mcflirt -in /data/user/fmri3 -out mcf3 -mats -plots -refvol 90 -rmsrel -rmsabs; betfunc mcf3 bet3
...

Submit this job with:

swarm -f myswarmfile --module fsl

If each process requires more than the default 4 GB RAM, submit with

swarm -g # -f myswarmfile --module fsl
where '#' is the number of Gigabytes required by a single process (1 line in your swarm command file above).

Parallel FSL programs

All parallelization in FSL v4.0 and up is done via the fsl_sub command that is built into several tools. fsl_sub has been modified to use the Biowulf swarm utility.

The following programs in FSL can use parallelization: FEAT, MELODIC, TBSS, BEDPOSTX, FSLVBM, POSSUM. See the FSL website for more information.

Some FSL applications do not currently parallelize correctly. One example is FEAT. If your FEAT jobs are not producing the correct output files, you can choose to run in serial mode. Set the environment variable NOBATCH in your script as follows:

export NOBATCH=true

before the FEAT line in your batch scripts. The FEAT parts will then run in sucession without using swarm.

Sample session running bedpostx in parallel:

[susanc@biowulf]$  module load fsl

[susanc@biowulf]$ bedpostx sampledataset
subjectdir is /data/susanc/bedpost/sampledataset
Making bedpostx directory structure
Queuing preprocessing stages
Input args=-T 60 -m as -N bpx_preproc -l /data/susanc/bedpost/sampledataset.bedpostX/logs /usr/local/apps/fsl/5.0/fsl/bin/bedpostx_preproc.sh /data/susanc/bedpost/sampledataset 0

Queuing parallel processing stage

----- Bedpostx Monitor -----
Input args=-j 12050 -l /data/susanc/bedpost/sampledataset.bedpostX/logs -M susanc@helix.nih.gov -N bedpostx -t /data/susanc/bedpost/sampledataset.bedpostX/commands.txt

Queuing post processing stage
Input args=-j 12051 -T 60 -m as -N bpx_postproc -l /data/susanc/bedpost/sampledataset.bedpostX/logs /usr/local/apps/fsl/5.0/fsl/bin/bedpostx_postproc.sh /data/susanc/bedpost/sampledataset

[susanc@biowulf]$ 

The job is submitted in 3 parts: pre-processing, bedpost, and post-processing. Each part is dependent on the completion of the previous part, and will only run after the previous part is completed.

If you use 'sjobs' or some variant of 'squeue' to monitor your jobs, you will see at first:

[susanc@biowulf ~]$ sjobs
User    JobId       JobName    Part  St  Reason      Runtime   Wallt  CPUs  Memory    Dependency          Nodelist
==================================================================================================================
susanc 7264_0      bpx_preproc  norm  R    ---         0:04   2:00:00      2  4GB/node                    cn1824
susanc 7265_[0-71] bedpostx     norm  PD   ---         0:05   2:00:00      2  4GB/node  afterany:7264_* 
susanc 7266_[0]    bpx_postproc norm  PD  Dependency   0:00   2:00:00      1  4GB/node  afterany:7265_*

ie. 3 jobs with dependencies for the 2nd and 3rd jobs. Once the pre-processing (job 7264in this example) is over, the main bedpost jobs (job 7265) will run, and you will see something like this:

[susanc@biowulf ~]$ sjobs
JOBID            NAME      TIME        ST      CPUS  MIN_ME   NODE   DEPENDENCY       NODELIST(REASON)
7264_0         bedpost     0:07        R       1     4G       1                          p1718
7264_1         bedpost     0:07        R       1     4G       1                          p1719
7264_2         bedpost     0:07        R       2     4G       1                          p999
7264_3         bedpost     0:07        R       2     4G       1                          p999
...etc...
7264_[33-71]   bedpost      0:00       PD      1     4G       1                         (QOSMaxCpusPerUserLimit)
7264_[0]       bpx_postproc 0:00       PD      1     4G       1      afterany:9517_*    (Dependency)
and once those are completed, the post-processing step (job 9518) will run.

FSL parallel jobs and memory

By default, all parallel FSL jobs submitted through fsl_sub will be submitted requesting 4 GB memory. In some cases, this may not be enough memory for the job. In those cases, the user can set an environment variable, FSL_MEM, which will set the memory required for the jobs. For example, if a bedpost run as above failed due to lack of memory, you could run as follows:
[susanc@biowulf]$ module load fsl

[susanc@biowulf]$ export FSL_MEM=16

[susanc@biowulf]$ bedpostx sampledataset
[...]

[susanc@biowulf]$ sjobs
JOBID            NAME      TIME        ST      CPUS  MIN_ME   NODE   DEPENDENCY       NODELIST(REASON)
7264_0         bedpost     0:07        R       1     16G       1                          p1718
7264_1         bedpost     0:07        R       1     16G       1                          p1719
7264_2         bedpost     0:07        R       2     16G       1                          p999
7264_3         bedpost     0:07        R       2     16G       1                          p999
...etc...
7264_[33-71]   bedpost      0:00       PD      1     16G       1                         (QOSMaxCpusPerUserLimit)
7264_[0]       bpx_postproc 0:00       PD      1     16G       1      afterany:9517_*    (Dependency)
In the example above, the value of FSL_MEM was set to 16 (GB), and therefore the jobs were submitted requesting 16 GB of memory.

Running an interactive job on Biowulf

To run an interactive FSL job on Biowulf,2 you simply need to use the sinteractive command to allocate an interactive core (or more than 1 core).

[susanc@biowulf]$ sinteractive --mem=5g
salloc.exe: Granted job allocation 9777
slurm stepprolog here!

[susanc@p21]$ module load fsl
[susanc@p21]$ ..... any fsl command ... 
[susanc@p21]$ exit

To use the FSL GUI on an interactive node, you will need sinteractive -x. Once you are logged on to the allocated core, load the FSL module and type 'fsl' to start the FSL GUI, as in the example on Helix above.

FSL FIX
FIX is FMRIB's ICA-based Xnoiseifier. FIX attempts to auto-classify ICA components into "good" vs "bad" components, so that the bad components can be removed from the 4D FMRI data. FIX is intended to be run on single-session MELODIC ICA output. See the FSL-FIX site for more information.

Note that FIX is a beta package.

To use FIX on Helix/Biowulf, type 'module load fsl/fix1.06' and then the appropriate fix commands. Example batch script:

#!/bin/bash

cd /data/$USER/mydir

module load fsl/fix1.06
fix -c mel.ica training.RData thresh

Documentation