SpinWalk on Biowulf

SpinWalk lets researchers simulate how virtual MRI spins move through digital tissue structures such as cells, axons, and blood-vessel networks. It uses Monte Carlo random walks to model diffusion and MR signal behavior, including relaxation, magnetic-field variations, permeability, RF pulses, and gradient waveforms. Simulations can run on CPUs or NVIDIA GPUs, and the results are saved in HDF5 format for analysis with tools such as MATLAB or Python.

Documentation
Important Notes

Interactive job
Interactive jobs should be used for debugging, graphics, or applications that cannot be run as batch jobs.

Allocate an interactive session and run the program.
Sample session (user input in bold):

[user@biowulf]$ interactive --mem=8g --cpus-per-task=8 --gres=gpu:a100:1
salloc.exe: Pending job allocation 46116226
salloc.exe: job 46116226 queued and waiting for resources
salloc.exe: job 46116226 has been allocated resources
salloc.exe: Granted job allocation 46116226
salloc.exe: Waiting for resource configuration
salloc.exe: Nodes cn3144 are ready for job

[user@cn3144 ~]$ module load spinwalk
[+] Loading spinwalk  1.15.0  on cn3144 
[+] Loading gcc  8.5.0  ... 
[+] Loading boost libraries 1.91  ... 
[+] Loading HDF5  1.12.2 

[user@cn3144 ~]$ mkdir -pv /data/$USER/SPINWALK_TEST
[user@cn3144 ~]$ cd /data/$USER/SPINWALK_TEST


[user@cn3144 ~]$ spinwalk --help
Usage: spinwalk [OPTIONS] [SUBCOMMAND]

Options:
  -h,--help                   Print this help message and exit
  -v,--version                Display program version information and exit
  -g,--gpu_info               Print GPU information

Subcommands:
  phantom                     Phantom Generator
  sim                         Simulation

[user@cn3144 ~]$ spinwalk --gpu_info
Driver version: 13000, CUDA version: 12010
Number of devices: 1
Device 1, NVIDIA A100-SXM4-80GB
-Compute Capability: 8.0
-Free GPU Memory: 80725 MB (out of 81151 MB)

[user@cn3144 ~]$ spinwalk sim -c example.ini
 
 ____            _          __        __          _   _        
/ ___|   _ __   (_)  _ __   \ \      / /   __ _  | | | | __    
\___ \  | '_ \  | | | '_ \   \ \ /\ / /   / _` | | | | |/ /    
 ___) | | |_) | | | | | | |   \ V  V /   | (_| | | | |   <     
|____/  | .__/  |_| |_| |_|    \_/\_/     \__,_| |_| |_|\_\    
        |_|                                                    

SpinWalk ver. 1.15.0
Log file location: "/data/user/SPINWALK_TEST/spinwalk_0.log"
Running simulation for 1 config(s)...


Simulation starts...
Loading phantom: r8_0.78_4_90_600_600.h5
[====================================================================================================] 100%                                                                     
Simulation took 0.551 sec., everything else took 2.348 sec.
Saving the results to disk.
Loading phantom: r8_0.85_4_90_600_600.h5
[====================================================================================================] 100%                                                                     
Simulation took 0.547 sec., everything else took 2.954 sec.
Saving the results to disk.

Simulation(s) finished successfully! Total elapsed time = 5.906 second(s).

[user@cn3144 ~]$ exit
salloc.exe: Relinquishing job allocation 46116226
[user@biowulf ~]$

Batch job
Most jobs should be run as batch jobs.

Create a batch input file (e.g. spinwalk.sh). For example:

#!/bin/bash

module load spinwalk

spinwalk sim -c example.ini

Submit this job using the Slurm sbatch command.

sbatch --partition=gpu --cpus-per-task=8 --mem=8g --gres=gpu:a100:1 spinwalk.sh
Swarm of Jobs
A swarm of jobs is an easy way to submit a set of independent commands requiring identical resources.

Create a swarmfile (e.g. spinwalk.swarm). For example:

spinwalk sim -c example_01.ini
spinwalk sim -c example_02.ini
spinwalk sim -c example_03.ini

Submit this job using the swarm command.

swarm --partition=gpu -g 8 -t 8 --gres=gpu:a100:1 --module spinwalk spinwalk.swarm
where
-g # Number of Gigabytes of memory required for each process (1 line in the swarm command file)
-t # Number of threads/CPUs required for each process (1 line in the swarm command file).
--module spinwalk Loads the spinwalk module for each subjob in the swarm