Biowulf High Performance Computing at the NIH
JULIA on Biowulf
julia logo

Julia is a flexible dynamic language for scientific and numerical computing. Julia's performance is comparable to traditional statically-typed languages.

Julia has a high level syntax and has good support for interactive use. Julia is provided under the MIT license (free) and the source code is publically available on Github.

Documentation
Notes

Interactive job
Interactive jobs should be used for debugging, graphics, or applications that cannot be run as batch jobs.

Allocate an interactive session and run the program.
Sample session (user input in bold):

[user@biowulf]$ sinteractive
salloc.exe: Pending job allocation 46116226
salloc.exe: job 46116226 queued and waiting for resources
salloc.exe: job 46116226 has been allocated resources
salloc.exe: Granted job allocation 46116226
salloc.exe: Waiting for resource configuration
salloc.exe: Nodes cn3144 are ready for job

There may be multiple versions of julia available. An easy way of selecting the version is to use modules.To see the modules available, type

[user@cn3144 ~]$ module avail julia
-------------------- /usr/local/lmod/modulefiles -----------------------------------------------
   julialang/0.6.2    julialang/1.0.0    julialang/1.1.0 (L,D)

  Where:
   L:  Module is loaded
   D:  Default Module

Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

You can now load the module that corresponds to the desired Julia version and start an interactive Julia session as follows:

[user@cn3144 ~]$ module load julialang/1.1.0
[+] Loading git 2.23.0  ... 
[+] Loading julialang  1.1.0 

[user@cn3144 ~]$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.1.0 (2019-01-21)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> 5 / 2
2.5

julia> exit()

[user@cn3144 ~]$ exit
salloc.exe: Relinquishing job allocation 46116226
[user@biowulf ~]$
Package management

To install, update, and remove packages you can use Julia's package manager, Pkg. Pkg comes with a REPL shell that you can start from within Julia by typing ]. To exit the REPL enter ^C or backspace. Within the REPL you can obtain a list of available commands by typing ?:

julia> ]
(v1.1) pkg> ?
  Welcome to the Pkg REPL-mode. To return to the julia> prompt, either press backspace when the input line is
  empty or press Ctrl+C.

  Synopsis

  pkg> cmd [opts] [args]

  Multiple commands can be given on the same line by interleaving a ; between the commands.

  Commands

  activate: set the primary environment the package manager manipulates
  add: add packages to project
  build: run the build script for packages
  develop: clone the full package repo locally for development
  free: undoes a pin, develop, or stops tracking a repo
  gc: garbage collect packages not used for a significant time
  generate: generate files for a new project
  help: show this message
  instantiate: downloads all the dependencies for the project
  pin: pins the version of packages
  precompile: precompile all the project dependencies
  preview: previews a subsequent command without affecting the current state
  registry-add: add package registries
  registry-remove: remove package registries
  registry-status: information about installed registries
  registry-update: update package registries
  remove: remove packages from project or manifest
  resolve: resolves to update the manifest from changes in dependencies of developed packages
  status: summarize contents of and changes to environment
  test: run tests for packages
  update: update packages in manifest

To obtain a list of installed packages, for example, type status:

(v1.1) pkg> status
    Status `~/.julia/environments/v1.1/Project.toml`
  [6e4b80f9] BenchmarkTools v0.4.3
  [8f4d0f93] Conda v1.3.0
  [c91e804a] Gadfly v1.1.0
  [7073ff75] IJulia v1.19.0
  [6218d12a] ImageMagick v0.7.5
  [916415d5] Images v0.19.2
  [c601a237] Interact v0.10.3
  [47be7bcc] ORCA v0.3.0
  [f0f68f2c] PlotlyJS v0.13.0
  [91a5bcdd] Plots v0.26.1
  [438e738f] PyCall v1.91.2
  [d330b81b] PyPlot v2.8.1
  [0f1e0344] WebIO v0.8.11

To install a package use add, for example:

(v1.1) pkg> add Flux
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
 Installed TimerOutputs ─────── v0.5.3
 Installed CodecZlib ────────── v0.6.0
 Installed ZipFile ──────────── v0.8.3
 Installed Tracker ──────────── v0.2.6
 Installed GPUArrays ────────── v1.0.4
 Installed NNlib ────────────── v0.6.1
 Installed CUDAdrv ──────────── v3.1.0
[...]
  [0c68f7d7] + GPUArrays v1.0.4
  [929cbde3] + LLVM v1.3.2
  [872c559c] + NNlib v0.6.1
  [a759f4b9] + TimerOutputs v0.5.3
  [9f7883ad] + Tracker v0.2.6
  [3bb67fe8] + TranscodingStreams v0.9.5
  [a5390f91] + ZipFile v0.8.3
  Building CodecZlib → `~/.julia/packages/CodecZlib/5t9zO/deps/build.log`
  Building NNlib ────→ `~/.julia/packages/NNlib/Nksco/deps/build.log`
  Building ZipFile ──→ `~/.julia/packages/ZipFile/oD4uG/deps/build.log`

To update your packages use up:

(v1.1) pkg> up
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
 Installed PyPlot ────────────── v2.8.2
 Installed ORCA ──────────────── v0.3.1
 Installed IJulia ────────────── v1.20.2
 Installed DocStringExtensions ─ v0.8.1
 Installed Plots ─────────────── v0.28.4
 Installed Flux ──────────────── v0.10.0
 Installed Gadfly ────────────── v1.0.1
[...]
  [d330b81b] ↑ PyPlot v2.8.1 ⇒ v2.8.2
  [0796e94c] - Tokenize v0.5.7
  [9f7883ad] - Tracker v0.2.6
  [0f1e0344] ↑ WebIO v0.8.11 ⇒ v0.8.13
  [e88e6eb3] + Zygote v0.4.3
  [700de1a5] + ZygoteRules v0.2.0
  Building IJulia → `~/.julia/packages/IJulia/F1GUo/deps/build.log`
  Building Plots ─→ `~/.julia/packages/Plots/qZHsp/deps/build.log`
  Building WebIO ─→ `~/.julia/packages/WebIO/2nnB1/deps/build.log`
  Building ORCA ──→ `~/.julia/packages/ORCA/fiEmb/deps/build.log`

You can also use Pkg's REPL to create new environments and organize dependencies for each environment. For example the following creates a new environment, installs the Plots dependency in that environment, then switches back to the v1.1 environment:

(v1.1) pkg> activate ~/.julia/environments/new_environment/
[ Info: activating new environment at ~/.julia/environments/new_environment.

(new_environment) pkg> status
    Status `~/.julia/environments/new_environment/Project.toml`
  (empty environment)

(new_environment) pkg> add Plots
 Resolving package versions...
 Installed Requires ─── v1.0.0
 Installed ColorTypes ─ v0.9.0
 Installed Colors ───── v0.11.0
  Updating `~/.julia/environments/new_environment/Project.toml`
  [91a5bcdd] + Plots v0.28.4
  Updating `~/.julia/environments/new_environment/Project.toml`
  [b99e7846] + BinaryProvider v0.5.8
  [3da002f7] + ColorTypes v0.9.0
  [5ae59095] + Colors v0.11.0
[...]
  [3fa0cd96] + REPL 
  [9a3f8284] + Random 
  [ea8e919c] + SHA 
  [9e88b42a] + Serialization 
  [6462fe0b] + Sockets 
  [2f01184e] + SparseArrays 
  [10745b16] + Statistics 
  [8dfed614] + Test 
  [cf7118a7] + UUIDs 
  [4ec0a83e] + Unicode 

(new_environment) pkg> status
    Status `~/.julia/environments/new_environment/Project.toml`
  [91a5bcdd] Plots v0.28.4

(new_environment) pkg> activate ~/.julia/environments/v1.1/

(v1.1) pkg> 

For the full documentation of Pkg's REPL, check the Pkg docs. There is a searchable list of registered Julia packages at Julia packages.

Julia on Jupyter

You can have access to a Julia kernel through Jupyter by installing the IJulia package in your home directory and setting up the appropriate tunneling (as described in Jupyter on Biowulf). To install the IJulia package do the following on a Julia interactive session first start the pkg mode with ]

(v1.3) pkg> add IJulia
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
 Installed Rmath_jll  v0.2.2+0
 Installed Gettext_jll  v0.20.1+1
 [...]
(v1.3) pkg>

In addition to installing the IJulia package, this will also install a jupyter kernel in your home directory under ~/.local/share/jupyter/kernels which will then be included in the kernels menu of jupyter notbook and jupyter lab.

Batch job
Most jobs should be run as batch jobs.

Create a batch input file (e.g. julia.sh). For example:

#!/bin/bash
set -e
module load julialang
myscript.jl < data.in > result.out

Submit this job using the Slurm sbatch command.

sbatch [--cpus-per-task=#] [--mem=#] julia.sh
Swarm of Jobs
A swarm of jobs is an easy way to submit a set of independent commands requiring identical resources.

Create a swarmfile (e.g. julia.swarm). For example:

myscript.jl < dataset1.in > result1.out
myscript.jl < dataset2.in > result2.out
myscript.jl < dataset3.in > result3.out
myscript.jl < dataset4.in > result4.out

Submit this job using the swarm command.

swarm -f julia.swarm [-g #] [-t #] --module julialang
where
-g # Number of Gigabytes of memory required for each process (1 line in the swarm command file)
-t # Number of threads/CPUs required for each process (1 line in the swarm command file).
--module julialang Loads the julialang module for each subjob in the swarm