Biowulf High Performance Computing at the NIH
megalodon on Biowulf

Megalodon is a research command line tool to extract high accuracy modified base and sequence variant calls from raw nanopore reads by anchoring the information rich basecalling neural network output to a reference genome/transcriptome.

References:

Documentation
Important Notes

Interactive job
Interactive jobs should be used for debugging, graphics, or applications that cannot be run as batch jobs.

Allocate an interactive session and run the program. The data in this example do not produce useful results but are for illustrative purposes only.
Sample session (user input in bold):

[user@biowulf ~]$ sinteractive -c20 --mem=32g --gres=lscratch:10,gpu:p100:2
salloc.exe: Pending job allocation 5091510
salloc.exe: job 5091510 queued and waiting for resources
salloc.exe: job 5091510 has been allocated resources
salloc.exe: Granted job allocation 5091510
salloc.exe: Waiting for resource configuration
salloc.exe: Nodes cn2369 are ready for job
srun: error: x11: no local DISPLAY defined, skipping

[user@cn2369 ~]$ cd /lscratch/$SLURM_JOB_ID

[user@cn2369 5091510]$ module load megalodon
[+] Loading megalodon  2.2.9  on cn2369
[+] Loading singularity  3.7.0  on cn2369

[user@cn2369 5091510]$ cp -r /usr/local/apps/megalodon/2.2.9/example/ .

[user@cn2369 5091510]$ cd example/

[user@cn2369 example]$ megalodon fast5/ \
     --guppy-server-path ${MEGALODON_GUPPY_PATH} \
     --outputs basecalls mappings mod_mappings mods \
     --output-directory mega_out \
     --reference /fdb/igenomes/Homo_sapiens/UCSC/hg38/Sequence/WholeGenomeFasta/genome.fa \
     --devices ${CUDA_VISIBLE_DEVICES} \
     --processes ${SLURM_CPUS_PER_TASK} \
     --overwrite \
     --guppy-config res_dna_r941_min_modbases_5mC_CpG_v001.cfg
[13:28:28] Running Megalodon version 2.2.9
[13:28:28] Loading guppy basecalling backend
[2020-12-29 13:28:35.358424] [0x00002aaaaaad8e00] [info]    Connecting to server as
[13:28:35] Loading reference
******************** WARNING: "mods" output requested, so "per_read_mods" will be added to outputs. ********************
[13:30:29] Using canonical alphabet ACGT and modified bases m=5mC (alt to C)
[13:30:29] Preparing workers to process reads
[13:30:31] Processing reads
Full output or empty input queues indicate I/O bottleneck
3 most common unsuccessful processing stages:
    -----                                                                                           [2020-12-29 13:30:32.044364] [0x00002aaaaaad8e00] [info]    Connecting to server as
[2020-12-29 13:30:32.334411] [0x00002aaaaaad8e00] [info]    Connecting to server as
[2020-12-29 13:30:32.615676] [0x00002aaaaaad8e00] [info]    Connecting to server as
[2020-12-29 13:30:32.896420] [0x00002aaaaaad8e00] [info]    Connecting to server as         | 0/10000
[...snip]
Read Processing: 100%|███████████████████████████| 5/5 [00:06<00:00,  1.39s/reads, samples/s=3.94e+4]
 input queue capacity extract_signal      :   0%|                                           | 1/10000
output queue capacity basecalls           :   0%|                                           | 0/10000
output queue capacity mappings            :   0%|                                           | 0/10000
output queue capacity per_read_mods       :   0%|                                           | 0/10000
[13:30:38] Unsuccessful processing types:
   100.0% (      5 reads) : No alignment                                                             
[13:30:39] Spawning modified base aggregation processes
******************** WARNING: No per-read variants or modified base statistics found for aggregation. ********************
[13:30:44] Mega Done

[user@cn2369 example]$ exit
exit
salloc.exe: Relinquishing job allocation 5091510

[user@biowulf ~]$ 

Batch job
Most jobs should be run as batch jobs.

Create a batch input file (e.g. megalodon.sh). For example:

#!/bin/bash
set -e
module load megalodon
megalodon fast5/ \
     --guppy-server-path ${MEGALODON_GUPPY_PATH} \
     --outputs basecalls mappings mod_mappings mods \
     --output-directory mega_out \
     --reference /fdb/igenomes/Homo_sapiens/UCSC/hg38/Sequence/WholeGenomeFasta/genome.fa \
     --devices ${CUDA_VISIBLE_DEVICES} \
     --processes ${SLURM_CPUS_PER_TASK} \
     --overwrite

Submit this job using the Slurm sbatch command.

sbatch [--cpus-per-task=#] [--mem=#] [--gres=cpu:#] megalodon.sh
Swarm of Jobs
A swarm of jobs is an easy way to submit a set of independent commands requiring identical resources.

Create a swarmfile (e.g. megalodon.swarm). For example:

megalodon dir1/ \
     --guppy-server-path ${MEGALODON_GUPPY_PATH} \
     --outputs basecalls mappings mod_mappings mods \
     --output-directory out1 \
     --reference /fdb/igenomes/Homo_sapiens/UCSC/hg38/Sequence/WholeGenomeFasta/genome.fa \
     --devices ${CUDA_VISIBLE_DEVICES} \
     --processes ${SLURM_CPUS_PER_TASK}
megalodon dir2/ \
     --guppy-server-path ${MEGALODON_GUPPY_PATH} \
     --outputs basecalls mappings mod_mappings mods \
     --output-directory out2 \
     --reference /fdb/igenomes/Homo_sapiens/UCSC/hg38/Sequence/WholeGenomeFasta/genome.fa \
     --devices ${CUDA_VISIBLE_DEVICES} \
     --processes ${SLURM_CPUS_PER_TASK}
megalodon dir3/ \
     --guppy-server-path ${MEGALODON_GUPPY_PATH} \
     --outputs basecalls mappings mod_mappings mods \
     --output-directory out3 \
     --reference /fdb/igenomes/Homo_sapiens/UCSC/hg38/Sequence/WholeGenomeFasta/genome.fa \
     --devices ${CUDA_VISIBLE_DEVICES} \
     --processes ${SLURM_CPUS_PER_TASK}

Submit this job using the swarm command.

swarm -f megalodon.swarm [-g #] [-t #] [--gres <gpu>:#] --module megalodon
where
-g # Number of Gigabytes of memory required for each process (1 line in the swarm command file)
-t # Number of threads/CPUs required for each process (1 line in the swarm command file).
--gres # GPU type and number to use for each subjob.
--module megalodon Loads the megalodon module for each subjob in the swarm