The 'Introduction to Biowulf' class is now an online, self-paced class. New Biowulf users are encouraged to work through the entire class, and experienced Biowulf users can view specific videos to brush up on a particular section.

How To: The general process is to watch the relevant video, then work through the associated hands-on section. The References point to the documentation associated with that section.

Questions? Users can email the HPC staff (staff@hpc.nih.gov) to get help, or come by one of the monthly Walk-In Consults (now virtual)

A familiarity with Linux and the Bash Shell is vital for using the Biowulf cluster. Submitting, monitoring, and managing jobs on the cluster largely involves command-line operations. To test your Linux/Bash familiarity, take the quiz below. If you score below 8/10, we highly recommend that you work your way through one or more of the Linux tutorials

   Take the Linux/Bash Quiz! (10 random multiple-choice questions)

Already familiar with Biowulf? Test your knowledge with the Biowulf quiz. If you get less than 8/10, you should brush up on the appropriate sections in the online class below.

   Take the Biowulf-specific quiz! (10 random multiple-choice questions)

• Introduction to the class. Overview of NIH HPC Core Facility. When to use a cluster/Biowulf. Navigating the HPC website.
      Video (14 mins)

• What is a cluster? Cluster basics: nodes, cores, CPUs and hyperthreading
      Video (7 mins)

• Biowulf architecture, compute nodes.
      Video (5 mins)

• Accounts, and getting started.
     Applying for an account
      Video (5 mins) -- Biowulf Accounts and the Login Node.
      How to Connect from a Mac, Windows or Linux system.
      Hands-On: Connect to the Biowulf login node.

• Environment modules, how to access applications on the system.
      Video (13 mins)
      Hands-On: Using environment modules
      References: Environment Modules on Biowulf

• Data storage, checking your quota, snapshots .
      Video (24 mins)
      Hands-On: Disk space, checkquota, snapshots
      References: Storage overview, Backups, Snapshots.

• Data transfer, Globus
      Video: Data transfer to/from Biowulf (9 mins)
      Video: Data transfer and sharing with Globus (8 mins)
      Hands-On: take a look at the data transfer page and figure out which data transfer options will work for you. Try Globus.
      References: Transferring data to/from the HPC systems, Globus

• Introduction to the Slurm batch system, processes and threads, binding
      Video (5 mins)

• Submitting a simple serial job, deleting a job. Output and error files. Using modules in batch scripts.
      Video (14 mins)
      Hands-On: submit a job with sbatch
      References: Job Submission in the Biowulf User Guide

• How to submit a multi-threaded job.
      Video (11 mins)
      Hands-On: submit a multi-threaded job
      References: Job Submission in the Biowulf User Guide

• Interactive jobs. How to run GUIs on Biowulf.
      Video (11 mins)
      Hands-On: start an interactive session and run a process
      References: Interactive Jobs in the Biowulf User Guide

• Introduction to the Swarm utility. Swarms of single- and multi-threaded jobs. Using modules in swarm jobs.
      Video (15 mins)
      References: Swarm documentation
      Hands-On: submit a swarm of jobs

• Additional swarm options. Sbatch vs swarm for job submission.
      Video (9 mins)
     References: Swarm documentation

• Swarm bundling, the --devel option, and walltimes for bundled swarms.
      Video (7 mins)
      Hands-On: submit a bundled swarm job
     References: Swarm documentation

• Swarm Demo Video (15 mins) -- a demo showing submission of swarm jobs, how to test swarm parameters, and the effects of bundling a swarm.

• Slurm Resources, Partitions, and Scheduling.
      Video (14 mins)
      Hands-On: freen and batchlim
     References: Biowulf utilities

• Walltimes.
      Video (-4 mins)
      Thought exercises: walltimes
     References: Swarm documentation

• How to utilize the local disk on a compute node
      Video (10 mins)
      Hands-On: interactive job using local disk
      References: Using local disk in the Biowulf User Guide

• Allocating GPUs
      Video (7 mins)
      References: Allocating GPUs, Partitions in the Biowulf User Guide.

• Job monitoring: jobload, the user dashboard, and jobhist
      Video (21 mins)
      Hands-On: Submit a job, use jobload and the dashboard to monitor it, and jobhist after it completes
      References: Monitoring jobs, Biowulf utilities in the Biowulf User Guide

• Job dependencies
      Video (9 mins)
      Hands-On: submit a series of jobs with dependencies
      References: Job dependencies in the Biowulf User Guide, Building pipelines using slurm dependencies

• Molecular Dynamics jobs.
      Video (27 mins)
      References: Making efficient use of Biowulf's multinode partition.

• Relion tips and tricks.

• Pulling it all together: Working through converting a serial job into a swarm. How to decide on the resources to allocate for the jobs, submitting and monitoring the jobs.

• Tidbits:
  • The largemem and unlimited partitions.
  • Why are my jobs pending?
  • Licensed applications
  • Environment variables set by Slurm

• Advanced Quiz -- test your understanding of sbatch and swarm.

• Help beyond the class topics: The HPC staff includes scientists and system administrators who respond to user questions. The staff supports 2000+ users in diverse research areas, and thus is not available for individual collaborations, but can help with Biowulf-related questions if you send email to staff@hpc.nih.gov.

  We also host Walk-In Consults around campus once a month, where you can discuss your problems with the staff directly. The Walk-In Schedule is emailed to users and is on the calendar.

• Additional training videos on more advanced topics, such as Jupyter notebooks, Matlab, tensorflow.

Additional topics? Anything insufficiently covered? Please send any comments regarding this online class to staff@hpc.nih.gov