The 'Introduction to Biowulf' class is now an online, self-paced class. New Biowulf users are encouraged to work through the entire class, and experienced Biowulf users can view specific videos to brush up on a particular section.
How To: The general process is to watch the relevant video, then work through the associated hands-on section. The References point to the documentation associated with that section.
Questions? Users can email the HPC staff (staff@hpc.nih.gov) to get help, or come by one of the monthly Walk-In Consults (now virtual)
A familiarity with Linux and the Bash Shell is vital for using the Biowulf cluster. Submitting, monitoring, and managing jobs on the cluster largely involves command-line operations. To test your Linux/Bash familiarity, take the quiz below. If you score below 8/10, we highly recommend that you work your way through one or more of the Linux tutorials
Take the Linux/Bash Quiz! (10 random multiple-choice questions)
Already familiar with Biowulf? Test your knowledge with the Biowulf quiz. If you get less than 8/10, you should brush up on the appropriate sections in the online class below.
Take the Biowulf-specific quiz! (10 random multiple-choice questions)
- Introduction to the class. Overview of NIH HPC Core Facility. When to use a cluster/Biowulf. Navigating the HPC website.
Video (14 mins)
- What is a cluster? Cluster basics: nodes, cores, CPUs and hyperthreading
Video (7 mins)
- Biowulf architecture, compute nodes.
Video (5 mins)
- Accounts, and getting started.
Applying for an account
Video (5 mins) -- Biowulf Accounts and the Login Node.
How to Connect from a Mac, Windows or Linux system.
Hands-On: Connect to the Biowulf login node.
- Environment modules, how to access applications on the system.
Video (13 mins)
Hands-On: Using environment modules
References: Environment Modules on Biowulf - Data storage, checking your quota, snapshots .
Video (24 mins)
Hands-On: Disk space, checkquota, snapshots
References: Storage overview, Backups, Snapshots. - Data transfer, Globus
Video: Data transfer to/from Biowulf (9 mins)
Video: Data transfer and sharing with Globus (8 mins)
Hands-On: take a look at the data transfer page and figure out which data transfer options will work for you. Try Globus.
References: Transferring data to/from the HPC systems, Globus
- Introduction to the Slurm batch system, processes and threads, binding
Video (5 mins)
- Submitting a simple serial job, deleting a job. Output and error files. Using modules in batch scripts.
Video (14 mins)
Hands-On: submit a job with sbatch
References: Job Submission in the Biowulf User Guide - How to submit a multi-threaded job.
Video (11 mins)
Hands-On: submit a multi-threaded job
References: Job Submission in the Biowulf User Guide - Interactive jobs. How to run GUIs on Biowulf.
Video (11 mins)
Hands-On: start an interactive session and run a process
References: Interactive Jobs in the Biowulf User Guide
- Introduction to the Swarm utility. Swarms of single- and multi-threaded jobs. Using modules in swarm jobs.
Video (15 mins)
References: Swarm documentation
Hands-On: submit a swarm of jobs - Additional swarm options. Sbatch vs swarm for job submission.
Video (9 mins)
References: Swarm documentation - Swarm bundling, the --devel option, and walltimes for bundled swarms.
Video (7 mins)
Hands-On: submit a bundled swarm job
References: Swarm documentation - Swarm Demo Video (15 mins) -- a demo showing submission of
swarm jobs, how to test swarm parameters, and the effects of bundling a swarm.
- Slurm Resources, Partitions, and Scheduling.
Video (14 mins)
Hands-On: freen and batchlim
References: Biowulf utilities - Walltimes.
Video (-4 mins)
Thought exercises: walltimes
References: Swarm documentation - How to utilize the local disk on a compute node
Video (10 mins)
Hands-On: interactive job using local disk
References: Using local disk in the Biowulf User Guide - Allocating GPUs
Video (7 mins)
References: Allocating GPUs, Partitions in the Biowulf User Guide.
- Job monitoring: jobload, the user dashboard, and jobhist
Video (21 mins)
Hands-On: Submit a job, use jobload and the dashboard to monitor it, and jobhist after it completes
References: Monitoring jobs, Biowulf utilities in the Biowulf User Guide - Job dependencies
Video (9 mins)
Hands-On: submit a series of jobs with dependencies
References: Job dependencies in the Biowulf User Guide, Building pipelines using slurm dependencies
- Molecular Dynamics jobs.
Video (27 mins)
References: Making efficient use of Biowulf's multinode partition. - Relion tips and tricks.
- Pulling it all together: Working through converting a serial job into a swarm. How to decide on the resources to allocate for the jobs, submitting and monitoring the jobs.
- Tidbits:
- The largemem and unlimited partitions.
- Why are my jobs pending?
- Licensed applications
- Environment variables set by Slurm
- Advanced Quiz -- test your understanding of sbatch and swarm.
- Help beyond the class topics: The HPC staff includes scientists and system administrators who respond to user questions. The staff supports 2000+ users in diverse research areas, and thus is not available for individual collaborations, but can help with Biowulf-related questions if you send email to staff@hpc.nih.gov.
We also host Walk-In Consults around campus once a month, where you can discuss your problems with the staff directly. The Walk-In Schedule is emailed to users and is on the calendar.
- Additional training videos on more advanced topics, such as Jupyter notebooks, Matlab, tensorflow.
Additional topics? Anything insufficiently covered? Please send any comments regarding this online class to staff@hpc.nih.gov