The NIH HPC staff maintains installations of commonly-used software and, for many users, these centrally installed applications and versions will be sufficient. However, some users will need stable, unchanging versions of a particular application, or will need an application that is not appropriate for central installation. For these users, a personal installation is a good solution.

Managing your own software installations allows you to:

• control exact versions for a stable analysis pipeline.
• use a particular application that is not centrally installed on Biowulf.

Terminology

Build system

In the context of software, it's a tool used to organize the setup, processing, and installation of another software project's source code. Cmake is an example of a software build system.

Container

An archive that contains the user-facing parts of an operating system. Containers can be built to include specific software installations and used transparently on other systems. Only Singularity and Apptainer containers are supported on Biowulf, but Docker containers can be converted.

Package manager

A tool to install and remove software. When installing a selected software package, the package manager identifies all of its requirements and installs them as well.

General package managers fall into two categories, which we will term user-level and system-level. The former can be used by anyone, while the latter type requires administrative privileges on the system.

User-level Package Managers

User-level package managers such as conda, homebrew, and guix do not require any special privileges. They can be used to install almost any version of any package.

See for Biowulf-specific usage notes:

• Conda

System-level Package Managers

The requirement of administrative privileges for system-level package managers makes them an option when building containers. When considering system package managers for this purpose, keep in mind that they:

• only manage a single, system-wide environment, which is always active.
• have limited flexibility for choosing software versions. This ensures consistency, minimizing conflicts and incompatibilities between software.

Many programming language communities have their own user-level package managers: pip for Python, cpan for Perl, and so on. The primary limitation of these tools is that they cannot install all of a package's third-party requirements for you if any of them were written in a different language. This is often the case if you are installing a pipeline or if your application requires an interface to a low-level compiled library. If this is not a concern, then the language-specific package managers such as pip and cpan can work for you.

Where applicable, these will be documented on the language's page among the NIH HPC Apps:

• R
• MATLAB

Environment modules are used on Biowulf to dynamically set up environments for different applications. You can also set up your own environment modules for software you have installed. This option is worth exploring if, for example, the software you need is not available as a conda package or if you would like greater control over the process. The preparation of a personal module ranges in complexity from simply loading multiple other existing modules at once, to being the final step of a process in which you have manually downloaded and compiled software from its source code.

For how to set up and use them, see:

• the Personal Modulefiles section of the Biowulf User Guide
• Environment Modules: Advanced Topics

At this point, what you have to do depends on how the developers have organized their software project. When you use a package manager, you are spared this variability since a packager has already dealt with this to make a unified installation interface, but here you are on your own. The developers may have used a common software build system, in which case, you might be able to get by with minimal documentation on the developer's part. If not, there is no recourse other than whatever the developers may have documented or answered (if they can be reached for questions). A mastery of Linux and the programming language used for the project can help to a great extent, but there is no substitute for the developer's firsthand knowledge. Contact the HPC staff if you need assistance.

For more on using common software build systems, see

• Slides from last workshop on managing personal software installations