Running LLM Agents on Biowulf

This page is a landing site for documentation regarding running LLM agents on Biowulf.

At the moment, we can only provide links to other documentation for other sites, but in the future there will be more content.

 

 

Common problems with LLM agents on HPC

Problem What can go wrong Safer practice on Biowulf
Cluster stability An agent may repeatedly submits jobs, polls the scheduler, spawns runaway loops, or generates excessive I/O. Keep the session supervised, ideally don't run more than one agent, avoid tight polling, and require approval for job actions.
Agent on a login node The agent runs commands, scans files, or consumes resources on shared login node. Stop it and start the agent only after entering a Biowulf interactive compute session, or through HPC Ondemand.
Code and data confidentiality Code, file contents, and error messages are sent to an external LLM provider’s servers. Sensitive data, unpublished results, or secrets (passwords, API keys, tokens) may be exposed. You might also expose code to other users of the shared HPC node. Do not work with PII and PHI data. Give an agent access only to the project directory it needs—not your entire /home, /data, or shared filesystem. Check the privacy policy of the LLM tools you are using (up to 5-year retention period for Claude)
Prompt injection A README, webpage, issue, or agent “skill” can contain instructions that manipulate the agent into unsafe actions. Cautious about external browsing, review installed skills/plugins, and require confirmation from agents before taking actions after reading external content.
Autonomous file changes Files can be overwritten, moved, or deleted faster than a user can intervene. Use a dedicated workspace, version control/backups where appropriate, and ask the agent which commands to run and execute them yourself.
Wrong Slurm assumptions The model invents flags, partitions, resource limits, GPU requests, or launch patterns. Verify Biowulf documentation and your working job scripts.
Dependency risk An agent may install packages from PyPI, npm, Conda-forge, CRAN, or other registries without adequate review. Review dependencies first; prefer existing Biowulf modules; never run an agent with elevated privileges.
Hallucinated scientific results Plausible code or analysis may be technically or scientifically wrong. Run tests, compare against known cases, inspect outputs, benchmark performance, and independently verify scientific conclusions.

Recommended global rules for Biowulf

Global rules: https://hpc.nih.gov/AGENTS.md

Agent harnesses use different names and locations for persistent instructions. Global rules should apply to every session, Biowulf-managed content should be placed at the top of ~/.codex/AGENTS.md or ~/.claude/CLAUDE.md (or the agent’s equivalent).

Codex loads ~/.codex/AGENTS.md as global user guidance. If that file does not exist, download the supplied file there. If it already exists, preserve the user's guidance and add the complete contents of .codex/AGENTS.md to the top of it.

Project-specific AGENTS.md files can add more specific instructions when Codex works inside a project.

If ~/.codex/AGENTS.md does not exist


mkdir -p ~/.codex
curl -fL -o ~/.codex/AGENTS.md https://hpc.nih.gov/AGENTS.md
chmod 0600 ~/.codex/AGENTS.md

If ~/.codex/AGENTS.md already exists, do not overwrite it. Add the complete contents of codex/.codex/AGENTS.md to the top of the existing file, leave the existing user guidance below it, and keep a blank line between the two. Then set the combined file to owner-only access:


sh -c 'mkdir -p ~/.codex; curl -fL -o ~/.codex/AGENTS_biowulf.md https://hpc.nih.gov/AGENTS.md; touch ~/.codex/AGENTS.md; { cat ~/.codex/AGENTS_biowulf.md; printf "\n"; cat ~/.codex/AGENTS.md; } > ~/.codex/AGENTS.md.new; mv ~/.codex/AGENTS.md.new ~/.codex/AGENTS.md; chmod 0600 ~/.codex/AGENTS.md'

Restart Codex after installing or updating this file so that it loads the current global guidance.

For Claude, replacing ~/.codex/AGENTS.md with ~/.claude/CLAUDE.md, the newer version of Claude (>=2.1.277) support AGETNS.md, but it only loads AGENTS.md automatically if no CLAUDE.md file exists.

Documentation

Please see here about using Codex on Biowulf.