Extra command line utilities not available by default, all in one convenient location, as simple as module load extrautils.

Extrautils includes some modernized, alternative verions of classic command line utilities as well as novel CLI or TUI utilities, including some you might be surprised to learn about!

fd is a simple, fast and user-friendly alternative to find. While it does not aim to support all of find's powerful functionality, it provides sensible (opinionated) defaults for a majority of use cases.

• https://github.com/sharkdp/fd

  # Standard search
biowulf$ fd bam
SRR15982321.bam
test_directory/53JKL.100K.rep-1.bam
bioconda-recipes/recipes/bamm/bamm-makefile.patch

  # Regex search
biowulf$ fd '^.*\.bam$'
SRR15982321.bam
test_directory/53JKL.100K.rep-1.bam

  # Extension search
biowulf$ fd -e bam
SRR15982321.bam
test_directory/53JKL.100K.rep-1.bam

  # Command execution
biowulf$ fd bam -x ls -lh
-rw-r--r--. 1 user group 98M Feb 26 12:15 ./SRR15982321.bam
-rw-r--r--. 1 user group 13M Dec 28  2022 ./53JKL.100K.rep-1.bam

fd has many of the same features as find, but frequently does just what you expect with a minimum of extra flags you need to memorize.

ripgrep (rg) is a line-oriented search tool that recursively searches the current directory for a regex pattern. ripgrep is much like grep, but with Unicode suppoert, a better user experience, faster, and tailored for searching code.

• https://github.com/BurntSushi/ripgrep

biowulf$ rg FastAPI .
./app.py
1: from fastapi import FastAPI, Request
3: app = FastAPI()

• ripgrep is generally faster, has some improved defaults like recursive search and automatic filtering that make it excel at code search.
• ripgrep is not drop-in compatible with grep
• ripgrep has many more features, please check it out on GitHub

A faster, more-featureful drop-in replacement for standard grep. Unlike ripgrep, ugrep may be safely symlinked as grep, egrep, etc.

• https://github.com/Genivia/ugrep

biowulf$ ug FastAPI .
app.py
     1: from fastapi import FastAPI, Request
     3: app = FastAPI()

lsd is a rewrite of GNU ls with lots of added features like colors, icons, tree-view, more formatting options etc. The project is heavily inspired by the super colorls project.

Note: lsd works best with either nerd-font or font-awesome fonts available to your remote terminal to provide icons. Have a look at the Nerd Font README for more installation instructions. Don't forget to setup your terminal in order to use the correct font.

• https://github.com/lsd-rs/lsd

biowulf$ lsd -la ~/code/
drwxr-xr-x user group 4.0 KB Tue Sep 26 15:29:32 2023  .
drwx------ user group 116 KB Mon Apr  1 11:57:34 2024  ..
drwxr-xr-x user group 4.0 KB Mon Jun 12 11:38:27 2023   __pycache
.rw-r--r-- user group 252 B  Mon Jun 12 11:38:26 2023   app.py

A modern, maintained replacement for ls. A replacement for the now-unmaintained exa

• https://github.com/eza-community/eza

biowulf$ eza -la ~/code
drwxr-xr-x   - user 12 Jun  2023 __pycache__
.rw-r--r-- 252 user 12 Jun  2023 app.py

The extrautils module also provides exa as a symlink to eza for convenience.

An improved version of the tree command that:

• lists directory structure in a tree-like diagram, like the classics.
• skips ignored files in git repositories per .gitignore setting.
• creates shell aliases for each listing that opens the files for you.
• outputs in colors, respecting LS_COLORS settings when they exist.

• https://github.com/dduan/tre

biowulf$ tre ~/code
/home/user/code
├── app.py
└── __pycache__
    └── app.cpython-311.pyc

du + rust = dust. Like du but more intuitive. dust provides an easy-to-use visualization tool for disk space usage allow you to easily find the subfolders eating up all of your quota.

• https://github.com/bootandy/dust

biowulf$ dust ~/code
4.0K   ┌── app.py               │                           ██████████ │  25%
4.0K   │ ┌── app.cpython-311.pyc│                  ░░░░░░░░░██████████ │  25%
8.0K   ├─┴ __pycache__          │                  ███████████████████ │  50%
 16K ┌─┴ code                   │█████████████████████████████████████ │ 100%

dust has additional flags to control search parameters, output, exlusion criteria, and more!

ncdu is an interactive disk usage analyzer with a text UI designed to find space-hogging directories.

• https://dev.yorhel.nl/ncdu

biowulf$ ncdu
ncdu 1.18.1 ~ Use the arrow keys to navigate, press ? for help   
--- /spin1/home/linux/user/code ------------------------------
    8.0 KiB [#################################] /__pycache__     
    4.0 KiB [################                 ]  app.py


 Total disk usage:  16.0 KiB  Apparent size:   9.0 KiB  Items: 3 

ncdu includes an interactive help menu if you hit the question mark key.

diskonaut is an interactive disk usage analyzer with a semi-graphical text UI with a treemap view to visualize space usage in your directories.

• https://github.com/imsnif/diskonaut

biowulf$ diskonaut
Total: 8.0K (3 files), freed: 0 | /spin1/home/linux/user/code     
┌────────────────────────────────────────────┬───────────────────────────────────────────┐
│                                            │                                           │
│                                            │                                           │
│                                            │                                           │
│                                            │                                           │
│                                            │                                           │
│       __pycache__/ (+1 descendants)        │                  app.py                   │
│                                            │                                           │
│                 4.0K (50%)                 │                4.0K (50%)                 │
│                                            │                                           │
│                                            │                                           │
│                                            │                                           │
│                                            │                                           │
│                                            │                                           │
└────────────────────────────────────────────┴───────────────────────────────────────────┘
                                                                        (x = Small files)
 ←↓↑→/<ENTER>/<ESC>: navigate, <BACKSPACE>: del

bat is like cat, but so much more, supporting syntax highlighting, paging (like less or more), git integration, and more while still being able to operate as a drop-in replacement for cat in non-interactive context.

• https://github.com/sharkdp/bat

biowulf$ bat app.py
───────┬────────────────────────────────────────────────────────────────────────────────
       │ File: app.py
───────┼────────────────────────────────────────────────────────────────────────────────
   1   │ from fastapi import FastAPI, Request
   2   │
   3   │ app = FastAPI()
   4   │
   5   │ @app.get("/")
   6   │ async def root(request: Request):
   7   │     user = request.headers.get("X-Remote-User", "user").split("@")[0]
   8   │     #return {"message": f"Hello {user}!"}
   9   │     return request.headers
───────┴─────────────────────────────────────────────────────────────────────────────────

sd is an intuitive find & replace CLI. In particular, sd uses now-familiar regex syntax from languages like Python or JavaScript. Introduces very basic "string-literal" mode for simple replacements to avoid the need for complex shell quoting, and generally makes simple find/replace easier. More limited in scope than sed but with common-sense defaults.

• https://github.com/chmln/sd

biowulf$ rg FastAPI .
./app.py
1: from fastapi import FastAPI, Request
3: app = FastAPI()

biowulf$ sd FastAPI SlowAPI app.py # Defaults to in-place

biowulf$ rg SlowAPI .
./app.py
1: from fastapi import SlowAPI, Request
3: app = SlowAPI()

biowulf$ sd '\n' ',' app.py # Simple manipulation of newlines (newline to comma)

biowulf$ bat app.py
───────┬────────────────────────────────────────────────────────────────────────────────
       │ File: app.py
───────┼────────────────────────────────────────────────────────────────────────────────
   1   │ from fastapi import FastAPI, Request,,app = FastAPI(),@app.get("/"),async def r
       │ oot(request: Request):,    user = request.headers.get("X-Remote-User", "user").
       │ split("@")[0],    #return {"message": f"Hello {user}!"},    return request.heade
       │ rs,
───────┴─────────────────────────────────────────────────────────────────────────────────

micro is a terminal-based text editor that aims to be easy to use and intuitive, while also taking advantage of the capabilities of modern terminals.

As its name indicates, micro aims to be somewhat of a successor to the nano editor by being easy to install and use. It strives to be enjoyable as a full-time editor for people who prefer to work in a terminal, or those who regularly edit files over SSH.

• https://github.com/zyedidia/micro

biowulf$ micro app.py
 1 from fastapi import FastAPI, Request
 2 
 3 app = FastAPI()
 4 
 5 @app.get("/")
 6 async def root(request: Request):
 7     user = request.headers.("X-Remote-User", "user").("@")[0]
 8     #return {"message": f"Hello {user}!"}
 9     return request.headers
10 

app.py (10,1) | ft:python | unix | utf-8               Alt-g: bindings, Ctrl-g: help
^Q Quit, ^S Save, ^O Open, ^G Help, ^E Command Bar, ^K Cut Line
^F Find, ^Z Undo, ^Y Redo, ^A Select All, ^D Duplicate Line, ^T New Tab

Tip: Hit Alt-G to see a list of useful key bindings.

nnn is a full-featured terminal file manager. It is designed to be unobtrusive with smart workflows to match the trains of thought.

• https://github.com/jarun/nnn

biowulf$ nnn
1 2 3 4 ~/code

 __pycache__/
 app.py


 1/2 2023-06-12 11:38 drwxr-xr-x 4K

dyff is a "diff" tool to compare YAML or JSON files by showing only the parts of the files that changed, ignoring formatting and whitespace.

• https://github.com/homeport/dyff

biowulf$ dyff between data.json data2.json
    _         __  __
  _| | _   _ / _|/ _|  between data.json
 / _' | | | | |_| |_       and data2.json
| (_| | |_| |  _|  _|
 \__,_|\__, |_| |_|   returned one difference
       |___/

samples
- two list entries removed:     + two list entries added:
 - id: sample1                   - id: sampleA
 | data: sample1.fastq.gz        | data: sampleA.fastq.gz
 - id: sample2                   - id: sampleB
 | data: sample2.fastq.gz        | data: sampleB.fastq.gz

dyff follows the convention of diff and shows the change by going from the from input file to the to input file:
dyff between <from> <to>

A portable command-line YAML, JSON, XML, CSV, TOML and properties processor. Similar to jq, with the same syntax, but supports YAML, XML, and more.

• https://github.com/mikefarah/yq

biowulf$ yq '.[]' data.json
11:13:59 initCommand [WARN] JSON file output is now JSON by default (instead of yaml). Use '-oy' or '--output-format=yaml' for yaml output
[
  {
    "id": "sample1",
    "data": "sample1.fastq.gz"
  },
  {
    "id": "sample2",
    "data": "sample2.fastq.gz"
  }
]

biowulf$ yq -oy '.[]' data.json
- id: sample1
  data: sample1.fastq.gz
- id: sample2
  data: sample2.fastq.gz

Interactive terminal JSON viewer & processor.

• https://fx.wtf/
• https://github.com/antonmedv/fx

Interactive demo coming soon!

CLI tool and python library that converts the output of popular command-line tools, file-types, and common strings to JSON, YAML, or Dictionaries. This allows piping of output to tools like jq and simplifying automation scripts.

• https://github.com/kellyjonbrazil/jc

biowulf$ ls -lh | jc --ls | jq
[
  {
    "filename": "app.py",
    "flags": "-rw-r--r--",
    "links": 1,
    "owner": "user",
    "group": "group",
    "size": 242,
    "date": "Apr 2 08:24"
  },
  {
    "filename": "data2.json",
    "flags": "-rw-r--r--",
    "links": 1,
    "owner": "user",
    "group": "group",
    "size": 108,
    "date": "Apr 2 08:37"
  },
  {
    "filename": "data.json",
    "flags": "-rw-r--r--",
    "links": 1,
    "owner": "user",
    "group": "group",
    "size": 107,
    "date": "Apr 2 08:36"
  },
  {
    "filename": "__pycache__",
    "flags": "drwxr-xr-x",
    "links": 2,
    "owner": "user",
    "group": "group",
    "size": 4,
    "date": "Jun 12 2023"
  }
]

CLI tool to filter JSON and JSON Lines data. (Similar to jq, but with possibly more-familiar Python syntax)

• https://github.com/kellyjonbrazil/jello

biowulf$ jello '_["samples"][0]' -f data.json
{
  "id": "sample1",
  "data": "sample1.fastq.gz"
}

CLI tool to convert JSON and JSON Lines to terminal, CSV, HTTP, and markdown tables.

• https://github.com/kellyjonbrazil/jtbl

biowulf$ jq '.["samples"]' data.json | jtbl
id       data
-------  ----------------
sample1  sample1.fastq.gz
sample2  sample2.fastq.gz

biowulf$ jq '.["samples"]' data.json | jtbl -c
id,data
sample1,sample1.fastq.gz
sample2,sample2.fastq.gz

🌸 A command-line fuzzy finder

• https://github.com/junegunn/fzf

Interactive demo coming soon!

Fuzzy Finder in rust!

• https://github.com/lotabout/skim

Interactive demo coming soon!

A modern alternative to the watch command, records the differences in execution results and can check these differences later.

• https://github.com/blacknon/hwatch

biowulf$ hwatch ls
Every     2.000s: ls       Display help with h key!       2024-04-02 14:04:24.345
            [Number] [Color] [Output:Output] [Active: history] [Diff: None      ]
app.py                                                  >>latest                 
data2.json                                                2024-04-02 14:04:02.312
data.json
__pycache__

Run arbitrary commands when files change.

• https://github.com/eradman/entr

  # Launch and auto-reload a node.js server
biowulf$ ls *.js | entr -r node app.js

Sloc, Cloc and Code: scc is a very fast accurate code counter with complexity calculations and COCOMO estimates written in pure Go.

• https://github.com/boyter/scc

biowulf$ scc ~/code
───────────────────────────────────────────────────────────────────────────────
Language                 Files     Lines   Blanks  Comments     Code Complexity
───────────────────────────────────────────────────────────────────────────────
JSON                         2         2        0         0        2          0
Python                       1         9        2         1        6          0
───────────────────────────────────────────────────────────────────────────────
Total                        3        11        2         1        8          0
───────────────────────────────────────────────────────────────────────────────
Estimated Cost to Develop (organic) $169
Estimated Schedule Effort (organic) 0.51 months
Estimated People Required (organic) 0.03
───────────────────────────────────────────────────────────────────────────────
Processed 457 bytes, 0.000 megabytes (SI)
───────────────────────────────────────────────────────────────────────────────

A syntax-highlighting pager for git, diff, and grep output.

• https://github.com/dandavison/delta

biowulf$ delta *.json

Δ data2.json ⟶   data.json
─────────────────────────────────────────────────────────────────────────────────

─────┐
• 1: │
─────┘
{"samples": [{"id": "sampleA", "data": "sampleA.fastq.gz"}, {"id": "sampleA",
"data": "sampleA.fastq.gz"}]}
{"samples": [{"id": "sample1",
"data": "sample1.fastq.gz"}, {"id": "sample2", "data": "sample2.fastq.gz"}]}
\ No newline at end of file

A simple yet powerful command-line HTTP and API testing client for the API era.

• https://github.com/httpie/cli
• https://httpie.io/

biowulf$ https httpie.io/hello
HTTP/1.1 200 OK
Age: 0
Cache-Control: public, max-age=0, must-revalidate
Connection: keep-alive
Content-Length: 264
Content-Type: application/json; charset=utf-8
Date: Tue, 02 Apr 2024 18:19:48 GMT
Etag: "fcegchauwd78"
Server: Vercel
Strict-Transport-Security: max-age=63072000
X-Matched-Path: /api/hello
X-Vercel-Cache: MISS
X-Vercel-Id: iad1::iad1::8q5nz-1712081986564-3c141ddb1231

{
    "ahoy" [
        "Hello, World! 👋 Thank you for trying out HTTPie 🥳",
        "We hope this will become a friendship."
    ],
    "links": {
        "discord": "https://httpie.io/discord",
        "github": "https://github.com/httpie",
        "homepage": "https://httpie.io",
        "twitter": "https://twitter.com/httpie"
    }
}

A command-line benchmarking tool.

• URhttps://github.com/sharkdp/hyperfineL

biowulf$ hyperfine "grep -P '\s18005\d+\s' allchrs_data.txt | wc -l" \
                   "rg '\s18005\d+\s' allchrs_data.txt | wc -l" \
                   "ug '\s18005\d+\s' allchrs_data.txt | wc -l"

Benchmark 1: grep -P '\s18005[0-9]+\s' allchrs_data.txt | wc -l
  Time (mean ± σ):     810.5 ms ±   4.9 ms    [User: 768.0 ms, System: 84.3 ms]
  Range (min … max):   807.7 ms … 823.9 ms    10 runs

Benchmark 2: rg '\s18005\d+\s' allchrs_data.txt | wc -l
  Time (mean ± σ):     130.9 ms ±   1.8 ms    [User: 103.8 ms, System: 53.9 ms]
  Range (min … max):   128.6 ms … 137.2 ms    22 runs

Benchmark 3: ug '\s18005\d+\s' allchrs_data.txt | wc -l
  Time (mean ± σ):     320.4 ms ±   0.8 ms    [User: 278.6 ms, System: 59.0 ms]
  Range (min … max):   319.2 ms … 322.0 ms    10 runs

Summary
  rg '\s18005\d+\s' allchrs_data.txt | wc -l ran
    2.45 ± 0.03 times faster than ug '\s18005\d+\s' allchrs_data.txt | wc -l
    6.20 ± 0.09 times faster than grep -P '\s18005\d+\s' allchrs_data.txt | wc -l

Collaborative cheatsheets for console commands

• https://tldr.sh/

biowulf$ tldr sd

  sd

  Intuitive find and replace.
  More information: https://github.com/chmln/sd.

  Trim some whitespace using a regular expression (output stream: stdout):

    ffff99echo 'lorem ipsum 23   ' | sd '\s+$' ''

 Replace words using capture groups (output stream: stdout):

    echo 'cargo +nightly watch' | sd '(\w+)\s+\+(\w+)\s+(\w+)' 'cmd: $1, channel: $2, subcmd: $3'

  Find and replace in a specific file (output stream: stdout):

    sd -p 'window.fetch' 'fetch' path/to/file.js

  Find and replace in all files in the current project (output stream: stdout):

    sd 'from "react"' 'from "preact"' "$(find . -type f)"

An interactive cheatsheet tool for the command-line.

• https://github.com/denisidoro/navi

Interactive demo coming soon!

Linux tool to show progress for cp, mv, dd, ... (formerly known as cv)

• https://github.com/Xfennec/progress

Interactive demo coming soon!

A single binary that runs lists of commands in parallel and waits for their termination

• https://github.com/slavaGanzin/await
• https://await-cli.app/

Interactive demo coming soon!

rng streams input from a file or stdin to stdout. It is the love child of sed -n '<START>,<END>p' and tail -n.

• https://github.com/nickolasburr/rng

biowulf$ rng 3,5 app.py
app = FastAPI()

@app.get("/")

biowulf$ rng 6 app.py
async def root(request: Request):
    user = request.headers.get("X-Remote-User", "user").split("@")[0]
    #return {"message": f"Hello {user}!"}
    return request.header

FIGlet is a program for making large letters out of ordinary text.

• http://www.figlet.org/

biowulf$ figlet extrautils
           _                   _   _ _
  _____  _| |_ _ __ __ _ _   _| |_(_) |___
 / _ \ \/ / __| '__/ _` | | | | __| | / __|
|  __/>  <| |_| | | (_| | |_| | |_| | \__ \
 \___/_/\_\\__|_|  \__,_|\__,_|\__|_|_|___/

Do you need more color in your life?

• https://github.com/ur0/lolcat

biowulf$ figlet EXTRAUTILS | lolcat
 _______  _______ ____      _   _   _ _____ ___ _     ____
| ____\ \/ /_   _|  _ \    / \ | | | |_   _|_ _| |   / ___|
|  _|  \  /  | | | |_) |  / _ \| | | | | |  | || |   \___ \
| |___ /  \  | | |  _ <  / ___ \ |_| | | |  | || |___ ___) |
|_____/_/\_\ |_| |_| \_\/_/   \_\___/  |_| |___|_____|____/