bashfoo.yaml

https://github.com/dannguyen/bashfoo

Dan Nguyen's personally curated list of bash/command-line commands and snippets that are useful but yet he keeps forgetting

date add a UTC timestamp in ISO-8601 format to a filename
du recursively list directories and their disk space size
ffmpeg convert a video to gif
ffmpeg encode .mkv video to .mp4
ffmpeg optimize video and reduce its file size
ffmpeg resize a video and preserve its aspect ratio
find and execute command on each file
find and list the top 10 most recently modified subdirectories
find and tally total kilobytes of hard disk space for files with given extension(s)
find directory name recursively
find file by name
magick convert image to favicon.ico
ogr2ogr convert a shapefile into CSV + GeoJSON
pandoc convert a Markdown file into a Word docx
pgrep and get all process info
pkill using a file pattern
printf to stderr
pygmentize a code snippet into highlighted rich text that I can paste into GMail
rsync the contents of one directory into another
ssh-add a new SSH key to system ssh-agent (macOS)
stem a filename, i.e. get filename sans path or extension
tar extraction, verbose
unzip only an archive's CSV files and pipe to stdout
xargs (BSD) to pipe results into another command, one at a time
youtube-dl to download just a video's transcript/subtitle files

`date` add a UTC timestamp in ISO-8601 format to a filename

# Example
# Just the current UTC date using GNU date, i.e. gdate on macOS
printf "mydata_%s.csv\n" $(gdate --utc -I)
# >>>> mydata_2021-08-03.csv

# UTC date with hours+minutes, GNU and BSD compatible
printf "mydata_%s.csv\n" $(date -u +"%Y-%m-%d-%H%M")
# >>>> mydata_2021-08-03-1248.csv

References:

date command --iso-8601 option

`du` recursively list directories and their disk space size

# Example
# GNU du (verbose)
du --human-readable --max-depth=2 ./bashfoo

# GNU du (concise) / BSD du (i.e. MacOS)
du -h -d 2 ./bashfoo

Output:

2.4M  ./bashfoo/.git/objects
4.0K  ./bashfoo/.git/info
 44K  ./bashfoo/.git/logs
 48K  ./bashfoo/.git/hooks
 16K  ./bashfoo/.git/refs
2.6M  ./bashfoo/.git
3.6M  ./bashfoo/assets/samples
3.6M  ./bashfoo/assets
6.3M  ./bashfoo

References:

Notes:

More variations and usecases for GNU du:

List only directories of a minimum size

# (concise) $ du -t 3M ./bashfoo
$ du --threshold 3M ./bashfoo
3660  bashfoo/assets/samples
3728  bashfoo/assets
6436  bashfoo/

Specify a size unit, e.g. K, M, G for: kb, mb, gb

# (concise) $ du -B M ./bashfoo
$ du --block-size M ./bashfoo
1M  ./bashfoo/.git/objects/61
...
1M  ./bashfoo/assets/samples/chicago_neighborhoods
4M  ./bashfoo/assets/samples
4M  ./bashfoo/assets
7M  ./bashfoo/

List largest 5 directories using human-numeric-sort

$ du -h ~/Downloads | sort -h -r | head -n 5
2.6G  /Users/me/Downloads
851M  /Users/me/Downloads/ebooks
233M  /Users/me/Downloads/OLD_JUNK
25M   /Users/me/Downloads/names
9.5M  /Users/me/Downloads/bootstrap-4.5.2

`ffmpeg` convert a video to gif

# Example
# minimalist conversion (infinite looping GIF, i.e. -loop=0, is the default)
ffmpeg -i input.mp4 output.gif

# yes-to-overwrite, 15 frames-per-second
# scaled to 225px wide w/ lanczos scaler, and non-looping
ffmpeg -y -i input.mp4 \
  -vf "fps=15,scale=225:-1:flags=lanczos" -loop -1 \
  output.gif

# generate a palette png file that favors areas with motion
ffmpeg -y -i input.mp4 -vf "palettegen=stats_mode=diff" palette.png

# and then use that palette png
ffmpeg -y -i input.mp4 -i palette.png -lavfi paletteuse output.gif

References:

`ffmpeg` encode .mkv video to .mp4

# Example
ffmpeg -i input.mkv -acodec aac -vcodec libx264 output.mp4

References:

How can I convert an m4v video to a widely viewable format using ffmpeg?

`ffmpeg` optimize video and reduce its file size

# Example
ffmpeg -i my_video.mp4 -vcodec libx264 -crf 28 my_video_optimized.mp4

References:

How can I reduce a video's size with ffmpeg?

`ffmpeg` resize a video and preserve its aspect ratio

# Example
# to rescale a video's width to 450 pixels and autoscale its height
ffmpeg -i in_video.mp4 -vf scale=450:-2 out_video.mp4

# to rescale a video's height to 204 pixels and autoscale its width
ffmpeg -i in_video.mp4 -vf scale=-2:204 out_video.mp4

# Note that the given pixel width/height should be an even number
# to avoid getting an error like: height not divisible by 2 (500x201)

References:

`find` and execute command on each file

# Example
find ./PATTERN -exec FOO BAR {} \;

find PlainText/*.md -exec wc -l {} \;

References:

How to run find -exec?

`find` and list the top 10 most recently modified subdirectories

# Example
find ~/a -mindepth 1 -maxdepth 2 -type d \
    -not -name '_*' -not -name '.*' \
    -print0
  | xargs -0 -n1 -I{} \
      stat  -f '%Sm %N' -t '%Y-%m-%d %H:%M:%S' {} \
  | sort -rn | head -n10

# GNU variant, e.g. with gnu-coretools on macOS
gfind . -mindepth 1 -maxdepth 2 -type d \
    -not -name '_*' -not -name '.*'  \
    -printf '%T+ %p\n'  \
  | sort -rn \
  | head -n 10

Output:

2020-11-27 14:05:45 /Users/dan/Downloads/r-book
2020-09-25 11:39:45 /Users/dan/Desktop/sf-shelter-data
2020-06-18 22:45:15 /Users/dan/Downloads/journalism-syllabi
2020-08-26 20:16:55 /Users/dan/Desktop/vocal-samples
2020-07-09 14:48:38 /Users/dan/Downloads/svelte-project
2020-07-09 14:29:43 /Users/dan/Desktop/matplotlibguide
2020-05-29 22:00:28 /Users/dan/Downloads/buzzfeed-archives
2020-03-18 14:57:03 /Users/dan/Downloads/transcribe-texts
2020-01-18 05:55:32 /Users/dan/Desktop/oldstuff
2020-01-17 16:43:39 /Users/dan/Desktop/random_images

References:

How to recursively find and list the latest modified files in a directory with subdirectories and times (my answer

`find` and tally total kilobytes of hard disk space for files with given extension(s)

# Example
echo $(find . -type f \
        \( -iname "*.csv" -o -iname '*.xls*' \) \
        -printf "(%k/1024)+" \
        2>/dev/null; \
        echo 0;) | bc

Output:

References:

Find the total size of certain files within a directory branch

Notes:

Requires the use of gnu-find (gfind on my MacOS)
use -printf "%s+" to print size by bytes
2>/dev/null hides error messages
-iname is case-insensitive

`find` directory name recursively

# Example
find START_DIR -type d -name "PATTERN"

References:

How can I recursively search for directory names with a particular string where the string is only part of the directory name

`find` file by name

# Example
find . -name "foo*"

References:

How can I recursively find all files in current and subfolders based on wildcard matching?

`magick` convert image to favicon.ico

# Example
magick /tmp/testimage.png -background none -resize 128x128 -density 128x128 favicon.ico

References:

Convert PNG to ICO

`ogr2ogr` convert a shapefile into CSV + GeoJSON

# Example
ogr2ogr -f CSV \
    -dialect sqlite \
    -sql 'SELECT *, AsGeoJSON(geometry) AS geom FROM chicago_neighborhoods' \
    chicago_neighborhoods_output.csv  \
    chicago_neighborhoods.shp

References:

Notes:

Chicago neighborhood shapefile comes from the city data portal

Download a copy of the zipped shapefile: chicago_neighborhoods.zip

This is what a shapefile converted into CSV looks like: chicago_neighborhoods_output.csv

`pandoc` convert a Markdown file into a Word docx

# Example
pandoc README.md -f markdown -t docx -o README.docx

References:

Convert Markdown into a Word Document

`pgrep` and get all process info

# Example
# MacOS
pgrep -fil 'rails'

# Linux
pgrep -af 'rails'

Output:


47502 rails master RBENV_VERSION=2.5.1 TERMINAL_FONT=Monaco
47517 rails worker[0] RBENV_VERSION=2.5.1 TERMINAL_FONT=Monaco

References:

How to get pgrep to display full process info

`pkill` using a file pattern

# Example
pkill -fil ipython

Output:

kill -15 90396
kill -15 90523

References:

How to kill all processes with a given partial name?

`printf` to stderr

# Example
>&2 printf 'Error: %s\n' 'There was a problem' 'And another problem'

Output:

Error: There was a problem
Error: And another problem

References:

print output to stderr, not stdout

`pygmentize` a code snippet into highlighted rich text that I can paste into GMail

# Example
# Converting plaintext to rich-text format and copying to clipboard
printf "SELECT 1, 2, 3 \nFROM table;" \
  | pygmentize -f rtf -l sql \
  | pbcopy

# Producing a PNG of a code snippet, using the "vim" stylesheet,
# 36-point font (default is 14), 21px line spacing (default is 2px),
# and highlighting lines 2, 4, and 6 in hot pink (line numbers are on by default)
printf 'SELECT *\n\t, name, id\nFROM\n\tdTable AS Texas\nWHERE\n\tid > "Howdy"\nORDER BY\n\tname DESC;' \
  | pygmentize -f png -l sql \
    -O 'style=vim' \
    -O 'font_size=36,line_pad=21' \
    -O 'hl_color=hotpink,hl_lines=2 4 6' \
  > assets/pretty-pyg-sql.png

Example of rich-text format (-f rtf) being pasted into GMail:

pygmentize -f rtf -l

Example of writing pygmentize into PNG file:

pygmentize -f png

References:

Notes:

install via pip: pip install --upgrade Pygments
Use -L to get your installed version of pygmentize's most complete list of all styles, formatters, and lexers

`rsync` the contents of one directory into another

# Example
rsync -ahv --progress src_dir/ target_dir

Output:

building file list ... done
created directory /target_dir
./
.gitignore
README.md
static/media/
static/media/demos/
static/media/demos/find.mp4

sent 452477 bytes  received 2378 bytes  50909710.00 bytes/sec
total size is 472016  speedup is 1.00

References:

How To Use Rsync to Sync Local and Remote Directories on a VPS

Notes:

only the source directory should have a trailing slash
-v makes for verbose output
-n does a dry run
-z compresses files
--progress provides a progress bar
-u updates only the destination files older than the source files
h human-readable numbers

`ssh-add` a new SSH key to system ssh-agent (macOS)

# Example
# put this in bash profile
eval "$(ssh-agent -s)"

# note: use -t rsa -b 4096 if system doesn't support Ed25519 algorithm
ssh-keygen -t ed25519 -C "[email protected]" \
  -f ~/.ssh/myid_as_ed25519

# add key to ssh-agent and store passprhase in keychain
$ ssh-add -K ~/.ssh/myid_as_ed25519

Output:

Agent pid 59566

Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/justme/.ssh/myid_as_ed25519.
Your public key has been saved in /Users/justme/.ssh/myid_as_ed25519.pub.
The key fingerprint is:
SHA256:9zABCDEx9sp+zyxTestTest+LoremIpsum [email protected]
The key's randomart image is:
+--[ED25519 256]--+
|.  ..     .   .o.|
|     .o.. .o .o .|
|                 |
|     ..o + + = * |
|                 |
|B       o        |
|    o... o . . .+|
|     ...T o   +o.|
|     oo.  .  .== |
+----[SHA256]-----+

References:

Notes:

May need to edit ~/.ssh/config to point to non traditional IdentityFile location, e.g.

Host *
 AddKeysToAgent yes
 UseKeychain yes
 IdentityFile ~/.ssh/id_rsa
 IdentityFile ~/.ssh/myid_as_ed25519

`stem` a filename, i.e. get filename sans path or extension

# Example
fullname=/tmp/hello/world.txt
newname="a_whole_new_$(basename ${fullname%.*})"
echo $newname

Output:

a_whole_new_world

References:

Extract filename and extension in Bash

`tar` extraction, verbose

# Example
tar xzfv ARCHIVE.TAR.GZ

References:

The tar command explained

`unzip` only an archive's CSV files and pipe to stdout

# Example
unzip -p schools.zip "*.csv" > schools.csv

`xargs` (BSD) to pipe results into another command, one at a time

# Example
echo Alice Bob Charlie | xargs -I{} -n1 echo 'Hey, {} is a great name!'

Output:

Hey, Alice is a great name!
Hey, Bob is a great name!
Hey, Charlie is a great name!

References:

Execute a command once per line of piped input?

`youtube-dl` to download just a video's transcript/subtitle files

# Example
youtube-dl --write-sub --skip-download https://www.youtube.com/watch?v=PMp_-OX15Jc

Output:

[youtube] PMp_-OX15Jc: Downloading webpage
[youtube] PMp_-OX15Jc: Downloading MPD manifest
[info] Writing video subtitles to: What is Public Domain-PMp_-OX15Jc.en.vtt

bashfoo
bashfoo copied to clipboard

Metadata

bashfoo.yaml

TOC

`date` add a UTC timestamp in ISO-8601 format to a filename

`du` recursively list directories and their disk space size

`ffmpeg` convert a video to gif

`ffmpeg` encode .mkv video to .mp4

`ffmpeg` optimize video and reduce its file size

`ffmpeg` resize a video and preserve its aspect ratio

`find` and execute command on each file

`find` and list the top 10 most recently modified subdirectories

`find` and tally total kilobytes of hard disk space for files with given extension(s)

`find` directory name recursively

`find` file by name

`magick` convert image to favicon.ico

`ogr2ogr` convert a shapefile into CSV + GeoJSON

`pandoc` convert a Markdown file into a Word docx

`pgrep` and get all process info

`pkill` using a file pattern

`printf` to stderr

`pygmentize` a code snippet into highlighted rich text that I can paste into GMail

`rsync` the contents of one directory into another

`ssh-add` a new SSH key to system ssh-agent (macOS)

`stem` a filename, i.e. get filename sans path or extension

`tar` extraction, verbose

`unzip` only an archive's CSV files and pipe to stdout

`xargs` (BSD) to pipe results into another command, one at a time

`youtube-dl` to download just a video's transcript/subtitle files

← Metadata

Owner

Metadata

bashfoo bashfoo copied to clipboard

Metadata

bashfoo.yaml

TOC

date add a UTC timestamp in ISO-8601 format to a filename

du recursively list directories and their disk space size

ffmpeg convert a video to gif

ffmpeg encode .mkv video to .mp4

ffmpeg optimize video and reduce its file size

ffmpeg resize a video and preserve its aspect ratio

find and execute command on each file

find and list the top 10 most recently modified subdirectories

find and tally total kilobytes of hard disk space for files with given extension(s)

find directory name recursively

find file by name

magick convert image to favicon.ico

ogr2ogr convert a shapefile into CSV + GeoJSON

pandoc convert a Markdown file into a Word docx

pgrep and get all process info

pkill using a file pattern

printf to stderr

pygmentize a code snippet into highlighted rich text that I can paste into GMail

rsync the contents of one directory into another

ssh-add a new SSH key to system ssh-agent (macOS)

stem a filename, i.e. get filename sans path or extension

tar extraction, verbose

unzip only an archive's CSV files and pipe to stdout

xargs (BSD) to pipe results into another command, one at a time

youtube-dl to download just a video's transcript/subtitle files

← Metadata

Owner

Metadata

bashfoo
bashfoo copied to clipboard

`date` add a UTC timestamp in ISO-8601 format to a filename

`du` recursively list directories and their disk space size

`ffmpeg` convert a video to gif

`ffmpeg` encode .mkv video to .mp4

`ffmpeg` optimize video and reduce its file size

`ffmpeg` resize a video and preserve its aspect ratio

`find` and execute command on each file

`find` and list the top 10 most recently modified subdirectories

`find` and tally total kilobytes of hard disk space for files with given extension(s)

`find` directory name recursively

`find` file by name

`magick` convert image to favicon.ico

`ogr2ogr` convert a shapefile into CSV + GeoJSON

`pandoc` convert a Markdown file into a Word docx

`pgrep` and get all process info

`pkill` using a file pattern

`printf` to stderr

`pygmentize` a code snippet into highlighted rich text that I can paste into GMail

`rsync` the contents of one directory into another

`ssh-add` a new SSH key to system ssh-agent (macOS)

`stem` a filename, i.e. get filename sans path or extension

`tar` extraction, verbose

`unzip` only an archive's CSV files and pipe to stdout

`xargs` (BSD) to pipe results into another command, one at a time

`youtube-dl` to download just a video's transcript/subtitle files