shell-scripting-templates
shell-scripting-templates copied to clipboard
Shell scripting utility functions and a bash script boilerplate template
Shell Scripting Templates and Utilities
A collection of BASH utility functions and script templates used to ease the creation of portable and hardened BASH scripts with sane defaults.
Usage
There are two Script Templates located in the root level of this repository. The usage of these templates is described in detail below.
BASH Utility functions are located within the utilities/
folder.
Complex sed
find/replace operations are supported with the files located in sedfiles/
. Read the usage instructions.
Automated testing is provided using BATS. All tests are in the tests/
repo. A git pre-commit hook provides automated testing is located in the .hooks/
directory. Read about how to install the hook.
Bash Script Templates Usage
To create a new script, copy one of the script templates to a new file and make it executable chmod 755 [newscript].sh
. Place your custom script logic within the _mainScript_
function at the top of the script.
Script Templates
There are two templates located at the root level of this repository.
-
template.sh
- A lean template which attempts to source all the utility functions from this repository. You will need to update the path to the utilities folder sent to_sourceUtilities_
at the bottom of the script. This template will not function correctly if the utilities are not found. -
template_standalone.sh
- For portability, the standalone template does not assume that this repository is available. Copy and paste the individual utility functions under the### Custom utility functions
line.
Code Organization
The script templates are roughly split into three sections:
- TOP: Write the main logic of your script within the
_mainScript_
function. It is placed at the top of the file for easy access and editing. However, it is invoked at the end of the script after options are parsed and functions are sourced. - MIDDLE: Functions and default variable settings are located just below
_mainScript_
. - BOTTOM: Script initialization (BASH options, traps, call to
_mainScript_
, etc.) is at the bottom of the template
Default Options
These default options and global variables are included in the templates and used throughout the utility functions. CLI flags to set/unset them are:
-
-h, --help
: Prints the contents of the_usage_
function. Edit the text in that function to provide help -
--logfile [FILE]
Full PATH to logfile. (Default is${HOME}/logs/$(basename "$0").log
) -
loglevel [LEVEL]
: Log level of the script. One of:FATAL
,ERROR
,WARN
,INFO
,DEBUG
,ALL
,OFF
(Default is 'ERROR
') -
-n, --dryrun
: Dryrun, sets$DRYRUN
totrue
allowing you to write functions that will work non-destructively using the_execute_
function -
-q, --quiet
: Runs in quiet mode, suppressing all output to stdout. Will still write to log files -
-v, --verbose
: Sets$VERBOSE
totrue
and prints all debug messages to stdout -
--force
: If using the_seekConfirmation_
utility function, this skips all user interaction. ImpliedYes
to all confirmations.
You can add custom script options and flags to the _parseOptions_
function.
Script Initialization
The bottom of the script template file contains a block which initializes the script. Comment, uncomment, or change the settings here for your needs
trap '_trapCleanup_ ${LINENO} ${BASH_LINENO} "${BASH_COMMAND}" "${FUNCNAME[*]}" "${0}" "${BASH_SOURCE[0]}"' EXIT INT TERM SIGINT SIGQUIT SIGTERM ERR
# Trap errors in subshells and functions
set -o errtrace
# Exit on error. Append '||true' if you expect an error
set -o errexit
# Use last non-zero exit code in a pipeline
set -o pipefail
# Confirm we have BASH greater than v4
[ "${BASH_VERSINFO:-0}" -ge 4 ] || {
printf "%s\n" "ERROR: BASH_VERSINFO is '${BASH_VERSINFO:-0}'. This script requires BASH v4 or greater."
exit 1
}
# Make `for f in *.txt` work when `*.txt` matches zero files
shopt -s nullglob globstar
# Set IFS to preferred implementation
IFS=$' \n\t'
# Run in debug mode
# set -o xtrace
# Source utility functions
_sourceUtilities_
# Initialize color constants
_setColors_
# Disallow expansion of unset variables
set -o nounset
# Force arguments when invoking the script
# [[ $# -eq 0 ]] && _parseOptions_ "-h"
# Parse arguments passed to script
_parseOptions_ "$@"
# Create a temp directory '$TMP_DIR'
# _makeTempDir_ "$(basename "$0")"
# Acquire script lock
# _acquireScriptLock_
# Source GNU utilities for use on MacOS
# _useGNUUtils_
# Run the main logic script
_mainScript_
# Exit cleanly
_safeExit_
Utility Functions
The files within utilities/
contain BASH functions which can be used in your scripts. Each included function includes detailed usage information. Read the inline comments within the code for detailed usage instructions.
Including Utility Functions
Within the utilities
folder are many BASH functions meant to ease development of more complicated scripts. These can be included in the template in two ways.
1. Copy and paste into template_standalone.sh
You can copy any complete function from the Utilities and place it into your script. Copy it beneath the ### Custom utility functions
line. Scripts created this way are fully portable among systems
2. Source all the utility files by using template.sh
template.sh
contains a function to source all the utility files into the script. IMPORTANT: You will need to update the paths within the _sourceUtilities_
function to ensure your script can find this repository.
alerts.bash
-
_columns_
Prints a two column output from a key/value pair - -
_printFuncStack_
Prints the function stack in use. Used for debugging, and error reporting -
_alert_
Performs alerting functions including writing to a log file and printing to screen -
_centerOutput_
Prints text in the center of the terminal window -
_setColors_
Sets color constants for alerting (Note: Colors default to a dark theme.)
Basic alerting, logging, and setting color functions (included in scriptTemplate.sh
by default). Print messages to stdout and to a user specified logfile using the following functions.
debug "some text" # Printed only when in verbose (-v) mode
info "some text" # Basic informational messages
notice "some text" # Messages which should be read. Brighter than 'info'
warning "some text" # Non-critical warnings
error "some text" # Prints errors and the function stack but does not stop the script.
fatal "some text" # Fatal errors. Exits the script
success "some text" # Prints a success message
header "some text" # Prints a header element
dryrun "some text" # Prints commands that would be run if not in dry run (-n) mode
The following global variables must be set for the alert functions to work
-
$DEBUG
- Iftrue
, printsdebug
level alerts to stdout. (Default:false
) -
$DRYRUN
- Iftrue
does not eval commands passed to_execute_
function. (Default:false
) -
$LOGFILE
- Path to a log file -
$LOGLEVEL
- One of: FATAL, ERROR, WARN, INFO, DEBUG, ALL, OFF (Default:ERROR
) -
$QUIET
- Iftrue
, prints to log file but not stdout. (Default:false
)
arrays.bash
Utility functions for working with arrays.
-
_dedupeArray_
Removes duplicate array elements -
_forEachDo_
Iterates over elements and passes each to a function -
_forEachFilter_
Iterates over elements, returning only those that are validated by a function -
_forEachFind_
Iterates over elements, returning the first value that is validated by a function -
_forEachReject_
Iterates over elements, returning only those that are NOT validated by a function -
_forEachValidate_
Iterates over elements and passes each to a function for validation -
_inArray_
Determine if a value is in an array -
_isEmptyArray_
Checks if an array is empty -
_joinArray_
Joins items together with a user specified separator -
_mergeArrays_
Merges the values of two arrays together -
_randomArrayElement_
Selects a random item from an array -
_reverseSortArray_
Sorts an array from highest to lowest -
_setdiff_
Return items that exist in ARRAY1 that are do not exist in ARRAY2 -
_sortArray_
Sorts an array from lowest to highest
checks.bash
Functions for validating common use-cases
-
_commandExists_
Check if a command or binary exists in the PATH -
_functionExists_
Tests if a function is available in current scope -
_isInternetAvailable_
Checks if Internet connections are possible -
_isAlpha_
Validate that a given variable contains only alphabetic characters -
_isAlphaDash_
Validate that a given variable contains only alpha-numeric characters, as well as dashes and underscores -
_isAlphaNum_
Validate that a given variable contains only alpha-numeric characters -
_isDir_
Validate that a given input points to a valid directory -
_isEmail_
Validates that an input is a valid email address -
_isFQDN_
Determines if a given input is a fully qualified domain name -
_isFile_
Validate that a given input points to a valid file -
_isIPv4_
Validates that an input is a valid IPv4 address -
_isIPv6_
Validates that an input is a valid IPv6 address -
_isNum_
Validate that a given variable contains only numeric characters -
_isTerminal_
Checks if script is run in an interactive terminal -
_rootAvailable_
Validate we have superuser access as root (via sudo if requested) -
_varIsEmpty_
Checks if a given variable is empty or null -
_varIsFalse_
Checks if a given variable is false -
_varIsTrue_
Checks if a given variable is true
dates.bash
Functions for working with dates and time.
-
_convertToUnixTimestamp_
Converts a date to unix timestamp -
_countdown_
Sleep for a specified amount of time -
_dateUnixTimestamp_
Current time in unix timestamp -
_formatDate_
Reformats dates into user specified formats -
_fromSeconds_
Convert seconds to HH:MM:SS -
_monthToNumber_
Convert a month name to a number -
_numberToMonth_
Convert a month number to its name -
_parseDate_
Takes a string as input and attempts to find a date within it to parse into component parts (day, month, year) -
_readableUnixTimestamp_
Format unix timestamp to human readable format -
_toSeconds_
Converts HH:MM:SS to seconds
debug.bash
Functions to aid in debugging BASH scripts
-
_pauseScript_
Pause a script at any point and continue after user input -
_printAnsi_
Helps debug ansi escape sequence in text by displaying the escape codes -
_printArray_
Prints the content of array as key value pairs for easier debugging
files.bash
Functions for working with files.
-
_backupFile_
Creates a backup of a specified file with .bak extension or optionally to a specified directory. -
_decryptFile_
Decrypts a file withopenssl
-
_encryptFile_
Encrypts a file withopenssl
-
_extractArchive_
Extract a compressed file -
_fileBasename_
Gets the basename of a file from a file name -
_fileContains_
Tests whether a file contains a given pattern -
_filePath_
Gets the absolute path to a file -
_fileExtension_
Gets the extension of a file -
_fileName_
Prints a filename from a path -
_json2yaml_
Convert JSON to YAML uses python -
_listFiles_
Find files in a directory. Use either glob or regex. -
_makeSymlink_
Creates a symlink and backs up a file which may be overwritten by the new symlink. If the exact same symlink already exists, nothing is done. -
_parseYAML_
Convert a YAML file into BASH variables for use in a shell script -
_printFileBetween_
Prints block of text in a file between two regex patterns -
_readFile_
Prints each line of a file -
_sourceFile_
Source a file into a script -
_createUniqueFilename_
Ensure a file to be created has a unique filename to avoid overwriting other files -
_yaml2json_
Convert a YAML file to JSON with python
macOS.bash
Functions useful when writing scripts to be run on macOS
-
_guiInput_
Ask for user input using a Mac dialog box -
_haveScriptableFinder_
Determine whether we can script the Finder or not -
_homebrewPath_
Adds Homebrew bin directory to PATH -
_useGNUUtils_
Add GNU utilities to PATH to allow consistent use of sed/grep/tar/etc. on MacOS
misc.bash
Miscellaneous functions
-
_acquireScriptLock_
Acquire script lock to prevent running the same script a second time before the first instance exits -
_detectLinuxDistro_
Detects the host computer's distribution of Linux -
_detectMacOSVersion_
Detects the host computer's version of macOS -
_detectOS_
Detect the the host computer's operating system -
_endspin_
Clears output from the spinner -
_execute_
Executes commands with safety and logging options. RespectsDRYRUN
andVERBOSE
flags. -
_findBaseDir_
Locates the real directory of the script being run. Similar to GNU readlink -n -
_generateUUID_
Generates a unique UUID -
_progressBar_
Prints a progress bar within a for/while loop -
_runAsRoot_
Run the requested command as root (via sudo if requested) -
_seekConfirmation_
Seek user input for yes/no question -
_spinner_
Creates a spinner within a for/while loop. -
_trapCleanup_
Cleans up after a trapped error.
services.bash
Functions to work with external services
-
_haveInternet_
Tests to see if there is an active Internet connection -
_httpStatus_
Report the HTTP status of a specified URL -
_pushover_
Sends a notification via Pushover (Requires API keys)
strings.bash
Functions for string manipulation
-
_cleanString_
Cleans a string of text -
_decodeHTML_
Decode HTML characters with sed. (Requires sed file) -
_decodeURL_
Decode a URL encoded string -
_encodeHTML_
Encode HTML characters with sed (Requires sed file) -
_encodeURL_
URL encode a string -
_escapeString_
Escapes a string by adding\
before special chars -
_lower_
Convert a string to lowercase -
_ltrim_
Removes all leading whitespace (from the left) -
_regexCapture_
Use regex to validate and parse strings -
_rtrim_
Removes all leading whitespace (from the right) -
_splitString_
Split a string based on a given delimeter -
_stringContains_
Tests whether a string matches a substring -
_stringRegex_
Tests whether a string matches a regex pattern -
_stripANSI_
Strips ANSI escape sequences from text -
_stripStopwords_
Removes common stopwords from a string using a list of sed replacements located in an external file. -
_trim_
Removes all leading/trailing whitespace -
_upper_
Convert a string to uppercase
template_utils.bash
Functions required to allow the script template and alert functions to be used
-
_makeTempDir_
Creates a temp directory to house temporary files -
_safeExit_
Cleans up temporary files before exiting a script -
_setPATH_
Add directories to $PATH so script can find executables
Coding conventions
- Function names use camel case surrounded by underscores:
_nameOfFunction_
- Local variable names use camel case with a starting underscore:
_localVariable
- Global variables are in ALL_CAPS with underscores seperating words
- Exceptions to the variable an function naming rules are made for alerting functions and colors to ease my speed of programming. (Breaking years of habits is hard...) I.e.
notice "Some log item: ${blue}blue text${reset}
Wherenotice
is a function and$blue
and$reset
are global variables but are lowercase. - Variables are always surrounded by quotes and brackets
"${1}"
(Overly verbose true, but a safe practice) - Formatting is provided by shfmt using 4 spaces for indentation
- All scripts and functions are fully Shellcheck compliant
- Where possible, I follow defensive BASH programming principles.
A Note on Code Reuse and Prior Art
I compiled these scripting utilities over many years without having an intention to make them public. As a novice programmer, I have Googled, GitHubbed, and StackExchanged a path to solve my own scripting needs. I often lift a function whole-cloth from a GitHub repo don't keep track of its original location. I have done my best within these files to recreate my footsteps and give credit to the original creators of the code when possible. Unfortunately, I fear that I missed as many as I found. My goal in making this repository public is not to take credit for the code written by others. If you recognize something that I didn't credit, please let me know.
License
MIT