Bash Tools Framework

• 1: Documentation
• 1.1: Framework Guides
• 1.1.1: Configuration Files
• 1.1.2: Docker Namespace
• 1.1.3: Best Practices
• 1.2: Framework Overview

1 - Documentation

1.1 - Framework Guides

General framework documentation explaining main framework features:

• Configuration files loading management
• Docker namespace usage
• Best practices and recipes

For creating your first binary file, see the Commands documentation.

1.1.1 - Configuration Files

Inspired by Evan “Hippy” Slatis work

1. Config file format and loading rules

Configuration files loading is following these rules or best practices:

• all env files are now loaded as properties file, it means bash arrays cannot be declared Eg: arrays should be converted to a string list separated by colon, and has to converted when needed.
• All .env files have the ability to override value by env variable
• all variables have to be written on one line, it is not possible to cut lines over multiple line
• First variable set takes precedence, so writing following file would result to VAR1=value1

VAR1=value1
VAR1=value2

2. Config file overloading values

• Best practice is to override variables only by
  • command argument –bash-framework-config to allow loading alternate env before other default files
  • command argument (–verbose, …) allows to override default displayed log level
  • in env files, always allow value to be overridden by prioritized variables
    • using bash variable default value mechanism, in following example, BASH_FRAMEWORK_LOG_LEVEL will be equal to 0 only if it hasn’t been set previously

BASH_FRAMEWORK_LOG_LEVEL="${BASH_FRAMEWORK_LOG_LEVEL:-0}"

• Provide –config argument to see resulting config file + information about order of loaded config files for debugging purpose.

• It is also possible to use environment variable, but highly discouraged to generalize this practice as it could lead to unwanted results if variables are not well scoped.

3. Config files loading order

The framework function Env::requireLoad loads the following files in this order if they are existing and are readable:

• files provided in BASH_FRAMEWORK_ENV_FILES env variable array
• ${FRAMEWORK_ROOT_DIR}/.framework-config if exists
• .framework-config from current directory if exists
• file from option –bash-framework-config

Options can override values provided by these env files:

• if –verbose or -v argument is passed, set BASH_FRAMEWORK_DISPLAY_LEVEL to 3 (INFO)
• if -vv argument is passed, set BASH_FRAMEWORK_DISPLAY_LEVEL to 4 (DEBUG)
• later on, will manage other kind of arguments
• additional files provided by this bash array variable, see below.
• framework default values file, see below.

Eg: additional environment files

BASH_FRAMEWORK_ENV_FILES=("${HOME}/.bash-tools/.env" "${HOME}/.env")

Eg: framework default values file

BASH_FRAMEWORK_LOG_LEVEL="${BASH_FRAMEWORK_LOG_LEVEL:-0}"
BASH_FRAMEWORK_DISPLAY_LEVEL="${BASH_FRAMEWORK_DISPLAY_LEVEL:-${__LEVEL_WARNING}}"
BASH_FRAMEWORK_LOG_FILE="${BASH_FRAMEWORK_LOG_FILE:-"${FRAMEWORK_ROOT_DIR}/logs/${SCRIPT_NAME}.log"}"
BASH_FRAMEWORK_LOG_FILE_MAX_ROTATION="${BASH_FRAMEWORK_LOG_FILE_MAX_ROTATION:-5}"

Activity diagram to explain how Env::requireLoad is working:

activity diagram source code.

1.1.2 - Docker Namespace

Usage example: try to pull image from 3 tags in order (from more specific or recent to the less one)

# try to pull image from 3 tags in order (from more specific or recent to the less one)
args=(
  'id.dkr.ecr.eu-west-1.amazonaws.com/bash-tools:d93e03d5ab9e127647f575855f605bd189ca8a56'
  'id.dkr.ecr.eu-west-1.amazonaws.com/bash-tools:branchName'
  'id.dkr.ecr.eu-west-1.amazonaws.com/bash-tools:master'
)
digestPulled="$(Docker::pullImage "${args[@]}")"

# build the image using eventual image pulled as cache
# image will be tagged bash-tools:latest upon successful build
args=(
  "." ".docker/Dockerfile" "bash-tools"
  # it's important to not double quote following instruction
  $(Docker::getBuildCacheFromArg ${digestPulled})
  # you can add any additional docker build arg as needed
  --build-arg USER_ID="$(id -u)"
  --build-arg GROUP_ID="$(id -g)"
)
Docker::buildImage "${args[@]}"

# tag the image with a remote tag
args=(
  "id.dkr.ecr.eu-west-1.amazonaws.com/bash-tools"
  "bash-tools:latest"
  # tags list
  "branchName" "d93e03d5ab9e127647f575855f605bd189ca8a56"
)
Docker::tagImage "${args[@]}"

# finally push the image
args=(
  "id.dkr.ecr.eu-west-1.amazonaws.com/bash-tools"
  "bash-tools:latest"
  # tags list
  "branchName" "d93e03d5ab9e127647f575855f605bd189ca8a56"
)
Docker::pushImage "${args[@]}"

1.1.3 - Best Practices

DISCLAIMER: Some of the best practices mentioned are not fully applied in this project as they were written during development.

1. Framework-Specific Recommendations

1.1. Using @embed Keyword

The @embed keyword is really useful to inline configuration files. However, to run framework functions using sudo, it is recommended to call the same binary but passing options to change the behavior. This way the content of the script file does not seem to be obfuscated.

1.2. Function Organization

Follow the framework’s naming conventions:

• Use Namespace::functionName pattern
• Place functions in appropriate namespace directories
• Include comprehensive documentation using shdoc annotations
• Write unit tests for every function

1.3. Testing Strategy

• Run tests on multiple Bash versions (4.4, 5.0, 5.3)
• Test on both Ubuntu and Alpine environments
• Use # bats test_tags=ubuntu_only for Ubuntu-specific tests
• Leverage stub/mock capabilities for external dependencies

1.2 - Framework Overview

This framework is a collection of several bash functions and commands that helps you to lint files, generate shell documentation, compile bash files, and many more, …

The Bash Tools Framework provides 150+ unit-tested functions organized by namespace. This section provides an overview of the available namespaces and their key functions.

1. Generated Documentation

The complete function reference documentation is automatically generated from source code using shdoc. This documentation includes detailed information about:

• Function parameters
• Return values
• Exit codes
• Usage examples
• Environment variables

To generate the documentation:

bin/doc

2. Framework Overview

Here an excerpt of the namespaces available in Bash tools framework:

• Apt: several functions to abstract the use of ubuntu apt-get function. these functions are using some default arguments and manage retry automatically.
  • Linux::Apt::addRepository
  • Linux::Apt::install
  • Linux::Apt::remove
  • Linux::Apt::update
• Args: functions to ease some recurrent arguments like -h|–help to display help
• Array: functions to ease manipulation of bash arrays like Array::clone or Array::contains that checks if an element is contained in an array
• Assert: various checks like
  • Assert::expectUser, Assert::expectNonRootUser, Assert::expectRootUser exits with message if current user is not the expected one
  • Assert::commandExists checks if command specified exists or exits with error message if not
  • Assert::windows determines if the script is executed under windows (git bash, wsl)
  • Assert::validPath checks if path provided is a valid linux path, it doesn’t have to exist yet
  • Assert::bashFrameworkFunction checks if given name respects naming convention of this framework’s functions
  • …
• Backup::file, Backup::dir allows to create a backup of a file or a directory in a folder configured in a .env file managed by the framework (see Env::requireLoad)
• Aws: Aim is to abstract the use of some aws cli commands, for the moment only Aws::imageExists has been implemented allowing to check that a docker image exists with tags provided on AWS ecr(AWS docker repository)
• Bats::installRequirementsIfNeeded allows to install bats vendor requirements for this project, it uses mainly the useful function Git::shallowClone
• Cache : various cache methods to provide files or env variable with expiration management
• Command::captureOutputAndExitCode calls a command capturing output and exit code and displaying it also to error output to follow command’s progress
• Conf : allows to manage the loading of .env file that contains configuration used by some functions of this framework.
• Database : abstraction of several mysql queries, like:
  • Database::dump dump db limited to optional table list
  • Database::query mysql query on a given db
  • Database::dropTable drop table if exists
  • Database::dropDb drop database if exists
  • Database::createDb create database if not already existing
  • Database::isTableExists check if table exists on given db
  • Database::ifDbExists check if given database exists
  • all these methods need to call Database::newInstance in order to reference target db connection
• Dns : various methods like Dns::pingHost or allowing etc/hosts manipulation.
• Docker : various docker cli abstractions that allowed to construct bin/buildPushDockerImage command.
• Env : functions allowing to load env variables or to alter them like Env::pathAppend allowing to add a bin path to PATH variable
• File : files and file paths manipulations.
• Filters : various functions to filter files using grep, awk or sed eg: Filters::bashFrameworkFunctions allows to find all the bash framework functions used in a file
• Git : provides git abstractions like Git::cloneOrPullIfNoChange, Git::pullIfNoChanges or Git::shallowClone
• Install : copy directory or file, backup them before if needed.
• Github : major feature is install automatically latest binary release using Github::upgradeRelease
• Log::display\* output colored message on error output and log the message
  • Log::fatal error message in red bold and exits with code 1
  • Log::displayError error message in red
  • Log::displayWarning warning message in yellow
  • Log::displayInfo info message in white on lightBlue
  • Log::displaySuccess success message in green
  • Log::displayDebug debug message in gray
• Log::log\* output message in a log file
  • Log::logError
  • Log::logWarning
  • Log::logInfo
  • Log::logSuccess
  • Log::logDebug
• Log::rotate automatically rotates the log file, this function is used internally by Log::log\* functions.
• OS: ubuntu related functions
• Profiles: methods mainly used by Bash-dev-env project that allows to indicate scripts list to install with the ability to include all the dependencies recursively. This file src/Profiles/lintDefinitions.sh is the precursor of a first bash interface implementation.
• Retry: retry a command on failure easily
• ShellDoc: this framework shell documentation generation
• Ssh: mainly Ssh::fixAuthenticityOfHostCantBeEstablished
• Sudo: executes command as sudo if needed
• UI
  • UI::askToContinue ask the user if he wishes to continue a process
  • UI::askYesNo ask the user a confirmation
  • UI::askToIgnoreOverwriteAbort ask the user to ignore(i), overwrite(o) or abort(a)
• Version
  • Version::checkMinimal ensure that command exists with expected version
  • Version::compare compares two versions
• Wsl: commands wslvar and wslpath are expensive, avoid multiple calls using cache
• src/_standalone regroups methods that do not respect framework naming conventions like assert_lines_count that is used to assert the number of lines of output in bats tests

3. Development Environment

3.1. Precommit hook

This repository uses pre-commit software to ensure every commits respects a set of rules specified by the .pre-commit-config.yaml file. It supposes pre-commit software is installed in your environment.

You also have to execute the following command to enable it:

pre-commit install --hook-type pre-commit --hook-type pre-push

3.2. UT

All the methods of this framework are unit tested, you can run the unit tests using the following command

./test.sh scrasnups/build:bash-tools-ubuntu-5.3 -r src -j 30

Launch UT on different environments:

./test.sh scrasnups/build:bash-tools-ubuntu-4.4 -r src -j 30
./test.sh scrasnups/build:bash-tools-ubuntu-5.0 -r src -j 30
./test.sh scrasnups/build:bash-tools-ubuntu-5.3 -r src -j 30
./test.sh scrasnups/build:bash-tools-alpine-4.4 -r src -j 30
./test.sh scrasnups/build:bash-tools-alpine-5.0 -r src -j 30
./test.sh scrasnups/build:bash-tools-alpine-5.3 -r src -j 30

3.3. Debug bats

use the following command:

vendor/bats/bin/bats -r src/Conf/loadNearestFile.bats --trace --verbose-run --filter "Conf::loadNearestFileFileFoundInDir1"

3.4. connect to container manually

Alpine with bash version 4.4

docker run --rm -it -w /bash -v "$(pwd):/bash" --entrypoint="" --user 1000:1000 bash-tools-alpine-4.4-user bash

Ubuntu with bash version 5.1

docker run --rm -it -w /bash -v "$(pwd):/bash" --entrypoint="" --user 1000:1000 bash-tools-ubuntu-5.1-user bash

3.5. auto generated bash doc

generated by running

bin/doc

3.6. github page

The web page uses Hugo with the Docsy theme to generate a static documentation site.

To preview the website locally, you need to clone my-documents repository

git clone git@github.com:fchastanet/my-documents.git

And run the following command from the root of this repository:

SITE=bash-tools-framework make start-site

Navigate to http://localhost:1313/bash-tools-framework/

4. Troubleshooting

4.1. compile.bats embed not working on alpine investigation

exit code 127 is returned but process seems to go until the end. This error only occurs on alpine.

commands to compile and debug:

# run docker alpine interactively
docker run --rm -it -w /bash -v "$(pwd):/bash" --entrypoint="" --user 1000:1000 build:bash-tools-alpine-4.4-user bash

# launch bats test that fails
vendor/bats/bin/bats -r src/_binaries/compile.bats --filter embed

# launch directly compile command that returns the same exit code
bin/compile src/_binaries/testsData/bin/embed.sh --template-dir src --bin-dir bin --root-dir $PWD --src-dir src/_binaries/testsData/src
echo $? # prints 127

# try to get more logs
KEEP_TEMP_FILES=1 BASH_FRAMEWORK_DISPLAY_LEVEL=4 bin/compile \
  src/_binaries/testsData/bin/embed.sh \
  --template-dir src \
  --bin-dir bin \
  --root-dir "${PWD}" \
  --src-dir src/_binaries/testsData/src

# try to use strace
docker run --rm -it \
  -w /bash -v "$(pwd):/bash" \
  --entrypoint="" \
  build:bash-tools-alpine-4.4-user bash
apk update
apk add strace

Strace didn’t helped me a lot. But as I added recently this option shopt -u lastpipe, I removed it from compile binary and the issue disappeared.

As I was suspecting the while piped inside Compiler::Embed::inject. I added the following code in this function to remove the tracing just after the error occurs:

trap 'set +x' EXIT
set -x

It allows me to find that the last command executed was read -r line.

Finally I understand that the issue comes when read -r line exits with code 1 because of end of file.

previous simplified code:

cat file | {
  local line
  while IFS="" read -r line; do
    # ...
  done
}

Resulting in exit code 127 because of pipe and shopt -u lastpipe.

Fixed code is to remove error if :

cat file | {
  local line
  while true; do
    local status=0
    IFS="" read -r line || status=$?
    if [[ "${status}" = "1" ]]; then
      # end of file
      return 0
    elif [[ "${status}" != "0" ]]; then
      # other error
      return "${status}"
    fi
    # ...
  done
}

5. External Resources

For comprehensive guides on Bash best practices, please refer to these documents:

• General Bash Best Practices - Core Bash scripting practices
• Linux Commands Best Practices - Effective Linux command usage
• Bats Best Practices - Testing best practices with Bats

6. Related Projects

This framework is part of a suite of projects:

• My Documents
• Bash Tools
• Bash Dev Env
• Bash Compiler