This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Documentation

Comprehensive guides and best practices

1. Browse Documentation

Use the sidebar navigation to explore:

  • Bash Scripts - Best practices for writing efficient Bash scripts
  • How-To Guides - Step-by-step guides for Docker, Jenkins, and other technologies
  • Reference Lists - Curated lists of tools and resources
  • Other Projects - Related documentation sites and tools

2. My recent articles

Saml2Aws Setup

Guide to setting up and using Saml2Aws for AWS access

How to Write Dockerfiles

Best practices for writing efficient and secure Dockerfiles

How to Write Docker Compose Files

Guide to writing and organizing Docker Compose files

Debug Hugo

Debug Hugo: Quick guide to locate Hugo templates, use templateMetrics, override priorities, and fix common Docsy/Hugo template issues for faster debugging.

Basic Best Practices
Basic Best Practices

Foundational best practices for writing Bash scripts

Linux Commands Best Practices

Best practices for using Linux commands in Bash scripts

Bats Testing Framework

Best practices for testing Bash scripts with Bats framework

Quick Reference - Hugo Site Development
Quick Reference - Hugo Site Development

A quick reference guide for developing and maintaining the Hugo documentation site

How Jenkins Works

Understanding Jenkins architecture and concepts

Alpine vs Ubuntu Image: Performance Comparison for Docker Builds

A detailed comparison between Alpine and Ubuntu Docker images, focusing on performance differences in build times and test execution.

1 - My Documents

Articles about this repository, its structure, and how to use it effectively

1. Available Guides

  • My Documents static site generation - Analysis of migrating from Docsify to an SEO-optimized static site generator
  • My Documents Multi repositories Site Generation - Analysis of generating a documentation site from multiple repositories using Hugo and GitHub Pages
  • My Documents Technical Architecture - Overview of the technical architecture of this documentation site, including the use of Hugo, GitHub Pages, and CI/CD pipelines
  • My Documents Trigger Workflow - Guide on how to trigger the documentation build workflow using GitHub Actions

2. Getting Started

Select a documentation topic from the sidebar to begin.

Articles in this section

TitleDescriptionUpdated
Quick Reference - Hugo Site DevelopmentA quick reference guide for developing and maintaining the Hugo documentation site2026-05-11 00:18:45 +0200 +0200
Multi repositories Site GenerationComprehensive documentation of the Hugo migration for multi-site documentation2026-05-04 22:23:53 +0200 +0200
Trigger Reusable Workflow DocumentationOverview of the technical architecture and implementation details of the My Documents reusable workflow for triggering documentation builds2026-05-04 22:23:53 +0200 +0200
Technical ArchitectureComplete technical architecture guide for the Hugo documentation system with reusable GitHub Actions2026-04-21 17:55:47 +0200 +0200
Deploy Static Site on Cloudflare PagesNo description2026-04-12 08:00:00 +0100 +0100
Static Site Generation Migration AnalysisAnalysis of migrating from Docsify to an SEO-optimized static site generator2026-02-22 08:00:00 +0100 +0100

1.1 - Technical Architecture

Complete technical architecture guide for the Hugo documentation system with reusable GitHub Actions

1. Overview

The my-documents repository provides a reusable GitHub Action for building and deploying Hugo-based documentation sites using the Docsy theme. This architecture enables multiple documentation repositories to share common configurations, layouts, and assets while maintaining their independence.

1.1. Key Features

  • Reusable GitHub Action: Single workflow definition used across multiple repositories
  • Hugo Go Modules: Share layouts, assets, and configurations without file copying
  • No Authentication Complexity: Uses standard GITHUB_TOKEN (no GitHub Apps or PATs required)
  • Independent Deployments: Each repository controls its own build and deployment
  • Shared Theme Consistency: All sites use the same Docsy theme with consistent styling
  • SEO Optimized: Built-in structured data, meta tags, and sitemap generation

1.2. Managed Documentation Sites

SiteRepositoryLive URL
My Documentsfchastanet/my-documentshttps://devlab.top/
Bash Compilerfchastanet/bash-compilerhttps://bash-compiler.devlab.top/
Bash Toolsfchastanet/bash-toolshttps://bash-tools.devlab.top/
Bash Tools Frameworkfchastanet/bash-tools-frameworkhttps://bash-tools-framework.devlab.top/
Bash Dev Envfchastanet/bash-dev-envhttps://bash-dev-env.devlab.top/

2. Building Locally

2.1. Prerequisites

Install the required tools:

  • Hugo Extended v0.160.1 or higher (with Go support)
  • Go 1.24 or higher
  • Git

2.2. Quick Start

# Clone the repository
git clone https://github.com/fchastanet/my-documents.git
cd my-documents

# Download Hugo modules
hugo mod get -u

# Start local development server
hugo server -D

# Open browser to http://localhost:1313/my-documents/

The site will auto-reload when you edit content in content/docs/.

2.3. Building for Production

# Build optimized static site
hugo --minify

# Output is in public/ directory
ls -la public/

3. Reusable Action Architecture

3.1. Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│ my-documents Repository (Public)                                │
│                                                                 │
│  ├── .github/workflows/                                         │
│  │   ├── build-site-action.yml  ← Reusable action definition   │
│  │   └── build-site.yml          ← Own site build              │
│  │                                                              │
│  ├── configs/                                                   │
│  │   └── _base.yaml              ← Shared base configuration   │
│  │                                                              │
│  └── shared/                                                    │
│      ├── layouts/                ← Shared Hugo templates       │
│      ├── assets/                 ← Shared SCSS, CSS, JS        │
│      └── archetypes/             ← Content templates           │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘
                           │ Hugo Go Module Import
        ┌──────────────────┼──────────────────┬──────────────────┐
        │                  │                  │                  │
        ▼                  ▼                  ▼                  ▼
┌───────────────┐  ┌───────────────┐  ┌───────────────┐  ┌──────────────┐
│ bash-compiler │  │  bash-tools   │  │ bash-dev-env  │  │ Other Repos  │
│               │  │               │  │               │  │              │
│ go.mod        │  │ go.mod        │  │ go.mod        │  │ go.mod       │
│ hugo.yaml     │  │ hugo.yaml     │  │ hugo.yaml     │  │ hugo.yaml    │
│ content/      │  │ content/      │  │ content/      │  │ content/     │
│               │  │               │  │               │  │              │
│ .github/      │  │ .github/      │  │ .github/      │  │ .github/     │
│ workflows/    │  │ workflows/    │  │ workflows/    │  │ workflows/   │
│ build-site    │  │ build-site    │  │ build-site    │  │ build-site   │
│ .yml          │  │ .yml          │  │ .yml          │  │ .yml         │
│     │         │  │     │         │  │     │         │  │     │        │
│     └─────────┼──┼─────┼─────────┼──┼─────┼─────────┼──┼─────┘        │
│               │  │               │  │               │  │              │
└───────────────┘  └───────────────┘  └───────────────┘  └──────────────┘
        │                  │                  │                  │
        └──────────────────┴──────────────────┴──────────────────┘
                           │ Calls reusable action
              fchastanet/my-documents/
              .github/workflows/build-site-action.yml
              ┌────────────────────────┐
              │ 1. Checkout repo       │
              │ 2. Setup Hugo          │
              │ 3. Setup Go            │
              │ 4. Download modules    │
              │ 5. Build with Hugo     │
              │ 6. Deploy to Pages     │
              └────────────────────────┘

3.2. How It Works

The reusable action architecture follows this workflow:

  1. Developer pushes content to a documentation repository (e.g., bash-compiler)
  2. GitHub Actions triggers the build-site.yml workflow in that repository
  3. Workflow calls my-documents/.github/workflows/build-site-action.yml (reusable action)
  4. Hugo downloads modules including my-documents for shared resources
  5. Hugo builds site using merged configuration (base + site-specific overrides)
  6. GitHub Pages deploys the static site from the build artifact

3.3. Key Benefits

  • Zero Authentication Setup: No GitHub Apps, deploy keys, or PAT tokens required
  • Independent Control: Each repository owns its build and deployment
  • Shared Consistency: All sites use the same theme, layouts, and styling
  • Easy Maintenance: Update reusable action once, all sites benefit
  • Fast Builds: Parallel execution across repositories (~30-60s per site)
  • Simple Testing: Test locally with standard hugo server command

4. Creating a New Documentation Site

4.1. Prerequisites

Before creating a new documentation site, ensure you have:

  • Admin access to create a new repository or use existing repository
  • Basic understanding of Hugo and Markdown
  • Hugo Extended and Go installed locally for testing

4.2. Step-by-Step Guide

4.2.1. Create Content Structure

Create the standard Hugo directory structure in your repository:

# Create required directories
mkdir -p content/docs
mkdir -p static

# Create homepage
cat >content/_index.md <<'EOF'
---
title: My Project Documentation
description: Welcome to My Project documentation
---

# Welcome to My Project

This is the documentation homepage.
EOF

# Create first documentation page
cat >content/docs/_index.md <<'EOF'
---
title: Documentation
linkTitle: Docs
weight: 20
menu:
  main:
    weight: 20
---

# Documentation

Welcome to the documentation section.
EOF

4.2.2. Add go.mod for Hugo Modules

Create go.mod in the repository root:

module github.com/YOUR-USERNAME/YOUR-REPO

go 1.24

require (
 github.com/google/docsy v0.11.0 // indirect
 github.com/google/docsy/dependencies v0.7.2 // indirect
 github.com/fchastanet/my-documents master // indirect
)

Replace YOUR-USERNAME/YOUR-REPO with your actual repository path.

4.2.3. Create hugo.yaml with Base Import

Create hugo.yaml in the repository root:

# Import base configuration from my-documents
imports:
  - path: github.com/fchastanet/my-documents/configs/_base.yaml

# Site-specific overrides
baseURL: https://YOUR-USERNAME.github.io/YOUR-REPO
title: Your Project Documentation
locale: en-us

# Module configuration
module:
  # Import my-documents for shared resources
  imports:
    - path: github.com/fchastanet/my-documents
      mounts:
        # Mount shared layouts
        - source: shared/layouts
          target: layouts
        # Mount shared assets
        - source: shared/assets
          target: assets
        # Mount shared archetypes
        - source: shared/archetypes
          target: archetypes
    - path: github.com/google/docsy
    - path: github.com/google/docsy/dependencies

# Site-specific parameters
params:
  description: Documentation for Your Project

  # Customize theme colors
  ui:
    navbar_bg_color: '#007bff'  # Blue - choose your color
    sidebar_menu_compact: false

  # Repository configuration
  github_repo: https://github.com/YOUR-USERNAME/YOUR-REPO
  github_branch: master

  # Enable search
  offlineSearch: true

Replace placeholders:

  • YOUR-USERNAME with your GitHub username
  • YOUR-REPO with your repository name
  • Adjust navbar_bg_color for your preferred theme color

4.2.4. Add build-site.yml Workflow

Create .github/workflows/build-site.yml:

name: Build and Deploy Documentation

on:
  push:
    branches: [master]
    paths:
      - content/**
      - static/**
      - hugo.yaml
      - go.mod
      - .github/workflows/build-site.yml
  workflow_dispatch:

# Required permissions for GitHub Pages deployment
permissions:
  contents: read
  pages: write
  id-token: write

# Prevent concurrent deployments
concurrency:
  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
  cancel-in-progress: true

jobs:
  build-deploy:
    name: Build and Deploy
    uses: fchastanet/my-documents/.github/workflows/build-site-action.yml@master
    with:
      site-name: YOUR-REPO
      base-url: https://YOUR-USERNAME.github.io/YOUR-REPO
      checkout-repo: YOUR-USERNAME/YOUR-REPO
    permissions:
      contents: read
      pages: write
      id-token: write

Replace:

  • YOUR-USERNAME with your GitHub username
  • YOUR-REPO with your repository name

Important: Ensure the workflow file has Unix line endings (LF), not Windows (CRLF).

4.2.5. Configure GitHub Pages

In your repository settings:

  1. Navigate to SettingsPages
  2. Under Source, select GitHub Actions
  3. Click Save

4.2.6. Test and Deploy

Test locally first:

# Download modules
hugo mod get -u

# Start development server
hugo server -D

# Verify site at http://localhost:1313/

Deploy to GitHub Pages:

# Commit all files
git add .
git commit -m "Add Hugo documentation site"

# Push to trigger workflow
git push origin master

Monitor deployment:

  1. Go to Actions tab in your repository
  2. Watch the “Build and Deploy Documentation” workflow
  3. Once complete (green checkmark), visit your site at https://YOUR-USERNAME.github.io/YOUR-REPO

4.3. Post-Creation Checklist

After creating your site, verify:

  • Hugo builds locally without errors (hugo --minify)
  • Development server runs (hugo server -D)
  • All pages render correctly
  • Navigation menu shows correct structure
  • Search works (if enabled)
  • GitHub Actions workflow completes successfully
  • Site deploys to GitHub Pages
  • All links work on live site
  • Mobile view looks correct
  • Theme colors match expectations

5. GitHub Configuration

5.1. GitHub Pages Settings

Required Configuration:

  1. Source: GitHub Actions (NOT a branch)
  2. Custom Domain: Optional
  3. Enforce HTTPS: Recommended (enabled by default)

Why GitHub Actions Source?

Using GitHub Actions as the Pages source allows workflows to deploy directly using the actions/deploy-pages action. This is simpler than pushing to a gh-pages branch and more secure.

5.2. Workflow Permissions

Your build-site.yml workflow requires these permissions:

permissions:
  contents: read      # Read repository content
  pages: write        # Deploy to GitHub Pages
  id-token: write     # OIDC token for deployment

These permissions are:

  • Scoped to the workflow: Only this workflow has these permissions
  • Automatic: No manual configuration required
  • Secure: Uses GitHub’s OIDC authentication

5.3. No Secrets Required

Unlike traditional approaches, this architecture requires zero secrets:

  • ❌ No GitHub App credentials
  • ❌ No Personal Access Tokens (PAT)
  • ❌ No Deploy Keys
  • ✅ Standard GITHUB_TOKEN provided automatically

The workflow uses GitHub’s built-in authentication, making setup simple and secure.

6. Hugo Configuration Details

6.1. go.mod Structure

The go.mod file declares Hugo module dependencies:

module github.com/fchastanet/bash-compiler

go 1.24

require (
 github.com/google/docsy v0.11.0 // indirect
 github.com/google/docsy/dependencies v0.7.2 // indirect
 github.com/fchastanet/my-documents master // indirect
)

Key Components:

  • Module name: Must match your repository path
  • Go version: 1.24 or higher recommended
  • Docsy theme: Version 0.11.0 (update as needed)
  • Docsy dependencies: Bootstrap, Font Awesome, etc.
  • my-documents: Provides shared layouts and assets

Updating Modules:

# Update all modules to latest versions
hugo mod get -u

# Update specific module
hugo mod get -u github.com/google/docsy

# Tidy module dependencies
hugo mod tidy

6.2. hugo.yaml Structure

The hugo.yaml configuration file has two main parts:

6.2.1. Imports Section

# Import base configuration from my-documents
imports:
  - path: github.com/fchastanet/my-documents/configs/_base.yaml

This imports shared configuration including:

  • Hugo modules setup
  • Markup and syntax highlighting
  • Output formats (HTML, RSS, sitemap)
  • Default theme parameters
  • Language and i18n settings

6.2.2. Site-Specific Configuration

Override base settings for your site:

baseURL: https://bash-compiler.devlab.top
title: Bash Compiler Documentation
locale: en-us

module:
  imports:
    - path: github.com/fchastanet/my-documents
      mounts:
        - source: shared/layouts
          target: layouts
        - source: shared/assets
          target: assets
        - source: shared/archetypes
          target: archetypes
    - path: github.com/google/docsy
    - path: github.com/google/docsy/dependencies

params:
  description: Documentation for Bash Compiler
  ui:
    navbar_bg_color: '#007bff'
  github_repo: https://github.com/fchastanet/bash-compiler
  offlineSearch: true

6.3. Configuration Inheritance

Hugo merges configurations in this order:

  1. Base configuration (_base.yaml from my-documents)
  2. Site-specific overrides (your hugo.yaml)

Merge Behavior:

  • Scalar values: Site-specific overrides base
  • Objects: Deep merge (keys combined)
  • Arrays: Site-specific replaces base entirely

Example:

# Base (_base.yaml)
params:
  ui:
    showLightDarkModeMenu: true
    navbar_bg_color: "#563d7c"
  copyright: "My Documents"

# Site-specific (hugo.yaml)
params:
  ui:
    navbar_bg_color: "#007bff"
  copyright: "Bash Compiler"

# Result (merged)
params:
  ui:
    showLightDarkModeMenu: true    # From base
    navbar_bg_color: "#007bff"     # Overridden
  copyright: "Bash Compiler"       # Overridden

6.4. Site-Specific Overrides

Common parameters to override per site:

Required:

baseURL: https://YOUR-USER.github.io/YOUR-REPO
title: Your Site Title
params:
  description: Your site description
  github_repo: https://github.com/YOUR-USER/YOUR-REPO

Optional Theme Customization:

params:
  ui:
    navbar_bg_color: '#007bff'      # Navbar color
    sidebar_menu_compact: false      # Sidebar style
    navbar_logo: true                # Show logo in navbar

  links:
    user:
      - name: GitHub
        url: https://github.com/YOUR-USER/YOUR-REPO
        icon: fab fa-github

Navigation Menu:

menu:
  main:
    - name: Documentation
      url: /docs/
      weight: 10
    - name: Blog
      url: /blog/
      weight: 20

7. Workflow Configuration

7.1. build-site.yml Structure

The build-site.yml workflow in each repository calls the reusable action:

name: Build and Deploy Documentation

on:
  push:
    branches: [master]
    paths:
      - content/**
      - static/**
      - hugo.yaml
      - go.mod
      - .github/workflows/build-site.yml
  workflow_dispatch:

permissions:
  contents: read
  pages: write
  id-token: write

concurrency:
  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
  cancel-in-progress: true

jobs:
  build-deploy:
    name: Build and Deploy
    uses: fchastanet/my-documents/.github/workflows/build-site-action.yml@master
    with:
      site-name: bash-compiler
      base-url: https://bash-compiler.devlab.top
      checkout-repo: fchastanet/bash-compiler
    permissions:
      contents: read
      pages: write
      id-token: write

7.2. Calling the Reusable Action

The uses keyword calls the reusable action:

uses: fchastanet/my-documents/.github/workflows/build-site-action.yml@master

Format: OWNER/REPO/.github/workflows/WORKFLOW.yml@REF

  • OWNER/REPO: fchastanet/my-documents (the provider repository)
  • WORKFLOW: build-site-action.yml (the reusable workflow file)
  • REF: master (or specific tag/commit for stability)

7.3. Required Parameters

These parameters must be provided with with:

with:
  site-name: bash-compiler
  base-url: https://bash-compiler.devlab.top
  checkout-repo: fchastanet/bash-compiler

Parameter Details:

  • site-name: Identifier for the site (used in artifacts and jobs)
  • base-url: Full base URL where site will be deployed
  • checkout-repo: Repository to checkout (format: owner/repo)

7.4. Optional Parameters

The reusable action may support additional parameters:

with:
  hugo-version: 0.160.1           # Default: latest
  go-version: '1.24'              # Default: 1.24
  extended: true                  # Default: true (Hugo Extended)
  working-directory: .            # Default: repository root

Check the reusable action definition for all available parameters.

7.5. Triggers Configuration

Trigger on Content Changes:

on:
  push:
    branches: [master]
    paths:
      - content/**
      - static/**
      - hugo.yaml
      - go.mod

This triggers the workflow only when documentation-related files change, saving CI minutes.

Trigger Manually:

on:
  workflow_dispatch:

Allows manual workflow runs from the GitHub Actions UI.

Trigger on Schedule:

on:
  schedule:
    - cron: 0 0 * * 0    # Weekly on Sunday at midnight UTC

Useful for rebuilding with updated dependencies.

7.6. Permissions Details

Why These Permissions?

permissions:
  contents: read      # Clone repository and read content
  pages: write        # Upload artifact and deploy to Pages
  id-token: write     # Generate OIDC token for deployment

Scope:

  • Permissions apply only to this workflow
  • Defined at both workflow and job level for clarity
  • More restrictive than repository-wide settings

Security Note:

Never grant contents: write unless absolutely necessary. The reusable action only needs read access.

8. Shared Resources Access

8.1. Hugo Go Modules Setup

Hugo modules enable sharing resources across repositories without file copying.

Module Declaration (go.mod):

require (
 github.com/fchastanet/my-documents master // indirect
)

Download Modules:

# Download all declared modules
hugo mod get -u

# Verify modules downloaded
hugo mod graph

8.2. Accessing Layouts from my-documents

Module Mount Configuration:

module:
  imports:
    - path: github.com/fchastanet/my-documents
      mounts:
        - source: shared/layouts
          target: layouts

Available Layouts:

shared/layouts/
├── partials/
│   └── hooks/
│       └── head-end.html         # SEO meta tags, JSON-LD
├── shortcodes/
│   └── custom-shortcode.html     # Custom shortcodes
└── _default/
    └── baseof.html               # Optional: base template override

Using Shared Partials:

<!-- In your custom layout -->
{{ partial "hooks/head-end.html" . }}

Override Priority:

  1. Local layouts/ directory (highest priority)
  2. Mounted shared/layouts/ from my-documents
  3. Docsy theme layouts (lowest priority)

8.3. Accessing Assets from my-documents

Module Mount Configuration:

module:
  imports:
    - path: github.com/fchastanet/my-documents
      mounts:
        - source: shared/assets
          target: assets

Available Assets:

shared/assets/
└── scss/
    └── _variables_project.scss   # SCSS variables

Using Shared SCSS:

// Auto-imported by Docsy
// Defines custom variables used across all sites
$primary: #007bff;
$secondary: #6c757d;

Override Site-Specific Styles:

Create assets/scss/_variables_project.scss in your repository:

// Override specific variables
$primary: #ff6600;  // Orange theme

// Import base variables for other defaults
@import "shared/scss/variables_project";

8.4. Accessing Archetypes from my-documents

Module Mount Configuration:

module:
  imports:
    - path: github.com/fchastanet/my-documents
      mounts:
        - source: shared/archetypes
          target: archetypes

Available Archetypes:

shared/archetypes/
├── default.md       # Default content template
└── docs.md          # Documentation page template

Using Archetypes:

# Create new page using docs archetype
hugo new content/docs/guide.md

# Uses shared/archetypes/docs.md template

Archetype Example (docs.md):

---
title: "{{ replace .Name "-" " " | title }}"
description: ""
weight: 10
categories: []
tags: []
---

## 9. Overview

Brief overview of this topic.

## 10. Details

Detailed content here.

10.1. Module Mounts Configuration

Complete mounts example:

module:
  imports:
    # Mount my-documents shared resources
    - path: github.com/fchastanet/my-documents
      mounts:
        - source: shared/layouts
          target: layouts
        - source: shared/assets
          target: assets
        - source: shared/archetypes
          target: archetypes

    # Mount Docsy theme
    - path: github.com/google/docsy
      disable: false

    # Mount Docsy dependencies (Bootstrap, etc.)
    - path: github.com/google/docsy/dependencies
      disable: false

Mount Options:

  • source: Path in the module repository
  • target: Where to mount in your site
  • disable: Set to true to temporarily disable

Debugging Mounts:

# Show module dependency graph
hugo mod graph

# Verify mounts configuration
hugo config mounts

11. Troubleshooting

11.1. Workflow Not Running

Problem: Workflow doesn’t trigger on push

Solutions:

  1. Check file paths in trigger:

    on:
      push:
        paths:
          - content/**
          - static/**
          - hugo.yaml
    

    Ensure changed files match these patterns.

  2. Verify branch name:

    on:
      push:
        branches: [master]  # Check your default branch name
    
  3. Check workflow syntax:

    # Validate YAML syntax
    yamllint .github/workflows/build-site.yml
    
  4. Permissions issue: Ensure Actions are enabled in repository settings:

    • Settings → Actions → General → “Allow all actions and reusable workflows”

11.2. Hugo Build Failures

Problem: Hugo build fails with errors

Common Causes and Solutions:

11.2.1. Missing Modules

Error: module "github.com/fchastanet/my-documents" not found

Solution:

# Ensure module declared in go.mod
hugo mod get -u

# Verify modules
hugo mod graph

11.2.2. Configuration Errors

Error: failed to unmarshal YAML

Solution:

# Validate YAML syntax
yamllint hugo.yaml

# Check Hugo config
hugo config

11.2.3. Front Matter Errors

Error: invalid front matter

Solution:

<!-- Ensure front matter uses valid YAML -->
---
title: "My Page"
date: 2024-02-22
draft: false
---

11.2.4. Template Errors

Error: template: partial "missing.html" not found

Solution:

# Check partial exists in layouts/partials/
ls shared/layouts/partials/

# Verify module mounts
hugo config mounts

11.3. Hugo Modules Issues

Problem: Modules not updating or wrong version

Solutions:

  1. Clean module cache:

    hugo mod clean
    hugo mod get -u
    
  2. Verify module versions:

    # Show dependency graph
    hugo mod graph
    
    # Check go.sum for versions
    cat go.sum
    
  3. Force module update:

    # Remove go.sum and rebuild
    rm go.sum
    hugo mod get -u
    hugo mod tidy
    
  4. Check module path:

    # Ensure correct repository path
    imports:
      - path: github.com/fchastanet/my-documents
    

11.4. Deployment Failures

Problem: Build succeeds but deployment fails

Solutions:

  1. Check Pages source:

    • Settings → Pages → Source must be “GitHub Actions”
  2. Verify permissions:

    permissions:
      contents: read
      pages: write
      id-token: write
    
  3. Check deployment logs:

    • Actions tab → Click workflow run → Expand “Deploy to GitHub Pages” step
  4. Concurrency conflict:

    concurrency:
      group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
      cancel-in-progress: true  # Cancel in-progress runs to ensure only the latest commit is deployed
    
  5. Artifact upload size:

    # Check public/ directory size
    du -sh public/
    
    # GitHub has 10GB limit per artifact
    # Optimize images and remove unnecessary files
    

Problem: Broken links or missing pages

Solutions:

  1. Check relative links:

    <!-- Correct -->
    [Guide](/docs/guide/)
    
    <!-- Incorrect -->
    [Guide](docs/guide/)  <!-- Missing leading slash -->
    
  2. Verify baseURL:

    # Must match deployment URL exactly
    baseURL: https://username.github.io/repo-name
    
  3. Check content organization:

    content/
    └── en/
        ├── _index.md
        └── docs/
            ├── _index.md
            └── guide.md
    
  4. Front matter issues:

    ---
    title: "Guide"
    # Check for typos in keys
    linkTitle: "User Guide"
    weight: 10
    ---
    
  5. Test links locally:

    hugo server -D
    # Check all links work at http://localhost:1313
    

11.6. Debugging Checklist

When troubleshooting, work through this checklist:

  • Local build succeeds: hugo --minify completes without errors
  • Modules downloaded: hugo mod graph shows correct dependencies
  • Configuration valid: hugo config outputs without errors
  • Workflow syntax valid: YAML linter passes
  • Permissions correct: Workflow has pages: write permission
  • Pages source configured: GitHub Pages source is “GitHub Actions”
  • Actions enabled: Repository allows GitHub Actions
  • Branch correct: Workflow triggers on correct branch
  • Paths correct: Changed files match workflow path filters
  • Artifacts created: Workflow creates and uploads artifact
  • Deployment job runs: Separate deployment job executes
  • Content accessible: Files in content/ directory render
  • Links work: Internal navigation functions correctly

Verbose Build Output:

# Local debugging with verbose output
hugo --minify --verbose --debug

# Check Hugo environment
hugo env

Check GitHub Actions Logs:

  1. Go to repository → Actions tab
  2. Click failing workflow run
  3. Expand each step to see detailed output
  4. Look for ERROR or WARN messages

12. Advanced Topics

12.1. Per-Site Theme Customization

Each site can customize the Docsy theme while maintaining shared base styles.

Color Customization:

# hugo.yaml
params:
  ui:
    navbar_bg_color: '#007bff'     # Blue navbar
    sidebar_bg_color: '#f8f9fa'    # Light gray sidebar
    navbar_text_color: '#ffffff'   # White text

Custom SCSS Variables:

Create assets/scss/_variables_project.scss in your repository:

// Override primary color
$primary: #ff6600;
$secondary: #6c757d;

// Custom navbar height
$navbar-height: 70px;

// Import base variables for other defaults
@import "shared/scss/variables_project";

Custom Layouts:

Override specific templates by creating them locally:

layouts/
├── _default/
│   └── single.html          # Custom single page layout
├── partials/
│   └── navbar.html          # Custom navbar
└── shortcodes/
    └── callout.html         # Custom shortcode

Priority Order:

  1. Local layouts/ (highest)
  2. Mounted shared/layouts/ from my-documents
  3. Docsy theme layouts (lowest)

12.2. SEO Metadata

Shared SEO features are provided via shared/layouts/partials/hooks/head-end.html:

Automatic SEO Tags:

  • Open Graph meta tags
  • Twitter Card tags
  • JSON-LD structured data
  • Canonical URLs
  • Sitemap generation

Configure per Page:

---
title: "My Guide"
description: "Comprehensive guide to using the tool"
images: ["/images/guide-preview.png"]
---

Site-Wide SEO:

# hugo.yaml
params:
  description: Default site description
  images: [/images/site-preview.png]

  # Social links for structured data
  github_repo: https://github.com/user/repo

  # Google Analytics (optional)
  google_analytics: G-XXXXXXXXXX

Verify SEO:

# Check generated meta tags
hugo server -D
curl http://localhost:1313/page/ | grep -A5 "og:"

12.3. Menu Customization

Main Menu Configuration:

# hugo.yaml
menu:
  main:
    - name: Documentation
      url: /docs/
      weight: 10
    - name: About
      url: /about/
      weight: 20
    - name: GitHub
      url: https://github.com/user/repo
      weight: 30
      pre: <i class='fab fa-github'></i>

Per-Page Menu Entry:

---
title: "API Reference"
menu:
  main:
    name: "API"
    weight: 15
    parent: "Documentation"
---

Sidebar Menu:

The sidebar menu is automatically generated from content structure. Control it with:

---
title: "Section"
weight: 10              # Order in menu
linkTitle: "Short Name" # Display name (optional)
---

Disable Menu Item:

---
title: "Hidden Page"
menu:
  main:
    weight: 0
_build:
  list: false
  render: true
---

13. Contributing

13.1. How to Contribute to Reusable Action

The reusable action is defined in my-documents/.github/workflows/build-site-action.yml.

Contributing Process:

  1. Fork the repository:

    gh repo fork fchastanet/my-documents --clone
    cd my-documents
    
  2. Create a feature branch:

    git checkout -b feature/improve-action
    
  3. Make changes:

    • Edit .github/workflows/build-site-action.yml
    • Update documentation if needed
    • Test changes thoroughly
  4. Commit using conventional commits:

    git commit -m "feat(workflows): add support for custom Hugo version"
    
  5. Push and create PR:

    git push origin feature/improve-action
    gh pr create --title "Add custom Hugo version support"
    

13.2. Testing Changes

Test Reusable Action Changes:

  1. Push changes to your fork:

    git push origin feature/improve-action
    
  2. Update dependent repository to use your fork:

    # .github/workflows/build-site.yml
    jobs:
      build-deploy:
        uses: |-
          YOUR-USERNAME/my-documents/.github/workflows/build-site-action.yml@feature/improve-action
    
  3. Trigger workflow:

    git commit --allow-empty -m "Test workflow"
    git push
    
  4. Verify results:

    • Check Actions tab for workflow run
    • Ensure build and deployment succeed
    • Test deployed site

Test Configuration Changes:

# Test base configuration changes
cd my-documents
hugo server -D

# Test site-specific overrides
cd bash-compiler
hugo mod get -u
hugo server -D

Test Shared Resources:

# Add new shared layout
echo '<meta name="test" content="value">' >shared/layouts/partials/test.html

# Rebuild dependent site
cd ../bash-compiler
hugo mod clean
hugo mod get -u
hugo server -D

# Verify partial available
curl http://localhost:1313 | grep 'name="test"'

13.3. Best Practices

Workflow Development:

  • Test thoroughly: Changes affect all dependent sites
  • Use semantic versioning: Tag stable versions
  • Document parameters: Add clear comments
  • Handle errors gracefully: Add validation steps
  • Maintain backwards compatibility: Don’t break existing sites

Configuration Updates:

  • Test locally first: Verify hugo config output
  • Check all sites: Test impact on all dependent repositories
  • Document changes: Update this documentation
  • Use minimal diffs: Only change what’s necessary
  • Validate YAML: Use yamllint before committing

Shared Resources:

  • Keep layouts generic: Avoid site-specific code
  • Document usage: Add comments to complex partials
  • Version carefully: Breaking changes require coordination
  • Test across sites: Ensure compatibility with all sites
  • Optimize assets: Minimize SCSS and JS files

Communication:

  • Open issues: Discuss major changes before implementing
  • Tag maintainers: Use @mentions for review requests
  • Document breaking changes: Clearly mark in PR description
  • Update changelog: Keep CHANGELOG.md up to date
  • Announce deployments: Notify dependent site owners

14. CI/CD Workflows Reference

14.1. build-site-action.yml (Reusable)

Location: my-documents/.github/workflows/build-site-action.yml

Purpose: Reusable workflow called by dependent repositories to build and deploy Hugo sites.

Inputs:

inputs:
  site-name:
    description: Name of the site being built
    required: true
    type: string

  base-url:
    description: Base URL for the site
    required: true
    type: string

  checkout-repo:
    description: Repository to checkout (owner/repo)
    required: true
    type: string

  hugo-version:
    description: Hugo version to use
    required: false
    type: string
    default: latest

  go-version:
    description: Go version to use
    required: false
    type: string
    default: '1.24'

Steps:

  1. Checkout repository: Clones the calling repository
  2. Setup Hugo: Installs Hugo Extended
  3. Setup Go: Installs Go (required for Hugo modules)
  4. Download modules: Runs hugo mod get -u
  5. Build site: Runs hugo --minify
  6. Upload artifact: Uploads public/ directory
  7. Deploy to Pages: Uses actions/deploy-pages

Usage Example:

jobs:
  build-deploy:
    uses: fchastanet/my-documents/.github/workflows/build-site-action.yml@master
    with:
      site-name: bash-compiler
      base-url: https://bash-compiler.devlab.top
      checkout-repo: fchastanet/bash-compiler

14.2. build-site.yml (my-documents Own)

Location: my-documents/.github/workflows/build-site.yml

Purpose: Builds and deploys the my-documents site itself (not a reusable workflow).

Triggers:

on:
  push:
    branches: [master]
    paths:
      - content/**
      - static/**
      - shared/**
      - configs/**
      - hugo.yaml
      - go.mod
  workflow_dispatch:

Calls: The same build-site-action.yml reusable workflow

Configuration:

jobs:
  build-deploy:
    uses: ./.github/workflows/build-site-action.yml
    with:
      site-name: my-documents
      base-url: https://devlab.top
      checkout-repo: fchastanet/my-documents

14.3. main.yml

Location: my-documents/.github/workflows/main.yml

Purpose: Runs pre-commit hooks and MegaLinter on the repository and deploy documentation if master branch is updated.

Triggers:

on:
  push:
    branches: ['**']
  pull_request:
    branches: [master]
  workflow_dispatch:

Steps:

  1. Checkout code: Clones repository with full history
  2. Setup Python: Installs Python for pre-commit
  3. Install pre-commit: Installs pre-commit tool
  4. Run pre-commit: Executes all pre-commit hooks
  5. Run MegaLinter: Runs comprehensive linting
  6. Upload reports: Saves linter reports as artifacts
  7. Create auto-fix PR: Optionally creates PR with fixes (if not “skip fix” in commit)

Linters Run:

  • Markdown: mdformat, markdownlint
  • YAML: yamllint, v8r
  • JSON: jsonlint
  • Bash: shellcheck, shfmt
  • Spelling: cspell, codespell
  • Secrets: gitleaks, secretlint

Auto-fix Behavior:

If linters make changes and commit message doesn’t contain “skip fix”, an auto-fix PR is created automatically.

15. Summary

This documentation system uses a modern, reusable GitHub Actions architecture that simplifies deployment and maintenance:

Key Takeaways:

  • No complex authentication: Standard GITHUB_TOKEN only
  • Reusable action: One workflow definition, multiple sites
  • Hugo modules: Share resources without file copying
  • Independent control: Each repo owns its deployment
  • Easy testing: Standard Hugo commands work locally
  • Fast builds: Parallel execution across repositories

Getting Started:

  1. Create content structure in your repository
  2. Add go.mod, hugo.yaml, and build-site.yml
  3. Configure GitHub Pages to use “GitHub Actions” source
  4. Push to trigger automatic build and deployment

Next Steps:

For questions or issues, open an issue in the my-documents repository.

1.2 - Static Site Generation Migration Analysis

Analysis of migrating from Docsify to an SEO-optimized static site generator

Project: my-documents repository migration and multi-site consolidation

Goal: Migrate from Docsify to an SEO-optimized static site generator while maintaining simplicity and GitHub CI compatibility

1. Executive Summary

This document evaluates the current Docsify setup and recommends alternative static site generators that provide superior SEO performance while maintaining the simplicity and ease of deployment that made Docsify attractive.

Current Challenge: Docsify renders content client-side, which severely limits SEO capabilities and page load performance. This is critical for a documentation site seeking organic search visibility.

2. Current Solution Analysis: Docsify

2.1. Current Configuration

  • Type: Client-side SPA (Single Page Application)
  • Deployment: Direct to GitHub Pages (no build step)
  • Content Format: Markdown
  • Theme: Simple Dark (customized)
  • Search: Built-in search plugin
  • Navigation: Manual sidebar and navbar configuration

2.2. Docsify Pros ✅

AdvantageImpact
Zero build step requiredInstant deployment, minimal CI/CD complexity
Simple file structureEasy to add new documentation files
No dependencies to manageFewer security concerns, simpler setup
Client-side renderingWorks directly with GitHub Pages
Lightweight theme systemEasy customization with CSS
Good for technical audienceFast navigation for users familiar with SPAs
Markdown-firstNatural for technical documentation

2.3. Docsify Cons ❌

LimitationImpact
Client-side renderingPoor SEO - Search engines struggle to index content
No static HTMLNo pre-rendered pages for crawlers
JavaScript dependentRequires JS in browser (security consideration)
Limited meta tags controlDifficult to optimize individual pages for SEO
Slow initial page loadJavaScript bundle must load first
No built-in sitemapManual sitemap generation needed
No RSS/feed supportHard to distribute content
Search plugin limitationsSite search not indexed by external search engines
No static asset optimizationAll images referenced as relative paths
Outdated dependency stackUses Vue 2 (Vue 3 available), jQuery, legacy patterns

2.4. Docsify SEO Score

Current Estimate: 2/10

  • ❌ No static pre-rendered HTML
  • ❌ Robot.txt and sitemap not automatically generated
  • ❌ Limited per-page meta tag control
  • ❌ No automatic JSON-LD schema generation
  • ❌ Poor mobile-first Core Web Vitals (JS-heavy)
  • ⚠️ Possible crawl budget waste
  • ⚠️ Delayed indexing (content hidden until JS loads)

3.1. Phase 1: Evaluation (This Phase)

  • Compare alternatives against criteria
  • Identify best fit for multi-site architecture
  • Plan migration strategy

3.2. Phase 2: Pilot

  • Set up new solution with one repository
  • Migrate content and test
  • Validate SEO improvements

3.3. Phase 3: Full Migration

  • Migrate remaining repositories
  • Set up CI/CD pipeline
  • Monitor performance metrics

3.4. Phase 4: Optimization

  • Fine-tune SEO settings
  • Implement analytics
  • Monitor search engine indexing

4. Alternative Solutions Comparison

Type: Go-based static site generator Build Time: <1s for most sites Theme System: Flexible with 500+ themes

4.1.1. Pros ✅

  • Extremely fast compilation - Processes 1000+ pages in milliseconds
  • Excellent for documentation - Purpose-built with documentation sites in mind
  • Superior SEO support - Generates static HTML, sitemaps, feeds, schemas
  • Simple setup - Single binary, no dependency hell
  • Markdown + frontmatter - Natural upgrade from Docsify
  • GitHub Actions ready - Hugo orb/actions available for CI/CD
  • Responsive themes - Many documentation-specific themes (Docsy, Relearn, Book)
  • Built-in features - Search indexes, RSS feeds, JSON-LD support
  • Content organization - Hierarchical content structure with archetypes
  • Output optimization - Image processing, minification, CSS purging
  • Flexible routing - Customize URLs, create custom taxonomies
  • Active community - Large ecosystem, frequent updates
  • Multi-language support - Built-in i18n capability

4.1.2. Cons ❌

  • Learning curve for Go templating (shortcodes, partials)
  • Theme customization requires understanding Hugo’s page model
  • Configuration in TOML/YAML (minor, but different from Docsify)
  • Less visual for live preview compared to Docsify

4.1.3. SEO Score: 9/10 ✅

  • ✅ Static HTML pre-rendering
  • ✅ Automatic sitemap generation
  • ✅ Per-page meta tags and structured data
  • ✅ RSS/Atom feeds
  • ✅ Canonical URLs
  • ✅ Image optimization
  • ✅ Performance optimizations (minification, compression)
  • ⚠️ JSON-LD not automated (requires theme customization)

4.1.4. GitHub CI/CD Integration

# .github/workflows/deploy.yml example
  - uses: peaceiris/actions-hugo@v2
    with:
      hugo-version: latest
      extended: true

  - name: Build
    run: hugo --minify

  - name: Deploy
    uses: peaceiris/actions-gh-pages@v3
    with:
      github_token: ${{ secrets.GITHUB_TOKEN }}
      publish_dir: ./public

4.1.5. Migration Effort

  • Content: Minimal - Markdown stays same, just add frontmatter
  • Structure: Organize into content sections (easy mapping from Docsify)
  • Navigation: Automatic from directory structure or config
  • Customization: Moderate - Theme customization required
  1. Docsy - Google-created, excellent documentation theme, built-in search
  2. Relearn - MkDocs-inspired, sidebar navigation like Docsify
  3. Book - Clean, minimal, perfect for tutorials
  4. Geek Docs - Modern, fast, developer-friendly

4.1.7. Best For

✅ Technical documentation ✅ Multi-site architecture ✅ SEO-critical sites ✅ GitHub Pages deployment ✅ Content-heavy sites (1000+ pages)


4.2. Option 2: Astro ⭐⭐⭐⭐

Type: JavaScript/TypeScript-based, island architecture Build Time: <2s typical Theme System: Component-based

4.2.1. Pros ✅

  • Outstanding SEO support - Static HTML generation, built-in meta tag management
  • Zero JavaScript by default - Only JS needed for interactive components
  • Modern stack - Latest JavaScript patterns, TypeScript support
  • Markdown + MDX support - Markdown with embedded React/Vue components
  • Component imports - Use React, Vue, Svelte components in Markdown
  • Fast performance - Island architecture means minimal JS shipping
  • Great for blogs/docs - Built-in content collections API
  • Image optimization - Automatic image processing and responsive images
  • Built-in integrations - Readily available for analytics, fonts, CSS
  • Flexible deployment - Works with any static host or serverless
  • TypeScript first - Better tooling and IDE support
  • Vite-based - Fast HMR and builds

4.2.2. Cons ❌

  • Newer ecosystem (less battle-tested than Hugo)
  • Small learning curve with Astro-specific patterns
  • Requires Node.js and npm (dependency management)
  • Theme ecosystem smaller than Hugo
  • MDX adds complexity if not needed

4.2.3. SEO Score: 9/10 ✅

  • ✅ Static HTML pre-rendering
  • ✅ Fine-grained meta tag control
  • ✅ JSON-LD schema support
  • ✅ Automatic sitemap generation
  • ✅ RSS/feed support
  • ✅ Image optimization with AVIF
  • ✅ Open Graph and Twitter cards
  • ✅ Performance metrics built-in

4.2.4. GitHub CI/CD Integration

# .github/workflows/deploy.yml example
  - name: Install dependencies
    run: npm ci

  - name: Build
    run: npm run build

  - name: Deploy
    uses: peaceiris/actions-gh-pages@v3
    with:
      github_token: ${{ secrets.GITHUB_TOKEN }}
      publish_dir: ./dist

4.2.5. Migration Effort

  • Content: Minimal - Markdown compatible with optional frontmatter
  • Structure: Convert to Astro collections (straightforward)
  • Navigation: Can use auto-generated from file structure
  • Customization: Moderate - Components offer more flexibility than Hugo
  1. Starlight - Official Astro docs template, excellent for documentation
  2. Docs Kit - Tailored for technical documentation
  3. Astro Paper - Blog-focused, highly customizable

4.2.7. Best For

✅ Modern tech stack preference ✅ Need for interactive components ✅ TypeScript-heavy teams ✅ Blogs + Documentation hybrid ✅ SEO + Performance critical


4.3. Option 3: 11ty (Eleventy) ⭐⭐⭐⭐

Type: JavaScript template engine Build Time: <1s typical Theme System: Template-based

4.3.1. Pros ✅

  • Incredibly flexible - Supports multiple template languages (Markdown, Nunjucks, Liquid, etc.)
  • Lightweight - Minimal opinion on structure, you decide
  • Fast builds - Blazingly fast incremental builds
  • JavaScript-based - Easier for Node.js teams than Go
  • Markdown-first - Natural Markdown support with plugins
  • No locked-in framework - Use vanilla HTML/CSS or any framework
  • Great community - Excellent documentation and starter projects
  • Simple config - .eleventy.js is readable JavaScript
  • Content collections - Flexible ways to organize content
  • Image processing - Built-in with popular plugins
  • GitHub Pages friendly - Easy integration with GitHub Actions
  • Low barrier to entry - Understand JavaScript, you understand Eleventy

4.3.2. Cons ❌

  • Less opinionated (requires more configuration)
  • Smaller pre-built theme ecosystem
  • Requires JavaScript knowledge for customization
  • No built-in search (needs separate solution)
  • Learning curve steeper if unfamiliar with template languages

4.3.3. SEO Score: 8/10 ✅

  • ✅ Static HTML generation
  • ✅ Manual sitemap generation (simple plugin)
  • ✅ Per-page meta tag control
  • ✅ Feed/RSS support (via plugins)
  • ✅ Image optimization (via plugins)
  • ⚠️ Schema/JSON-LD (requires custom implementation)

4.3.4. GitHub CI/CD Integration

# .github/workflows/deploy.yml example
  - name: Install dependencies
    run: npm ci

  - name: Build
    run: npm run build

  - name: Deploy
    uses: peaceiris/actions-gh-pages@v3
    with:
      github_token: ${{ secrets.GITHUB_TOKEN }}
      publish_dir: ./_site

4.3.5. Migration Effort

  • Content: Minimal - Markdown files work as-is
  • Structure: Very flexible, custom folder organization
  • Navigation: Can auto-generate from structure or manually configure
  • Customization: High - Maximum control but more work
  1. 11ty Base Blog - Simple starting point
  2. Eleventy High Performance Blog - Performance-focused
  3. Slinkity - Hybrid with component support

4.3.7. Best For

✅ Developers who want full control ✅ Simple, focused documentation ✅ JavaScript/Node.js teams ✅ Performance optimization focus ✅ Unique design requirements


4.4. Option 4: VuePress 2 ⭐⭐⭐

Type: Vue 3-based static site generator Build Time: 1-2s typical Theme System: Vue components

4.4.1. Pros ✅

  • Vue ecosystem - Use Vue components directly in Markdown
  • Documentation-first - Built specifically for docs
  • Markdown extensions - Plugin system for custom Markdown syntax
  • Built-in search - Local search with Algolia option
  • Plugin ecosystem - Rich ecosystem for docs sites
  • Good themes - VuePress Theme Default is solid
  • PWA support - Can work offline (if configured)
  • Git history - Can show last edited time from git
  • i18n built-in - Multi-language support
  • Flexible routing - Customizable URL structure

4.4.2. Cons ❌

  • Vue knowledge required
  • Smaller ecosystem than Hugo
  • Heavy JavaScript bundle (not as optimized as Astro)
  • Less mature than Hugo
  • Configuration can be verbose
  • Search indexing still client-side primarily

4.4.3. SEO Score: 6/10 ⚠️

  • ✅ Static HTML generation
  • ✅ Per-page meta tags
  • ✅ Sitemap support (via plugin)
  • ⚠️ Search still somewhat client-side
  • ⚠️ Performance not optimized (Vue overhead)
  • ⚠️ JSON-LD requires manual setup

4.4.4. GitHub CI/CD Integration

  - name: Install dependencies
    run: npm ci

  - name: Build
    run: npm run build

  - name: Deploy
    uses: peaceiris/actions-gh-pages@v3
    with:
      github_token: ${{ secrets.GITHUB_TOKEN }}
      publish_dir: ./dist

4.4.5. Migration Effort

  • Content: Minimal - Markdown compatible
  • Structure: Organized in .vuepress/config.js
  • Navigation: Configured in sidebar/navbar config
  • Customization: Moderate - Vue components for complex needs

4.4.6. Best For

✅ Vue-centric teams ✅ Need interactive components ✅ Plugin-heavy customization ✅ Smaller documentation sites ✅ Already using Vue ecosystem


4.5. Option 5: MkDocs ⭐⭐⭐

Type: Python-based documentation generator Build Time: <1s typical Theme System: Python template-based

4.5.1. Pros ✅

  • Documentation-optimized - Built by documentation enthusiasts
  • Simple configuration - mkdocs.yml is straightforward
  • Markdown-native - Pure Markdown with extensions
  • Great themes - Material for MkDocs is excellent
  • Low overhead - Minimal learning curve
  • Python-based - Good for Python-heavy teams
  • Fast builds - Quick incremental rebuilds
  • Search integration - Good local search, Algolia-ready
  • Git integration - Edit on GitHub features
  • Active community - Good documentation and examples

4.5.2. Cons ❌

  • Python dependency management
  • Smaller ecosystem than Hugo
  • Theme customization requires Python knowledge
  • Less flexibility than some alternatives
  • Setup requires Python environment

4.5.3. SEO Score: 7/10 ✅

  • ✅ Static HTML generation
  • ✅ Per-page meta tags
  • ✅ Sitemap support (via plugin)
  • ⚠️ Schema/JSON-LD minimal
  • ⚠️ Image optimization requires external tools

4.5.4. GitHub CI/CD Integration

  - name: Set up Python
    uses: actions/setup-python@v4
    with:
      python-version: '3.11'

  - name: Install dependencies
    run: pip install mkdocs mkdocs-material

  - name: Build
    run: mkdocs build

  - name: Deploy
    uses: peaceiris/actions-gh-pages@v3
    with:
      github_token: ${{ secrets.GITHUB_TOKEN }}
      publish_dir: ./site

4.5.5. Migration Effort

  • Content: Minimal - Markdown files work directly
  • Structure: Configured in mkdocs.yml
  • Navigation: Simple hierarchical structure
  • Customization: Easy for theming, harder for core customization

4.5.6. Best For

✅ Documentation-only focus ✅ Python-familiar teams ✅ Minimal configuration needed ✅ Material design preference ✅ Rapid setup priority


4.6. Option 6: Next.js / Vercel ⭐⭐

Type: React meta-framework Build Time: 5-10s typical Theme System: React components

4.6.1. Pros ✅

  • Powerful frameworks - React + Node.js backend possibility
  • Vercel optimization - Vercel specialist optimization
  • React ecosystem - Access to millions of components
  • SSR capable - Server-side rendering if needed
  • API routes - Can add serverless functions
  • Image optimization - Automatic image optimization
  • Incremental Static Regeneration - Change content without full rebuild
  • TypeScript native - First-class TypeScript support
  • Performance monitoring - Web vitals built-in

4.6.2. Cons ❌

  • Overkill for static docs - Too much complexity
  • Learning curve steep - React + Next.js knowledge required
  • Build times longer - Slower than purpose-built SSGs
  • More dependencies - Dependency management complexity
  • GitHub Pages less ideal - Optimized for Vercel deployment
  • Maintenance burden - React team required to maintain

4.6.3. SEO Score: 8/10 ✅

  • ✅ Static generation capability
  • ✅ Per-page meta tags via next/head
  • ✅ Sitemap and robots.txt support
  • ✅ Image optimization
  • ⚠️ Requires more configuration
  • ⚠️ Slower builds than dedicated SSGs

4.6.4. GitHub CI/CD Integration (Docsify level: Complex)

  - name: Install dependencies
    run: npm ci

  - name: Build
    run: npm run build

  - name: Static Export
    run: npm run export

  - name: Deploy
    uses: peaceiris/actions-gh-pages@v3

4.6.5. Migration Effort

  • Content: Moderate - Convert to Next.js structure
  • Structure: Pages directory structure required
  • Navigation: Custom component creation
  • Customization: High complexity

4.6.6. Best For

✅ React-centric teams ✅ Need dynamic functionality ✅ Willing to deploy on Vercel ✅ Complex sites with interactive elements ❌ NOT recommended for pure documentation


4.7. Option 7: Gatsby ⭐⭐

Type: React-based static site generator Build Time: 10-30s typical Theme System: React components + theme shadowing

4.7.1. Pros ✅

  • Powerful plugin system - Huge ecosystem
  • GraphQL querying - Flexible content queries
  • Performance optimization - Good performance features
  • React components - Full React power available
  • CMS integration - Works with many headless CMS

4.7.2. Cons ❌

  • Heavy and slow - Longest build times of alternatives
  • High complexity - Steep learning curve
  • Dependency bloat - Many dependencies to maintain
  • Not ideal for docs - Over-engineered for simple documentation
  • GitHub Pages unfriendly - Best with Netlify
  • Overkill - Too much power for static docs

4.7.3. SEO Score: 7/10 ✅

  • ✅ Static generation
  • ✅ Good plugin ecosystem for SEO
  • ⚠️ Heavy JavaScript overhead
  • ⚠️ Slower builds

4.7.4. Best For

NOT recommended for documentation migration


5. Comparison Matrix

CriteriaHugoAstro11tyVuePressMkDocsNext.jsGatsby
SEO Score9/109/108/106/107/108/107/10
Build Speed⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐
Learning Curve⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐
Customization⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐
GitHub Pages⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐
Static Output
Documentation Focus⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐
Theme Ecosystem⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐
Community Size⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐
GitHub Pages Native⚠️
Multiple Sites⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐

6. Improvements for New Solutions

Regardless of which SSG is chosen, implement these SEO improvements:

6.1. Technical SEO Baseline

  • Generate robots.txt automatically
  • Generate XML sitemap automatically
  • Implement per-page meta tags (title, description)
  • Add canonical URLs to prevent duplication
  • Implement JSON-LD schema (Article, BreadcrumbList, Organization)
  • Open Graph and Twitter card meta tags
  • Mobile-first responsive design
  • Fast page load (Core Web Vitals: LCP, CLS, FID)
  • Image optimization and lazy loading
  • Minify and compress assets

6.2. Content Structure

  • Implement breadcrumb navigation (visual + schema)
  • Hierarchical heading structure (H1, H2, H3)
  • Internal linking strategy
  • Related content suggestions
  • Table of contents for long articles
  • Reading time estimates
  • Last updated timestamps

6.3. Performance Optimizations

  • Code splitting and lazy loading
  • Image optimization (WebP, AVIF formats)
  • CSS/JS minification
  • Critical CSS inline
  • Service worker for offline access
  • Asset caching strategies
  • Compression (gzip/brotli)
  • CDN integration

6.4. Search and Indexing

  • Submit sitemap to Google Search Console
  • Monitor indexing status
  • Fix crawl errors
  • Optimize Core Web Vitals
  • Monitor search appearance (ratings, rich results)
  • Use Google Search Console to identify improvements

6.5. Advanced SEO

  • Implement full-text search with ranking
  • Add RSS/Atom feeds for content discovery
  • Implement structured data for articles
  • Add FAQ schema for common questions
  • Breadcrumb schema implementation
  • Organization/website schema
  • Add “edit on GitHub” links for engagement signals

6.6. Analytics and Monitoring

  • Google Analytics 4 integration
  • Search Console monitoring
  • Core Web Vitals tracking
  • Error tracking (Sentry/similar)
  • Performance monitoring dashboard
  • Keyword ranking tracking
  • Traffic Analysis

6.7. GitHub CI/CD Improvements

  • Semantic versioning for releases
  • Link checker in CI pipeline
  • SEO audit in CI (Lighthouse, lighthouse-ci)
  • Spell checker (already implemented)
  • Broken internal link detection
  • Mobile-first testing
  • Accessibility testing (a11y)
  • Build time monitoring
  • Automated lighthouse reports

7. Hugo-Specific Recommendations

If Hugo is chosen (recommended), implement:

# config.yaml example improvements
params:
  description: Collection of my documents on various subjects
  keywords: bash,best practices,learn,docker,jenkins
  openGraph:
    enabled: true
  twitterCards:
    enabled: true
  jsonLD:
    enabled: true

outputs:
  home:
    - HTML
    - JSON
    - RSS
  section:
    - HTML
    - JSON
    - RSS

taxonomies:
  category: categories
  tag: tags

mediaTypes:
  application/json:
    suffixes:
      - json

outputFormats:
  JSON:
    isPlainText: true
    mediaType: application/json

8. Astro-Specific Recommendations

If Astro is chosen, implement:

// astro.config.mjs example improvements
export default defineConfig({
    integrations: [
        sitemap(),
        robotsTxt(),
        react(),
        vue(),
    ],

    image: {
        remotePatterns: [{
            protocol: "https"
        }],
    },

    vite: {
        plugins: [
            sitemap(),
        ],
    },
});

9. Migration Strategy for Multiple Sites

github-sites-monorepo/
├── myDocuments/
│   ├── content/
│   ├── themes/
│   └── config.yaml
├── bashToolsFramework/
│   ├── content/
│   ├── themes/
│   └── config.yaml
├── bashTools/
│   ├── content/
│   ├── themes/
│   └── config.yaml
└── bashCompiler/
    ├── content/
    ├── themes/
    └── config.yaml

CI/CD Strategy:

  • Single workflow builds all sites
  • Each site has separate output directory
  • Deploy to respective GitHub Pages branches
  • Shared theme for consistency (git submodule or package)
  • Single dependency management file

10. Risk Assessment and Mitigation

RiskHugoAstro11tyMkDocsVuePress
Breaking changes⚠️ Low⚠️ Medium✅ Low✅ Low⚠️ Medium
Ecosystem longevity✅ Very High⚠️ High✅ Very High✅ High⚠️ Medium
Theme support✅ Excellent⚠️ Good⚠️ Good✅ Good⚠️ Good
GitHub Pages✅ Perfect✅ Perfect✅ Perfect✅ Perfect⚠️ Works
Team skills⚠️ Go required⚠️ JS required✅ JS (low level)✅ Python/Markdown⚠️ Vue required
Maintenance burden✅ Low⚠️ Medium⚠️ Medium✅ Low⚠️ Medium

11. Final Recommendation: Hugo

11.1. Justification

  1. SEO Excellence - 9/10 score meets all objectives
  2. Simplicity - Single Go binary, no dependency management
  3. Performance - <1s builds, scales to thousands of pages
  4. Documentation-First - Built for exactly this use case
  5. GitHub Pages Native - Zero friction deployment
  6. Multi-Site Scalability - Perfect for multiple repositories
  7. Community - Largest documentation site generator community
  8. Proven - 1000+ major documentation sites use it
  9. Themes - Docsy, Relearn excellent for technical docs
  10. Future-Proof - Stable, active development

11.2. Hugo Implementation Plan

Phase 1: Setup (1-2 weeks)

  • Install Hugo and select Docsy or Relearn theme
  • Create content structure
  • Configure SEO baseline
  • Set up GitHub Actions workflow
  • Test locally

Phase 2: Migration (2-3 weeks)

  • Convert Markdown files (minimal changes)
  • Migrate sidebar structure to Hugo config
  • Update internal links
  • Test all links and navigation
  • Performance testing

Phase 3: SEO Optimization (1-2 weeks)

  • Implement schema markup
  • Configure sitemaps and feeds
  • Submit to Google Search Console
  • Baseline performance metrics
  • Optimize Core Web Vitals

Phase 4: Deployment (1 week)

  • Validate all tests pass
  • Deploy to production
  • Monitor indexing and performance
  • Gather feedback

12. Alternative: Astro for Modern Setup

If your team prefers JavaScript/TypeScript and wants maximum flexibility with modern tooling, Astro with Starlight is the secondary recommendation:

  • Excellent SEO (equal to Hugo)
  • More flexible for custom components
  • Modern JavaScript ecosystem
  • Better DX with TypeScript
  • Slightly longer build times acceptable
  • GitHub Pages deployment straightforward
  • Docsify - Keep for simple internal documentation only, not public sites
  • Next.js - Overcomplicated for documentation, not ideal for GitHub Pages
  • Gatsby - Slow builds, high complexity, deprecated

14. Conclusion

Migrate to Hugo with Docsy theme for optimal balance of simplicity, SEO performance, and documentation focus. This will:

  • Improve SEO from 2/10 to 9/10
  • Reduce page load times significantly
  • Provide static pre-rendered pages for crawlers
  • Scale to multiple sites easily
  • Maintain simplicity in CI/CD
  • Future-proof your documentation infrastructure

Next Steps:

  1. Review this analysis with relevant stakeholders
  2. Set up pilot Hugo site with one repository
  3. Validate SEO improvements with Search Console
  4. Plan full migration timeline
  5. Document Hugo best practices for team

1.3 - Multi repositories Site Generation

Comprehensive documentation of the Hugo migration for multi-site documentation

Project: Migration from Docsify to Hugo with Docsy theme for multiple documentation repositories

Status: ✅ Completed

Repositories:

  • fchastanet/my-documents (orchestrator + own documentation)
  • fchastanet/bash-compiler
  • fchastanet/bash-tools
  • fchastanet/bash-tools-framework
  • fchastanet/bash-dev-env

Related Documentation: See doc/ai/2026-02-18-migrate-repo-from-docsify-to-hugo.md for detailed migration guide.

1. Technical Solutions Evaluated

1.1. Static Site Generator Solutions

1.1.1. Hugo (SELECTED)

Evaluation: ⭐⭐⭐⭐⭐ Type: Go-based static site generator

Pros:

  • Extremely fast compilation (<1s for most documentation sites)
  • Excellent for documentation with purpose-built features
  • Superior SEO support (static HTML, sitemaps, feeds, schemas) - 9/10 SEO score
  • Single binary with no dependency complications
  • Markdown + frontmatter support (natural progression from Docsify)
  • GitHub Actions ready with official actions
  • Large theme ecosystem (500+ themes) including specialized documentation themes
  • Built-in features: search indexes, RSS feeds, hierarchical content organization
  • Output optimization: image processing, minification, CSS purging
  • Active community with frequent updates
  • Multi-language support built-in

Cons:

  • Learning curve for Go templating (shortcodes, partials)
  • Theme customization requires understanding Hugo’s page model
  • Configuration in YAML/TOML format

GitHub CI/CD Integration: Native, simple integration with peaceiris/actions-hugo

Best For: Technical documentation, multi-site architecture, SEO-critical sites, GitHub Pages, content-heavy sites

1.1.2. Astro

Evaluation: ⭐⭐⭐⭐ Type: JavaScript/TypeScript-based with island architecture

Pros:

  • Outstanding SEO support (static HTML, zero JavaScript by default) - 9/10 SEO score
  • Modern JavaScript patterns with TypeScript support
  • Markdown + MDX support (embedded React/Vue components in Markdown)
  • Island architecture minimizes JavaScript shipping
  • Fast performance and build times (<2s)
  • Automatic image optimization (AVIF support)
  • Vite-based with fast HMR

Cons:

  • Newer ecosystem, less battle-tested than Hugo
  • Requires Node.js and npm dependency management
  • Smaller theme ecosystem
  • MDX adds complexity if not needed

Best For: Modern tech stacks, interactive components, TypeScript-heavy teams, blogs + documentation hybrids

1.1.3. 11ty (Eleventy)

Evaluation: ⭐⭐⭐⭐

Type: JavaScript template engine

Pros:

  • Incredibly flexible with multiple template language support
  • Lightweight and fast builds
  • JavaScript-based (easier for Node.js teams)
  • Low barrier to entry
  • No framework lock-in

Cons:

  • Less opinionated, requires more configuration
  • Smaller pre-built theme ecosystem
  • No built-in search (requires plugins)
  • SEO score: 8/10

Best For: Developers wanting full control, JavaScript/Node.js teams, unique design requirements

1.1.4. VuePress 2

Evaluation: ⭐⭐⭐

Type: Vue 3-based static site generator

Pros:

  • Documentation-first design
  • Built-in search functionality
  • Plugin ecosystem for documentation
  • Vue component integration in Markdown

Cons:

  • Vue.js knowledge required
  • Heavy JavaScript bundle (not as optimized as others)
  • Smaller ecosystem than Hugo
  • SEO score: 6/10

Best For: Vue-centric teams, smaller documentation sites

1.1.5. MkDocs

Evaluation: ⭐⭐⭐

Type: Python-based documentation generator

Pros:

  • Documentation-optimized out of the box
  • Simple configuration
  • Material for MkDocs theme is excellent
  • Fast builds

Cons:

  • Python dependency management required
  • Smaller ecosystem than Hugo
  • Limited flexibility
  • SEO score: 7/10

Best For: Documentation-only focus, Python-familiar teams, rapid setup

1.1.6. Next.js and Gatsby

Evaluation: ⭐⭐ - Not recommended for static documentation

Reasons:

  • Overkill complexity for pure documentation
  • Longer build times (5-30s vs <1s for Hugo)
  • Heavy JavaScript requirements
  • Optimized for different use cases (web apps, not docs)
  • Maintenance burden too high for static documentation

1.1.7. Comparison Summary

CriteriaHugoAstro11tyVuePressMkDocs
SEO Score9/109/108/106/107/10
Build Speed⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐
Learning Curve⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐
GitHub Pages⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐
Documentation Focus⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐
Theme Ecosystem⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐
Multi-Site Support⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐

1.2. Multi-Site Build Pipeline Solutions

1.2.1. Centralized Orchestrator (my-documents builds all sites) (SELECTED)

Evaluation: ⭐⭐⭐⭐⭐ Architecture:

my-documents (orchestrator)
├── .github/workflows/build-all-sites.yml  ← Builds all sites
├── configs/
│   ├── _base.yaml                         ← Shared config
│   ├── bash-compiler.yaml                 ← Site overrides
│   ├── bash-tools.yaml
│   └── bash-tools-framework.yaml
├── shared/
│   ├── layouts/                           ← Shared templates
│   ├── assets/                            ← Shared styles
│   └── archetypes/                        ← Content templates
└── content/                               ← my-documents own docs

Dependent repos (minimal):
bash-compiler/
├── .github/workflows/trigger-docs.yml     ← Triggers my-documents
└── content/en/                            ← Documentation only

How It Works:

  1. Push to bash-compiler → triggers my-documents via repository_dispatch
  2. my-documents workflow:
    • Checks out ALL repos (my-documents, bash-compiler, bash-tools, bash-tools-framework, bash-dev-env)
    • Builds each site in parallel using GitHub Actions matrix strategy
    • Merges configs (_base.yaml + site-specific overrides)
    • Deploys each site to its respective GitHub Pages

Pros:

  • ✅ All repos under same owner (fchastanet) simplifies permission management
  • ✅ One workflow update fixes all sites immediately
  • ✅ Guaranteed consistency across all documentation sites
  • ✅ Simpler per-repo setup (2 files: trigger workflow + content)
  • ✅ No Hugo modules needed (simpler dependency management)
  • ✅ Centralized theme customization with per-site overrides
  • ✅ Build all sites in ~60s (parallel matrix execution)
  • ✅ Single point of maintenance

Cons:

  • ⚠️ Requires authentication setup (GitHub App or deploy keys)
  • ⚠️ All sites rebuild together (cannot isolate to single site)
  • ⚠️ All-or-nothing failures (one site failure blocks others in same matrix job)
  • ⚠️ Slightly more complex initial setup

Best For: Related projects under same organization, shared theme/purpose, centralized maintenance preference

1.2.2. Decentralized with Reusable Workflows + Hugo Modules

Architecture:

my-documents (shared resources hub)
├── .github/workflows/hugo-build-deploy-reusable.yml  ← Reusable workflow
├── layouts/ (Hugo module export)
└── assets/ (Hugo module export)

bash-compiler/ (independent)
├── .github/workflows/hugo-build-deploy.yml  ← Calls reusable workflow
├── hugo.yaml (imports my-documents module)
├── go.mod
└── content/

How It Works:

  1. Each dependent repo has its own build workflow
  2. Workflow calls the reusable workflow from my-documents
  3. Hugo modules pull shared resources during build
  4. Each site builds and deploys independently

Pros:

  • ✅ Independent deployment (site failures isolated)
  • ✅ Automatic updates when reusable workflow changes
  • ✅ Version control (can pin to @v1.0.0 or @master)
  • ✅ No trigger coordination needed
  • ✅ Faster builds for single-site changes (~30s per site)
  • ✅ Per-repo flexibility if needed

Cons:

  • ⚠️ Hugo modules require Go toolchain
  • ⚠️ More files per repository (6 core files vs 2)
  • ⚠️ Learning curve for Hugo module system
  • ⚠️ Network dependency (modules fetched from GitHub)
  • ⚠️ Potential configuration drift if repos don’t update modules
  • ⚠️ More complex to enforce consistency

Best For: Fully independent projects, teams wanting flexibility, isolated failure tolerance

1.2.3. True Monorepo with Subdirectories

Architecture: All content in single repo with subdirectories for each project

Pros:

  • ✅ Simplest configuration
  • ✅ Single build process
  • ✅ Guaranteed consistency

Cons:

  • ❌ Loses separate GitHub Pages URLs
  • ❌ No independent repository control
  • ❌ Violates existing repository structure
  • ❌ Complicated permission management

Evaluation: Not recommended - Conflicts with requirement to maintain separate repository URLs

1.2.4. Pipeline Solution Comparison

CriteriaCentralized OrchestratorDecentralized ReusableMonorepo
ComplexityLow (minimal per-repo)Medium (per-repo setup)Low (single repo)
Build Time~60s all sites~30s per site~60s all sites
MaintenanceUpdate onceUpdate workflow × NUpdate once
Consistency✅ GuaranteedCan drift✅ Guaranteed
Failure IsolationAll-or-nothing✅ IndependentAll-or-nothing
Setup Effort1 workflow + N configs6 files × N reposSingle setup
Independent URLs✅ Yes✅ Yes❌ No
Hugo Modules❌ Not neededRequired❌ Not needed

2. Chosen Solutions & Rationale

2.1. Static Site Generator: Hugo + Docsy Theme

Choice: Hugo with Google’s Docsy theme

Rationale:

  1. SEO Requirements Met:

    • Static HTML pre-rendering (search engines can easily index)
    • Automatic sitemap and robots.txt generation
    • Per-page meta tags and structured data support
    • RSS/Atom feeds
    • Image optimization
    • Performance optimizations (minification, compression)
    • SEO improvement: 2/10 (Docsify) → 9/10 (Hugo)
  2. Technical Excellence:

    • Extremely fast builds (<1s for typical documentation site)
    • Simple deployment (single Go binary, no dependency hell)
    • GitHub Pages native support
    • Mature, stable, battle-tested (10+ years in production use)
  3. Documentation-Specific Features:

    • Docsy theme built by Google specifically for documentation
    • Built-in search functionality
    • Responsive design
    • Navigation auto-generation from content structure
    • Version management support
    • Multi-language support
  4. Developer Experience:

    • Markdown + frontmatter (minimal migration effort from Docsify)
    • Good documentation and large community
    • Extensive theme ecosystem
    • Active development and updates
  5. Multi-Site Architecture Support:

    • Excellent support for shared configurations
    • Hugo modules for code reuse
    • Flexible configuration merging
    • Content organization flexibility

Alternatives Considered:

  • Astro: Excellent option, but newer ecosystem and Node.js dependency management adds complexity
  • 11ty: Good flexibility, but less opinionated structure requires more setup work
  • MkDocs: Python dependencies and smaller ecosystem less ideal
  • VuePress/Next.js/Gatsby: Too heavy for pure documentation needs

2.2. Multi-Site Pipeline: Centralized Orchestrator

Choice: Centralized build orchestrator in my-documents repository

Rationale:

  1. Project Context Alignment:

    • All repositories under same owner (fchastanet)
    • All share same purpose (Bash tooling documentation)
    • All need consistent look and feel
    • Related projects benefit from coordinated updates
  2. Maintenance Efficiency:

    • Single workflow update affects all sites immediately
    • One place to fix bugs or add features
    • Guaranteed consistency across all documentation
    • Reduced mental overhead (one system to understand)
  3. Simplified Per-Repository Structure:

    • Only 2 essential files per dependent repo:
      • Trigger workflow (10 lines)
      • Content directory
    • No Hugo configuration duplication
    • No Go module management per repo
  4. Configuration Management:

    • Base configuration shared via configs/_base.yaml
    • Site-specific overrides in configs/{site}.yaml
    • Automatic merging with yq tool
    • No configuration drift possible
  5. Build Efficiency:

    • Parallel matrix execution builds all 5 sites simultaneously
    • Total time ~60s for all sites (vs 30s × 5 = 150s sequential)
    • Resource sharing in CI/CD (single Hugo/Go setup)
  6. Deployment Simplification:

    • Authentication centralized in my-documents (GitHub App)
    • Single set of deployment credentials
    • Easier to audit and manage security

Trade-offs Accepted:

  • ⚠️ All sites rebuild together (acceptable for related documentation)
  • ⚠️ More complex initial setup (one-time investment)
  • ⚠️ All-or-nothing failures (mitigated with fail-fast: false in matrix)

Alternatives Considered:

  • Decentralized Reusable Workflows: Good for truly independent projects, but adds complexity without benefit for our use case where all sites are related and share theme/purpose
  • Monorepo: Would lose independent GitHub Pages URLs, not acceptable

3. Implementation Details

3.1. Repository Architecture

Orchestrator Repository: fchastanet/my-documents

Responsibilities:

  • Build all documentation sites (including its own)
  • Manage shared configurations and theme customizations
  • Deploy to multiple GitHub Pages repositories
  • Coordinate builds triggered from dependent repositories

Dependent Repositories:

  • fchastanet/bash-compiler
  • fchastanet/bash-tools
  • fchastanet/bash-tools-framework
  • fchastanet/bash-dev-env

Responsibilities: Contain documentation content only, trigger builds in orchestrator

3.2. Directory Structure

3.2.1. my-documents (Orchestrator)

/home/wsl/fchastanet/my-documents/
├── .github/workflows/
│   └── build-all-sites.yml              ← Orchestrator workflow
├── configs/
│   ├── _base.yaml                       ← Shared configuration
│   ├── my-documents.yaml               ← my-documents overrides
│   ├── bash-compiler.yaml              ← bash-compiler overrides
│   ├── bash-tools.yaml
│   ├── bash-tools-framework.yaml
│   └── bash-dev-env.yaml
├── shared/
│   ├── layouts/                         ← Shared Hugo templates
│   ├── assets/                          ← Shared SCSS, JS
│   └── archetypes/                      ← Content templates
├── content/                             ← my-documents own content
├── hugo.yaml                            ← Generated per build
└── go.mod                               ← Hugo modules (Docsy)

Key Files:

3.2.2. Dependent Repository (Example: bash-compiler)

fchastanet/bash-compiler/
├── .github/workflows/
│   └── trigger-docs.yml                 ← Triggers orchestrator
└── content/en/                          ← Documentation content only
    ├── _index.md
    └── docs/
        └── *.md

3.3. Configuration Merging Strategy

Approach: Use yq tool for proper YAML deep-merging

Base Configuration: configs/_base.yaml

Contains:

  • Hugo module imports (Docsy theme)
  • Common parameters (language, SEO settings)
  • Shared markup configuration
  • Mount points for shared resources
  • Common menu structure
  • Default theme parameters

Site-Specific Overrides: Example configs/bash-compiler.yaml

Contains:

  • Site title and baseURL
  • Repository-specific links
  • Site-specific theme colors (ui.navbar_bg_color)
  • Custom menu items
  • SEO keywords specific to the project
  • GitHub repository links

Merging Process:

Implemented in .github/workflows/build-all-sites.yml:

yq eval-all 'select(fileIndex == 0) * select(fileIndex == 1)' \ configs/_base.yaml
\ configs/bash-compiler.yaml > hugo.yaml
...

Result: Clean, merged hugo.yaml with:

  • Base configuration as foundation
  • Site-specific overrides applied
  • Proper YAML structure preserved (no duplication)
  • Deep merge of nested objects

3.4. Build Workflow

Main Workflow: .github/workflows/build-all-sites.yml

Triggers:

  • workflow_dispatch - Manual trigger
  • repository_dispatch with type trigger-docs-rebuild - From dependent repos
  • push to master branch affecting:
    • content/**
    • shared/**
    • configs/**
    • .github/workflows/build-all-sites.yml

Strategy: Parallel matrix build

matrix:
  site:
    - name: my-documents
      repo: fchastanet/my-documents
      baseURL: https://devlab.top
      self: true
    - name: bash-compiler
      repo: fchastanet/bash-compiler
      baseURL: https://bash-compiler.devlab.top
      self: false
    # ... other sites

Build Steps (Per Site):

  1. Checkout Orchestrator: Clone my-documents repository
  2. Checkout Content: Clone dependent repository content (if not self)
  3. Setup Tools: Install Hugo Extended 0.160.1, Go 1.24, yq
  4. Prepare Build Directory:
    • For my-documents: Use orchestrator directory
    • For dependent repos: Create build-{site} directory
  5. Merge Configurations: Combine _base.yaml + {site}.yaml
  6. Copy Shared Resources: Link shared layouts, assets, archetypes
  7. Copy Content: Link content directory
  8. Initialize Hugo Modules: Run hugo mod init and hugo mod get -u
  9. Build Site: Run hugo --minify
  10. Deploy: Push to respective GitHub Pages

Concurrency: cancel-in-progress: true prevents duplicate builds

Failure Handling: fail-fast: false allows other sites to build even if one fails

3.5. Deployment Approach

Method: GitHub App authentication (migrated from deploy keys)

Authentication Flow:

  1. Generate App Token: Use actions/create-github-app-token@v1
  2. Deploy with Token: Use peaceiris/actions-gh-pages@v4

Secrets Required (in my-documents):

  • DOC_APP_ID - GitHub App ID
  • DOC_APP_PRIVATE_KEY - GitHub App private key (PEM format)

Deployment Step Example:

  - name: Generate GitHub App token
    id: app-token
    uses: actions/create-github-app-token@v1
    with:
      app-id: ${{ secrets.DOC_APP_ID }}
      private-key: ${{ secrets.DOC_APP_PRIVATE_KEY }}
      owner: fchastanet
      repositories: bash-compiler

  - name: Deploy to GitHub Pages
    uses: peaceiris/actions-gh-pages@v4
    with:
      github_token: ${{ steps.app-token.outputs.token }}
      external_repository: fchastanet/bash-compiler
      publish_dir: ./public
      publish_branch: gh-pages

Result URLs:

3.6. Trigger Mechanism

Dependent Repository Workflow Example: .github/workflows/trigger-docs.yml

name: Trigger Documentation Rebuild

on:
  push:
    branches: [master]
    paths:
      - content/**
      - .github/workflows/trigger-docs.yml

jobs:
  trigger:
    runs-on: ubuntu-latest
    env:
      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true # Forces Node 24 runtime
    steps:
      - name: Trigger my-documents build
        uses: peter-evans/repository-dispatch@v3
        with:
          token: ${{ secrets.DOCS_TRIGGER_PAT }}
          repository: fchastanet/my-documents
          event-type: trigger-docs-rebuild
          client-payload: |
            {
              "repository": "${{ github.repository }}",
              "ref": "${{ github.ref }}",
              "sha": "${{ github.sha }}"
            }

Required Secret: DOCS_TRIGGER_PAT - Personal Access Token with repo scope

3.7. Theme Customization

Shared Customizations: shared/

Contains:

  • Layouts: Custom Hugo templates override Docsy defaults
  • Assets: Custom SCSS variables, additional CSS/JS
  • Archetypes: Content templates for new pages

Per-Site Customization: Via configuration overrides in configs/{site}.yaml

Examples:

  • Theme colors: params.ui.navbar_bg_color: '#007bff' (blue for bash-compiler)
  • Custom links in footer or navbar
  • Site-specific SEO keywords and description
  • Logo overrides

Mount Strategy: Defined in configs/_base.yaml

module:
  mounts:
    - {source: shared/layouts, target: layouts}
    - {source: shared/assets, target: assets}
    - {source: shared/archetypes, target: archetypes}
    - {source: content, target: content}
    - {source: static, target: static}

Result: Shared resources available to all sites, with per-site override capability

4. Lessons Learned & Future Considerations

4.1. GitHub App Migration from Deploy Keys

Initial Approach: Deploy keys for each repository

  • Setup: Generate SSH key pair per repository, store private key in my-documents secrets
  • Secrets Required: DEPLOY_KEY_BASH_COMPILER, DEPLOY_KEY_BASH_TOOLS, etc. (4+ secrets)
  • Management: Per-repository key addition in Settings → Deploy keys

Problem: Scalability and management overhead

Migration to GitHub Apps:

Advantages:

  • Fine-grained permissions: Only Contents and Pages write access (vs full repo access)
  • Centralized management: One app for all repositories
  • Better security: Automatic token expiration and rotation
  • Audit trail: All actions logged under app identity
  • No SSH management: HTTPS with tokens instead of SSH keys
  • Easily revocable: Instant access revocation without key regeneration
  • Scalable: Add/remove repositories without creating new keys
  • Secrets reduction: 2 secrets (app ID + private key) vs 4+ deploy keys

GitHub Official Recommendation:

“We recommend using GitHub Apps with permissions scoped to specific repositories for enhanced security and more granular access control.”

Implementation: See doc/ai/2026-02-18-github-app-migration.md for complete migration guide

Outcome: Significantly improved security posture and simplified credential management

4.2. Trade-offs Discovered

4.2.1. All-Site Rebuild Trade-off

Trade-off: All sites rebuild together when any site content changes

Mitigation Strategies:

  • fail-fast: false in matrix strategy - One site failure doesn’t block others
  • ✅ Parallel execution - All 5 sites build simultaneously (~60s total)
  • ✅ Path-based triggers - Only rebuild when relevant files change
  • ✅ Concurrency control - Cancel duplicate builds

Acceptance Rationale:

  • Related documentation sites benefit from synchronized updates
  • Total build time (60s) acceptable for documentation updates
  • Ensures all sites stay consistent with latest shared resources
  • Simpler mental model: one build updates everything

4.2.2. Authentication Complexity

Trade-off: Initial setup requires GitHub App creation and secret configuration

Mitigation:

  • ✅ One-time setup effort well-documented
  • ✅ Improved security worth the complexity
  • ✅ Scales better than deploy keys (no per-repo setup needed for new sites)

Outcome: Initial investment pays off with easier ongoing management

4.2.3. Configuration Flexibility vs Consistency

Trade-off: Centralized configuration limits per-site flexibility

Mitigation:

  • ✅ Site-specific override files in configs/{site}.yaml
  • ✅ Shared base with override capability provides best of both worlds
  • ✅ yq deep-merge preserves flexibility where needed

Outcome: Achieved balance between consistency and customization

4.3. Best Practices Identified

4.3.1. Configuration Management

  • Use YAML deep-merge: yq eval-all properly merges nested structures
  • Separate concerns: Base configuration vs site-specific overrides
  • Version control everything: All configs in git
  • Document override patterns: Clear examples in base config

4.3.2. Build Optimization

  • Parallel matrix builds: Leverage GitHub Actions matrix for speed
  • Minimal checkout: Only fetch what’s needed (depth, paths)
  • Careful path triggers: Avoid unnecessary builds
  • Cancel redundant builds: Use concurrency groups

4.3.3. Dependency Management

  • Pin versions: Hugo 0.160.1, Go 1.24 (reproducible builds)
  • Cache when possible: Hugo modules could be cached (future optimization)
  • Minimal dependencies: yq only additional tool needed

4.3.4. Security

  • GitHub Apps over deploy keys: Better security model
  • Minimal permissions: Only what’s needed (Contents write, Pages write)
  • Secret scoping: Secrets only in orchestrator repo
  • Audit logging: GitHub App actions fully logged

4.4. Future Considerations

4.4.1. Potential Optimizations

Hugo Module Caching:

  • Current: Hugo modules downloaded fresh each build
  • Future: Cache Go modules directory to speed up builds
  • Benefit: Reduce build time by 5-10s per site

Conditional Site Builds:

  • Current: All sites build on any trigger
  • Future: Parse repository_dispatch payload to build only affected site
  • Benefit: Faster feedback for single-site changes
  • Trade-off: More complex logic, potential consistency issues

Build Artifact Reuse:

  • Current: Each site built independently
  • Future: Share Hugo module downloads across matrix jobs
  • Benefit: Reduced redundant network calls

4.4.2. Scalability Considerations

Adding New Documentation Sites:

  1. Create new repository with content
  2. Add trigger workflow (2-minute setup)
  3. Add site config to my-documents/configs/{new-site}.yaml
  4. Add site to matrix in build-all-sites.yml
  5. Install GitHub App on new repository
  6. Done - automatic builds immediately available

Estimated effort: 15-30 minutes per new site

4.4.3. Alternative Approaches for Future Projects

When Decentralized Makes Sense:

  • Truly independent projects (not related documentation)
  • Different teams with different update schedules
  • Need for isolated failure handling
  • Different Hugo/Docsy versions per project

When to Reconsider:

  • More than 10 sites (build time may become issue)
  • Sites diverge significantly in requirements
  • Team structure changes (separate maintainers per site)
  • Different deployment targets (not all GitHub Pages)

4.5. Success Metrics

Achieved:

  • SEO Improvement: 2/10 (Docsify) → 9/10 (Hugo with Docsy)
  • Build Time: ~60s for all 5 sites (parallel)
  • Maintenance Reduction: One workflow update vs 5× separate updates
  • Consistency: 100% - All sites use same base configuration
  • Security: GitHub App authentication with fine-grained permissions
  • Deployment: Automatic on content changes
  • Developer Experience: Simplified per-repo structure (2 files vs 6)
  • Independent URLs: All 5 repositories maintain separate GitHub Pages URLs
  • Theme Sharing: Shared Docsy theme customizations across all sites

Continuous Improvement:

  • Monitor build times as content grows
  • Gather feedback on developer experience
  • Iterate on shared vs per-site customizations
  • Evaluate caching opportunities
  • Consider additional SEO optimization (structured data, etc.)

5. Conclusion

The Hugo migration successfully addressed the SEO limitations of Docsify while establishing a scalable, maintainable multi-site documentation architecture. The centralized orchestrator approach provides the right balance of consistency and flexibility for related Bash tooling documentation projects.

Key Success Factors:

  1. Right tool for the job: Hugo’s documentation focus and SEO capabilities
  2. Architectural alignment: Centralized approach matches project relationships
  3. Security improvement: GitHub App migration enhanced security posture
  4. Maintainability: Single-point updates reduce ongoing effort
  5. Flexibility preserved: Configuration overrides allow per-site customization

Documentation maintained and current as of: 2026-02-18

Related Resources:

1.4 - Trigger Reusable Workflow Documentation

Overview of the technical architecture and implementation details of the My Documents reusable workflow for triggering documentation builds

1. Overview

The trigger-docs-reusable.yml workflow is a reusable GitHub Actions workflow that enables dependent repositories (bash-compiler, bash-tools, bash-tools-framework, bash-dev-env) to trigger documentation builds in the centralized my-documents orchestrator.

Benefits:

  • No secrets required in dependent repositories (GitHub handles authentication automatically)
  • Centralized configuration - All authentication handled by GitHub App in my-documents
  • Configurable - Override defaults for organization, repository, URLs, etc.
  • Secure - Uses GitHub App authentication with automatic token expiration
  • Simple integration - Just a few lines in dependent repo workflows

2. Quick Start

2.1. Basic Usage

Create .github/workflows/trigger-docs.yml in your dependent repository:

name: Trigger Documentation Build

on:
  push:
    branches: [master]
    paths:
      - content/**
      - static/**
      - go.mod
      - go.sum
  workflow_dispatch:

jobs:
  trigger-docs:
    uses: |-
      fchastanet/my-documents/.github/workflows/trigger-docs-reusable.yml@master
    secrets: inherit

That’s it! No secrets to configure, no tokens to manage.

3. How It Works

3.1. Architecture

      ┌─────────────────────────┐
      │  Dependent Repository   │
      │  (e.g., bash-compiler)  │
      │                         │
      │  Push to master branch  │
      │  ├─ content/**          │
      │  └─ static/**           │
      └────────────┬────────────┘
                   │ workflow_call
┌─────────────────────────────────────┐
│  my-documents Repository            │
│                                     │
│  .github/workflows/                 │
│    trigger-docs-reusable.yml        │
│                                     │
│  ┌────────────────────────────────┐ │
│  │ 1. Generate GitHub App Token   │ │
│  │    (using DOC_APP_ID secret)   │ │
│  └────────────┬───────────────────┘ │
│               │                     │
│  ┌────────────▼───────────────────┐ │
│  │ 2. Trigger repository_dispatch │ │
│  │    event in my-documents       │ │
│  └────────────┬───────────────────┘ │
└───────────────┼─────────────────────┘
                │ repository_dispatch
┌─────────────────────────────────────┐
│  my-documents Repository            │
│                                     │
│  .github/workflows/                 │
│    build-all-sites.yml              │
│                                     │
│  Builds all 5 documentation sites   │
│  Deploys to GitHub Pages            │
└─────────────────────────────────────┘

3.2. Authentication Flow

  1. Calling workflow runs in dependent repository context
  2. Reusable workflow executes in my-documents repository context
  3. GitHub App token generated using my-documents secrets:
    • DOC_APP_ID - GitHub App ID
    • DOC_APP_PRIVATE_KEY - GitHub App private key
  4. Token used to trigger repository_dispatch event
  5. Build workflow starts automatically in my-documents

Security Benefits:

  • No PAT tokens needed in dependent repositories
  • No secrets management in dependent repos
  • Automatic token expiration (1 hour)
  • Fine-grained permissions (Contents: write, Pages: write)
  • Centralized audit trail

4. Configuration

4.1. Input Parameters

All inputs are optional with sensible defaults:

InputDescriptionDefault
target_orgTarget organization/userfchastanet
target_repoTarget repository namemy-documents
event_typeRepository dispatch event typetrigger-docs-rebuild
docs_url_baseDocumentation URL basehttps://devlab.top/
workflow_filenameWorkflow filename to monitorbuild-all-sites.yml
source_repoSource repository${{ github.repository }}
(auto-detected if not provided)

4.2. Advanced Usage Examples

4.2.1. Custom Documentation URL

jobs:
  trigger-docs:
    uses: |-
      fchastanet/my-documents/.github/workflows/trigger-docs-reusable.yml@master
    with:
      docs_url_base: https://docs.example.com
    secrets: inherit

4.2.2. Different Target Repository

jobs:
  trigger-docs:
    uses: |-
      fchastanet/my-documents/.github/workflows/trigger-docs-reusable.yml@master
    with:
      target_org: myOrg
      target_repo: my-docs
      workflow_filename: build-docs.yml
    secrets: inherit

4.2.3. Manual Trigger with Custom Event Type

jobs:
  trigger-docs:
    uses: |-
      fchastanet/my-documents/.github/workflows/trigger-docs-reusable.yml@master
    with:
      event_type: custom-docs-rebuild
    secrets: inherit

5. Complete Example

Here’s a complete example for a dependent repository:

name: Trigger Documentation Build

on:
  # Trigger on content changes
  push:
    branches: [master]
    paths:
      - content/**        # Hugo content
      - static/**         # Static assets
      - go.mod            # Hugo modules
      - go.sum            # Hugo module checksums
      - configs/**        # If using custom configs

  # Allow manual triggering
  workflow_dispatch:

  # Trigger on releases
  release:
    types: [published]

jobs:
  trigger-docs:
    name: Trigger Documentation Build
    uses: |-
      fchastanet/my-documents/.github/workflows/trigger-docs-reusable.yml@master
    secrets: inherit

6. Secrets Configuration

6.1. In my-documents Repository

The reusable workflow requires these secrets to be configured in the my-documents repository:

SecretDescriptionHow to Get
DOC_APP_IDGitHub App IDFrom GitHub App settings
DOC_APP_PRIVATE_KEYGitHub App private key (PEM format)Generated when creating GitHub App

Setting up secrets:

  1. Go to https://github.com/fchastanet/my-documents/settings/secrets/actions
  2. Add DOC_APP_ID with your GitHub App ID
  3. Add DOC_APP_PRIVATE_KEY with the private key content

6.2. In Dependent Repositories

No secrets needed! The secrets: inherit directive allows the reusable workflow to access my-documents secrets when running.

7. Understanding Secrets: Inherit and Access Control

7.1. What is secrets: inherit?

secrets: inherit is a GitHub Actions feature that allows a reusable workflow to access repository secrets from the calling workflow’s repository when in the same repository context.

Important distinction:

When a dependent repository (like bash-compiler) calls this reusable workflow with secrets: inherit:

jobs:
  trigger-docs:
    uses: |-
      fchastanet/my-documents/.github/workflows/trigger-docs-reusable.yml@master
    secrets: inherit

It means:

“Pass any secrets from bash-compiler repository to the reusable workflow”

NOT:

“Pass secrets from my-documents to bash-compiler”

7.2. How Does It Work for Dependent Repositories?

The key to understanding this is the execution context:

  1. Workflow file location: .github/workflows/trigger-docs-reusable.yml lives in my-documents
  2. Calling workflow location: .github/workflows/trigger-docs.yml lives in bash-compiler (or other dependent repo)
  3. Execution context: When bash-compiler calls the reusable workflow, the reusable workflow still runs in the my-documents context

This means:

  • The reusable workflow has access to my-documents’ secrets, not bash-compiler’s secrets
  • secrets: inherit tells the reusable workflow “use my (the calling repo’s) secrets if needed”
  • But since the workflow runs in my-documents context, it automatically has access to my-documents’ secrets anyway

7.3. Secret Access Hierarchy

GitHub Actions processes reusable workflows within the repository where they’re defined:

┌────────────────────────────────────────────────────────────────────────────────────┐
│  bash-compiler repo                                                                │
│                                                                                    │
│  .github/workflows/                                                                │
│    trigger-docs.yml                                                                │
│                                                                                    │
│  calls: fchastanet/my-documents/.github/workflows/trigger-docs-reusable.yml@master │
│    secrets: inherit                                                                │
└───────────────────────────────────────┬────────────────────────────────────────────┘
                                        │ workflow_call (context: my-documents)
                      ┌─────────────────────────────────┐
                      │  my-documents repo              │
                      │  (workflow context)             │
                      │                                 │
                      │  .github/workflows/             │
                      │    trigger-docs-reusable.yml    │
                      │                                 │
                      │  ✅ Can access:                 │
                      │  - DOC_APP_ID                   │
                      │  - DOC_APP_PRIVATE_KEY          │
                      │  (my-documents secrets)         │
                      │                                 │
                      │  ❌ Cannot directly access:     │
                      │  - bash-compiler secrets        │
                      └─────────────────────────────────┘

7.4. Why This Workflow Can’t Be Used by Others

This workflow is tightly coupled to the my-documents infrastructure:

7.4.1. Reason 1: GitHub App is Organization-Specific

The workflow uses DOC_APP_ID and DOC_APP_PRIVATE_KEY secrets that are:

  • Configured only in the my-documents repository
  • Created from a GitHub App installed only on:
    • fchastanet/my-documents
    • fchastanet/bash-compiler
    • fchastanet/bash-tools
    • fchastanet/bash-tools-framework
    • fchastanet/bash-dev-env

If someone from outside this organization tries to use the workflow:

# In their-org/their-repo
jobs:
  trigger-docs:
    uses: |-
      fchastanet/my-documents/.github/workflows/trigger-docs-reusable.yml@master
    secrets: inherit

What happens:

  1. Workflow starts in their-org/their-repo context (calling workflow)
  2. Reusable workflow executes in fchastanet/my-documents context
  3. Reusable workflow tries to access DOC_APP_ID and DOC_APP_PRIVATE_KEY
  4. These secrets don’t exist in their-repo, so secrets: inherit doesn’t provide them
  5. The workflow fails with authentication error
Error: The variable has not been set, or it has been set to an empty string.
Evaluating: secrets.DOC_APP_ID

7.4.2. Reason 2: GitHub App Has No Access to Other Organizations

The GitHub App is installed only on specific fchastanet repositories:

  • When workflow tries to trigger repository_dispatch in my-documents using the app token
  • The token is only valid for repositories where the app is installed
  • If someone tries to point it to their own my-documents fork, the app has no permission

Error example:

Error: Resource not accessible by integration
  at https://api.github.com/repos/their-org/their-docs/dispatches

7.4.3. Reason 3: Secrets Are Repository-Specific

GitHub Actions secrets are stored at three levels:

LevelScopeAccessible By
RepositorySingle repositoryWorkflows in that repository only
EnvironmentSpecific deployment environmentWorkflows targeting that environment
OrganizationAll repositories in organizationAll workflows in the organization

My-documents secrets are stored at the repository level:

  • Only accessible to workflows executing in my-documents context
  • Not accessible to workflows in other organizations
  • Not inherited by other repositories, even if they call the reusable workflow

7.5. Practical Example: Why It Fails

Scenario: User john forks my-documents to john/my-documents-fork and tries to use the workflow:

# In john/bash-compiler (dependent repo fork)
jobs:
  trigger-docs:
    uses: |-
      john/my-documents-fork/.github/workflows/trigger-docs-reusable.yml@master
    secrets: inherit

Execution flow:

1. bash-compiler workflow starts (context: john)
   ❌ john/my-documents-fork doesn't have DOC_APP_ID or DOC_APP_PRIVATE_KEY secrets

2. Reusable workflow starts (context: john/my-documents-fork)
   ❌ Tries to access secrets.DOC_APP_ID
   ❌ Secrets don't exist in john/my-documents-fork
   ❌ secrets: inherit doesn't help (no secrets in john/bash-compiler either)

3. GitHub App access attempt
   ❌ GitHub App not installed on john/my-documents-fork
   ❌ Authentication fails with 403 error

7.6. How Someone Else Could Create Their Own Version

If someone wanted to use this pattern for their own orchestrator:

  1. Create their own GitHub App

    • In their organization settings
    • With Contents: write and Pages: write permissions
    • Install on their repositories
  2. Set up secrets in their my-documents repository

    DOC_APP_ID = their-app-id
    DOC_APP_PRIVATE_KEY = their-private-key
    
  3. Create their own reusable workflow

    • Copy and adapt the trigger-docs-reusable.yml
    • Reference their own secrets
    • Change target_org default to their organization
  4. Update dependent repositories

    • Point to their reusable workflow
    • Use secrets: inherit in their calls

Example for their-org:

# In their-org/bash-compiler
jobs:
  trigger-docs:
    uses: |-
      their-org/my-docs-orchestrator/.github/workflows/trigger-docs-reusable.yml@master
    secrets: inherit
    # This now references their-org's secrets, not fchastanet's

7.7. Summary: Why This Workflow is Fchastanet-Only

ComponentWhy It’s Fchastanet-SpecificCan Be Generalized?
Workflow logicGeneric, reusable for any workflow✅ Yes, with different inputs
DOC_APP_ID secretSpecific to fchastanet’s GitHub App❌ No, organization-specific
DOC_APP_PRIVATE_KEY secretSpecific to fchastanet’s GitHub App❌ No, organization-specific
Target repository (default)Hardcoded to my-documents✅ Yes, via target_repo input
Target organization (default)Hardcoded to fchastanet✅ Yes, via target_org input
GitHub App installationOnly on fchastanet repositories❌ No, would need own app

7.8. Conclusion

The secrets: inherit mechanism is elegant for internal workflows within an organization because:

  • For dependent repos in fchastanet: They can call the workflow without managing secrets (works perfectly)
  • For external users: They cannot use this workflow as-is because the GitHub App and secrets are organization-specific
  • This is intentional: It provides security and prevents unauthorized access to the build orchestration

This is not a limitation but a security feature - the workflow is designed to work only within the fchastanet organization where the GitHub App is installed.

8. Workflow Outputs

The workflow provides rich output and summaries:

8.1. Console Output

🔔 Triggering documentation build in fchastanet/my-documents...
✅ Successfully triggered docs build in fchastanet/my-documents
📖 Documentation will be updated at: https://bash-compiler.devlab.top/
ℹ️  Note: Documentation deployment may take 2-5 minutes

8.2. GitHub Actions Summary

The workflow creates a detailed summary visible in the Actions UI:

### 8.3. ✅ Documentation build triggered

**Source Repository:** `fchastanet/bash-compiler`
**Target Repository:** `fchastanet/my-documents`
**Commit:** `abc123def456`
**Triggered by:** `fchastanet`

🔗 [View build status](https://github.com/fchastanet/my-documents/actions/workflows/build-all-sites.yml)
📖 [View documentation](https://bash-compiler.devlab.top/)

9. Troubleshooting

9.1. Build Not Triggered

Symptoms:

  • Workflow runs successfully but build doesn’t start
  • HTTP 204 response but no activity in my-documents

Possible Causes:

  1. GitHub App not installed on target repository

    • Solution: Install the GitHub App on my-documents repository
  2. GitHub App permissions insufficient

    • Solution: Ensure app has Contents: write permission
  3. Event type mismatch

    • Solution: Verify event_type input matches what build-all-sites.yml expects

9.2. Authentication Failures

Symptoms:

  • HTTP 401 (Unauthorized) or 403 (Forbidden) errors
  • “Resource not accessible by integration” error

Possible Causes:

  1. Secrets not configured in my-documents

    • Solution: Add DOC_APP_ID and DOC_APP_PRIVATE_KEY secrets
  2. GitHub App private key incorrect

    • Solution: Regenerate private key in GitHub App settings
  3. GitHub App permissions revoked

    • Solution: Reinstall GitHub App on repositories

9.3. Workflow Not Found

Symptoms:

  • “Unable to resolve action” error
  • “Workflow file not found” error

Possible Causes:

  1. Wrong branch reference

    • Solution: Use @master not @main (my-documents uses master branch)
  2. Workflow file renamed or moved

    • Solution: Verify file exists at .github/workflows/trigger-docs-reusable.yml

9.4. Debug Mode

Enable debug logging in dependent repository:

jobs:
  trigger-docs:
    uses: |-
      fchastanet/my-documents/.github/workflows/trigger-docs-reusable.yml@master
    secrets: inherit

Then enable debug logs in repository settings:

  1. Go to repository settings → Secrets and variables → Actions
  2. Add repository variable: ACTIONS_STEP_DEBUG = true
  3. Add repository variable: ACTIONS_RUNNER_DEBUG = true

10. Migration Guide

10.1. From Old Trigger Workflow

If you’re migrating from the old PAT-based trigger workflow:

Old approach (deprecated):

jobs:
  trigger:
    runs-on: ubuntu-latest
    env:
      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true # Forces Node 24 runtime
    steps:
      - name: Trigger my-documents build
        run: |
          curl -X POST \
            -H "Authorization: token ${{ secrets.DOCS_BUILD_TOKEN }}" \
            ...

New approach (recommended):

jobs:
  trigger-docs:
    uses: |-
      fchastanet/my-documents/.github/workflows/trigger-docs-reusable.yml@master
    secrets: inherit

Benefits of migration:

  • ✅ Remove DOCS_BUILD_TOKEN secret from dependent repository
  • ✅ Simpler workflow (3 lines vs 50+ lines)
  • ✅ Centralized authentication
  • ✅ Automatic token management
  • ✅ Better security (GitHub App vs PAT)

11. Best Practices

11.1. Trigger Paths

Only trigger on content changes to avoid unnecessary builds:

on:
  push:
    branches: [master]
    paths:
      - content/**        # Documentation content
      - static/**         # Static assets
      - go.mod            # Hugo modules (theme updates)
      - go.sum

Don’t trigger on:

  • Test files
  • CI configuration changes
  • Source code changes (unless they affect docs)
  • README updates (unless it’s documentation content)

11.2. Concurrency Control

Prevent multiple concurrent builds:

jobs:
  trigger-docs:
    uses: |-
      fchastanet/my-documents/.github/workflows/trigger-docs-reusable.yml@master
    secrets: inherit
    concurrency:
      group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
      cancel-in-progress: true

11.3. Conditional Triggers

Only trigger for certain branches:

jobs:
  trigger-docs:
    if: github.ref == 'refs/heads/master'
    uses: |-
      fchastanet/my-documents/.github/workflows/trigger-docs-reusable.yml@master
    secrets: inherit

12. FAQ

12.1. Q: Do I need to configure secrets in my dependent repository?

A: No! When using secrets: inherit, the reusable workflow can access secrets from my-documents repository.

12.2. Q: Can I test the workflow before merging to master?

A: Yes, add workflow_dispatch trigger and manually run it from the Actions tab.

12.3. Q: How long does documentation deployment take?

A: Typically 2-5 minutes:

  • Trigger: ~5 seconds
  • Build (all sites): ~60 seconds
  • Deployment: ~1-3 minutes (GitHub Pages propagation)

12.4. Q: Can I use this with my own organization?

A: Yes, override target_org and target_repo inputs. You’ll need to set up your own GitHub App.

12.5. Q: What if the build fails?

A: Check the build status link in the workflow summary. The trigger workflow will still succeed; failures happen in the build workflow.

12.6. Q: Can I trigger builds for multiple repositories?

A: Yes, create multiple jobs in your workflow, each calling the reusable workflow with different source_repo values.

14. Support

For issues or questions:

  1. Check Troubleshooting section
  2. Review GitHub Actions logs
  3. Create an issue in my-documents repository

1.5 - Quick Reference - Hugo Site Development

A quick reference guide for developing and maintaining the Hugo documentation site

1. Local Development

1.1. Start

# Download dependencies (first time only)
hugo mod get -u

# Start development server
hugo server -D

# Open browser
# http://localhost:1313/my-documents/

1.2. Auto-reload

  • Edit markdown files
  • Browser auto-refreshes
  • Press Ctrl+C to stop

2. Adding Content

2.1. New Page in Existing Section

hugo new docs/bash-scripts/my-page.md

Edit the file with frontmatter:

---
title: My New Page
description: Brief description for SEO
weight: 10
categories: [Bash]
tags: [bash, example]
---

Your content here...

2.2. New Section

Create directory in content/en/docs/ and _index.md:

mkdir -p content/en/docs/new-section
touch content/en/docs/new-section/_index.md

2.3. Frontmatter Fields

---
title: Page Title              # Required, shown as H1
description: SEO description   # Required, used in meta tags
weight: 10                      # Optional, controls ordering (lower = earlier)
categories: [category-name]    # Optional, for content organization
tags: [tag1, tag2]             # Optional, for tagging
---

3. Content Organization

content/en/docs/
├── bash-scripts/          # Weight: 10 (first)
├── howtos/               # Weight: 20
│   └── howto-write-jenkinsfile/  # Subsection
├── lists/                # Weight: 30
└── other-projects/       # Weight: 40 (last)

Navigation: Automatic based on directory structure + weight frontmatter

4. Images and Assets

Place in static/ directory:

static/
├── howto-write-dockerfile/    # For Dockerfile guide images
├── howto-write-jenkinsfile/   # For Jenkins guide images
└── your-section/              # Create as needed

Reference in markdown:

![Alt text](/docs/docker/assets/Alpine-vs-Ubuntu.webp)

5. Common Docsy Shortcodes

5.1. Info Box


<div class="pageinfo pageinfo-primary">

This is an informational box.

</div>

5.2. Alert

<div class="alert alert-warning" role="alert"><div class="h4 alert-heading" role="heading">Warning</div>


This is a warning message.
</div>

5.3. Tabbed Content




<ul class="nav nav-tabs" id="tabs-2" role="tablist">
  <li class="nav-item">
      <button class="nav-link active"
          id="tabs-02-00-tab" data-bs-toggle="tab" data-bs-target="#tabs-02-00" role="tab"
          aria-controls="tabs-02-00" aria-selected="true">
        Tab 1
      </button>
    </li><li class="nav-item">
      <button class="nav-link"
          id="tabs-02-01-tab" data-bs-toggle="tab" data-bs-target="#tabs-02-01" role="tab"
          aria-controls="tabs-02-01" aria-selected="false">
        Tab 2
      </button>
    </li>
</ul>

<div class="tab-content" id="tabs-2-content">
    <div class="tab-pane fade show active"
        id="tabs-02-00" role="tabpanel" aria-labelled-by="tabs-02-00-tab" tabindex="2">
        <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">&lt;p&gt;Content for tab 1&lt;/p&gt;</span></span></code></pre></div>
    </div>
    <div class="tab-pane fade"
        id="tabs-02-01" role="tabpanel" aria-labelled-by="tabs-02-01-tab" tabindex="2">
        <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">&lt;p&gt;Content for tab 2&lt;/p&gt;</span></span></code></pre></div>
    </div>
</div>

See full list: https://www.docsy.dev/docs/reference/shortcodes/

6. Code Blocks

Specify language for syntax highlighting:

```bash
#!/bin/bash
echo "Hello World"
```

```yaml
key: value
nested:
  item: value
```

```python
def hello():
    print("Hello World")
```

Use relative paths:

[Link text](/docs/bash-scripts/page-name/)
[Link text](/docs/section/_index/)

Hugo resolves these automatically.

8. Building for Production

# Build minified site
hugo --minify

# Output goes to public/ directory
# GitHub Actions handles deployment automatically

9. Content Guidelines

  • Line length: 120 characters max (enforced by mdformat)
  • Headers: Use ATX style (#, ##, ###)
  • Lists: 2-space indentation
  • Code blocks: Always specify language
  • Images: Include alt text
  • Links: Use relative paths for internal, full URLs for external

10. Spell Checking

Add technical terms to .cspell/bash.txt:

echo "newWord" >>.cspell/bash.txt
pre-commit run file-contents-sorter # auto-sorts

11. Git Workflow

  1. Branch: Always use master
  2. Commit: Detailed message with changes
  3. Push: Triggers linting and Hugo build
  4. CI/CD: GitHub Actions handles rest
git add .
git commit -m "Add new documentation on topic"
git push origin master

12. Troubleshooting

12.1. Hugo server won’t start

rm go.sum
hugo mod clean
hugo mod get -u
hugo server -D

12.2. Module not found errors

hugo version # Check it says "extended"
hugo mod get -u

12.3. Build artifacts in way

rm -rf resources/ public/
hugo --minify
  • Check relative path is correct
  • Verify file exists in expected location
  • Internal links should start with /docs/

13. File Locations

ItemPath
Site confighugo.yaml
Home pagecontent/en/_index.html
Docs homecontent/en/docs/_index.md
Bash guidescontent/en/docs/bash-scripts/
How-TO guidescontent/en/docs/howtos/
Listscontent/en/docs/lists/
Imagesstatic/section-name/
Archetypesarchetypes/*.md
Theme confighugo.yaml params section

14. SEO Best Practices

  • ✅ Use descriptive titles and descriptions
  • ✅ Add weight to control ordering
  • ✅ Use categories and tags
  • ✅ Include proper alt text on images
  • ✅ Link to related content
  • ✅ Use clear heading hierarchy
  • ✅ Keep page descriptions under 160 chars

15. Submitting to Search Engines

  1. Build site: hugo --minify (GitHub Actions does this)
  2. GitHub Actions deploys to GitHub Pages
  3. Submit sitemap to search console:

16. Useful Commands

hugo server -D              # Run dev server
hugo --minify               # Build for production
hugo --printI18nWarnings    # Check for i18n issues
hugo --printPathWarnings    # Check path warnings
hugo --printUnusedTemplates # Check unused templates
pre-commit run -a           # Run all linters

17. Theme Customization

To override Docsy styles:

  1. Create /assets/scss/_custom.scss
  2. Add custom CSS
  3. Rebuild with hugo server

For more details: https://www.docsy.dev/docs/


Quick Links:

1.6 - Deploy Static Site on Cloudflare Pages

1. Introduction

I struggled a bit to deploy my web site. I wanted to have less cost as possible, I will explain here the difficulties I encountered and how I solved them.

2. Hugo

Originally my web site was using docsify which was giving a great website without a lot of configuration. But as it’s generating a single page application, it was not very good for SEO.

So I switched to Hugo which is a static site generator. It generates a static website that can be easily deployed on any hosting service.

You can check my other articles about Hugo:

3. Github Pages

With this new static site, I wanted to deploy it on Github Pages which is a free hosting service for static websites. But I encountered issues when I tried to reference the website through Google Search Console. Google Search Console was triggering “Page with redirect” error. After investigation, I understood that Github Pages is not redirecting http to https using 301 redirect but in an other way. This is an error for Google Search Console and it was not able to index my website. I didn’t find any solution to this problem, so I decided to switch to another hosting service.

4. Cloudflare Pages

I decided to switch to Cloudflare Pages which proposes a free hosting service for static websites.

I struggled a bit to understand how to deploy my website on Cloudflare Pages, but I finally succeeded.

There is a way to create cloudflare pages project from the Cloudflare dashboard. It proposes many worker configurations. You can for example choose to deploy React router, worker for static assets, React + postgresql, etc… But I wanted to deploy a static website without any worker configuration. There is also a way to make cloudflare to build your website from a github repository, but I have already a working github workflow to build my website and I just want to deploy the generated static files on cloudflare pages.

At one moment, I succeed via the Cloudflare dashboard to create a very simple project but I wasn’t able to do it again using the UI (maybe an upgrade of the dashboard or something else).

Anyway, I found a way using wrangler CLI.

4.1. Using wrangler CLI

Prerequisite, you need a cloudflare account.

Then you need to create a cloudflare API token with the permission to create and manage pages projects.

Save these credentials in your favorite password manager, you will need them later.

Then you need to login to cloudflare using wrangler CLI:

Here I had some issues with the authentication, as I’m using wsl, when authenticating through the windows browser, the authentication was not communicated to wsl.

So I installed google-chrome on wsl and authenticated through it, using this command line:

npx wrangler login --callback-host 0.0.0.0 --callback-port 8976 --browser=false

Open google-chrome on wsl, copy-paste the url provided by the command line and authenticate to cloudflare.

Then you need to create a cloudflare pages project using wrangler CLI:

npx wrangler pages project create bash-tools-framework --production-branch master --cwd build/bash-tools-framework/public

This command will create a cloudflare pages project named “bash-tools-framework” with the production branch “master” and the build output directory “build/bash-tools-framework/public”.

Already I could see my website deployed on cloudflare preview pages.

4.2. Connect to custom domain

Then I wanted to connect my custom domain to cloudflare pages.

I added a custom domain to my cloudflare pages project, using the cloudflare dashboard, and I added “devlab.top” as custom domain.

I simply replaced the default dns servers of my OVH custom domain by the cloudflare dns servers.

  • braelyn.ns.cloudflare.com
  • rudy.ns.cloudflare.com

Then I added a CNAME record to point my custom domain to the cloudflare pages project:

  • Type: CNAME
  • Name: www
  • Target: my-documents-dz6.pages.dev.

After a while, my custom domain was pointing to my cloudflare pages project and I could access my website through “www.devlab.top”. The web site you are reading right now.

5. Last step: automate deployment with Github Actions

Finally, I wanted to automate the deployment of my website using Github Actions.

You can check how I did it in my-documents build-site-action.yml.

There is one trick I had to figure out. This workflow is a sub action of my main workflow. So you need to ensure the secrets are passed from the main workflow to the sub workflow using secrets: inherit in the job triggering the sub workflow. See my-documents main workflow.

6. Conclusion

I finally succeeded and I hope this article will help you to deploy your static website on cloudflare pages and to automate the deployment using Github Actions.

2 - Brainstorming

In-depth brainstorming and analysis of documentation topics and strategies

1. Available Guides

  • Static Site Generation Migration Analysis - Analysis of migrating from Docsify to an SEO-optimized static site generator

2. Getting Started

Select a guide from the sidebar to begin.

3 - Bash Scripts

Best practices for writing Bash scripts

1. What You’ll Learn

This section covers:

  • Basic Best Practices - Foundational best practices for writing Bash scripts
  • Linux Commands Best Practices - Effective use of Linux commands in scripts
  • Bats Testing Framework - Testing Bash scripts with the Bats framework

2. Getting Started

Choose a topic from the sidebar to begin learning about Bash scripting best practices.

Articles in this section

TitleDescriptionUpdated
Basic Best PracticesFoundational best practices for writing Bash scripts2026-05-11 23:28:39 +0200 +0200
Linux Commands Best PracticesBest practices for using Linux commands in Bash scripts2026-05-11 23:28:39 +0200 +0200
Bats Testing FrameworkBest practices for testing Bash scripts with Bats framework2026-05-11 23:28:39 +0200 +0200

3.1 - Basic Best Practices

Foundational best practices for writing Bash scripts

1. External references

2. General best practices

  • cat << 'EOF' avoid to interpolate variables

  • use builtin cd instead of cd, builtin pwd instead of pwd, … to avoid using customized aliased commands by the user In this framework, I added the command unalias -a || true to remove all eventual aliases and also ensure to disable aliases expansion by using shopt -u expand_aliases. Because aliases have a very special way to load. In a script file changing an alias doesn’t occur immediately, it depends if script evaluated has been parsed yet or not. And alias changed in a function, will be applied outside of the function. But I experienced some trouble with this last rule, so I give up using aliases.

  • use the right shebang, avoid #!/bin/bash as bash binary could be in another folder (especially on alpine), use this instead #!/usr/bin/env bash

  • prefer to use printf vs echo

  • avoid global variables whenever possible, prefer using local

  • avoid to export variables whenever possible

3. escape quotes

help='quiet mode, doesn'\''t display any output'

# alternative
help="quiet mode, doesn't display any output"

4. Bash environment options

See Set bash builtin documentation

This framework uses these mode by default:

  • errexit
  • pipefail
  • errtrace

4.1. errexit (set -e | set -o errexit)

Check official doc but it can be summarized like this:

Exit immediately command returns a non-zero status.

I was considering this as a best practice because every non controlled command failure will stop your program. But actually

  • sometimes you need or expect a command to fail

Eg1: delete a folder that actually doesn’t exists. Use || true to ignore the error.

rm -Rf folder || true

Eg2: a command that expects to fail if conditions are not met. Using if will not stop the program on non-zero exit code.

if git diff-index --quiet HEAD --; then
  Log::displayInfo "Pull git repository '${dir}' as no changes detected"
  git pull --progress
  return 0
else
  Log::displayWarning "Pulling git repository '${dir}' avoided as changes detected"
fi
  • actually this feature is not well implemented everywhere

    • sometimes some commands that should fail doesn’t fail
    • the feature is not homogeneous across implementations
    • some commands expects to have non zero exit code
    • some commands exits with non zero error code but does not necessarily needs the program to exit
  • Finally it is preferable to check every command status code manually instead of relying to an automatic management.

4.1.1. Caveats with command substitution

#!/bin/bash
set -o errexit
echo $(exit 1)
echo $?

Output:

0

it is because echo has succeeded. the same result occurs even with shopt -s inherit_errexit (see below).

The best practice is to always assign command substitution to variable:

#!/bin/bash
set -o errexit
declare cmdOut
cmdOut=$(exit 1)
echo "${cmdOut}"
echo $?

Outputs nothing because the script stopped before variable affectation, return code is 1.

4.1.2. Caveats with process substitution

Consider this example that reads each line of the output of the command passed using process substitution in <(...)

parse() {
  local scriptFile="$1"
  local implementDirective
  while IFS='' read -r implementDirective; do
    echo "${implementDirective}"
  done < <(grep -E -e "^# IMPLEMENT .*$" "${scriptFile}")
}

If we execute this command with a non existent file, even if errexit, pipefail and inherit_errexit are set, the command will actually succeed.

It is because process substitution launch the command as as separated process. I didn’t find any clean way to manage this using process substitution (only workaround I found was to pass by file to pass the exit code to parent process).

So here the solution removing process substitution

parse() {
  local scriptFile="$1"
  local implementDirective
  grep -E -e "^# IMPLEMENT .*$" "${scriptFile}" | while IFS='' read -r implementDirective; do
    echo "${implementDirective}"
  done
}

But how to use readarray without using process substitution. Old code was:

declare -a interfacesFunctions
readarray -t interfacesFunctions < <(Compiler::Implement::mergeInterfacesFunctions "${COMPILED_FILE2}")
Compiler::Implement::validateInterfaceFunctions \
  "${COMPILED_FILE2}" "${INPUT_FILE}" "${interfacesFunctions[@]}"

I first think about doing this

declare -a interfacesFunctions
Compiler::Implement::mergeInterfacesFunctions "${COMPILED_FILE2}" | readarray -t interfacesFunctions

But interfacesFunctions was empty because readarray is run in another process, to avoid this issue, I could have used shopt -s lastpipe

But I finally transformed it to (the array in the same sub-shell so no issue):

Compiler::Implement::mergeInterfacesFunctions "${COMPILED_FILE2}" | {
  declare -a interfacesFunctions
  readarray -t interfacesFunctions
  Compiler::Implement::validateInterfaceFunctions \
    "${COMPILED_FILE2}" "${INPUT_FILE}" "${interfacesFunctions[@]}"
}

The issue with this previous solution is that commands runs in a sub-shell but using shopt -s lastpipe could solve this issue.

Another solution would be to simply read the array from stdin:

declare -a interfacesFunctions
readarray -t interfacesFunctions <<<"$(
  Compiler::Implement::mergeInterfacesFunctions "${COMPILED_FILE2}"
)"
Compiler::Implement::validateInterfaceFunctions \
  "${COMPILED_FILE2}" "${INPUT_FILE}" "${interfacesFunctions[@]}"

4.1.3. Process substitution is asynchronous

it is why you cannot retrieve the status code, a way to do that is to wait the process to finish

while read -r line; do
  echo "$line" &
done < <(
  echo 1
  sleep 1
  echo 2
  sleep 1
  exit 77
)

could be rewritten in

mapfile -t lines < <(
  echo 1
  sleep 1
  echo 2
  sleep 1
  exit 77
)
wait $!

for line in "${lines[@]}"; do
  echo "$line" &
done
sleep 1
wait $!
echo done

4.2. pipefail (set -o pipefail)

https://dougrichardson.us/notes/fail-fast-bash-scripting.html

If set, the return value of a pipeline is the value of the last (rightmost) command to exit with a non-zero status, or zero if all commands in the pipeline exit successfully. This option is disabled by default.

It is complementary with errexit, as if it not activated, the failure of command in pipe could hide the error.

Eg: without pipefail this command succeed

#!/bin/bash
set -o errexit
set +o pipefail # deactivate pipefail mode
foo | echo "a"  # 'foo' is a non-existing command
# Output:
# a
# bash: foo: command not found
# echo $? # exit code is 0
# 0

4.3. errtrace (set -E | set -o errtrace)

https://dougrichardson.us/notes/fail-fast-bash-scripting.html

If set, any trap on ERR is inherited by shell functions, command substitutions, and commands executed in a subShell environment. The ERR trap is normally not inherited in such cases.

4.4. nounset (set -u | set -o nounset)

https://dougrichardson.us/notes/fail-fast-bash-scripting.html

Treat unset variables and parameters other than the special parameters ‘@’ or ‘’, or array variables subscripted with ‘@’ or ‘’, as an error when performing parameter expansion. An error message will be written to the standard error, and a non-interactive shell will exit.

4.5. inherit error exit code in sub shells

https://dougrichardson.us/notes/fail-fast-bash-scripting.html

let’s see why using shopt -s inherit_errexit ?

set -e does not affect subShells created by Command Substitution. This rule is stated in Command Execution Environment:

subShells spawned to execute command substitutions inherit the value of the -e option from the parent shell. When not in POSIX mode, Bash clears the -e option in such subShells.

This rule means that the following script will run to completion, in spite of INVALID_COMMAND.

#!/bin/bash
# command-substitution.sh
set -e
MY_VAR=$(
  echo -n Start
  INVALID_COMMAND
  echo -n End
)
echo "MY_VAR is $MY_VAR"

Output:

./command-substitution.sh: line 4: INVALID_COMMAND: command not found
MY_VAR is StartEnd

shopt -s inherit_errexit, added in Bash 4.4 allows you to have command substitution parameters inherit your set -e from the parent script.

From the Shopt Builtin documentation:

If set, command substitution inherits the value of the errexit option, instead of unsetting it in the subShell environment. This option is enabled when POSIX mode is enabled.

So, modifying command-substitution.sh above, we get:

#!/bin/bash
# command-substitution-inherit_errexit.sh
set -e
shopt -s inherit_errexit
MY_VAR=$(
  echo -n Start
  INVALID_COMMAND
  echo -n End
)
echo "MY_VAR is $MY_VAR"

Output:

./command-substitution-inherit_errexit.sh: line 5: INVALID_COMMAND: command not found

4.6. posix (set -o posix)

Change the behavior of Bash where the default operation differs from the POSIX standard to match the standard (see Bash POSIX Mode). This is intended to make Bash behave as a strict superset of that standard.

5. Main function

An important best practice is to always encapsulate all your script inside a main function. One reason for this technique is to make sure the script does not accidentally do anything nasty in the case where the script is truncated. I often had this issue because when I change some of my bash framework functions, the pre-commit runs buildBinFiles command that can be recompiled itself. In this case the script fails.

another reason for doing this is to not execute the file at all if there is a syntax error.

Additionally you can add a snippet in order to avoid your function to be executed in the case where it is being source. The following code will execute main function if called as a script passing arguments, or will just import the main function if the script is sourced. See this stack overflow for more details

#!/usr/bin/env bash

main() {
  # main script
  set -eo pipefail
}

BASH_SOURCE=".$0"
[[ ".$0" != ".$BASH_SOURCE" ]] || main "$@"

6. Arguments

  • to construct complex command line, prefer to use an array
    • declare -a cmd=(git push origin :${branch})
    • then you can display the result using echo "${cmd[*]}"
    • you can execute the command using "${cmd[@]}"
  • boolean arguments, to avoid seeing some calls like this myFunction 0 1 0 with 3 boolean values. prefer to provide constants(using readonly) to make the call more clear like myFunction arg1False arg2True arg3False of course replacing argX with the real argument name. Eg: Filters::directive "${FILTER_DIRECTIVE_REMOVE_HEADERS}" You have to prefix all your constants to avoid conflicts.
  • instead of adding a new arg to the function with a default value, consider using an env variable that can be easily overridden before calling the function. Eg: SUDO=sudo Github::upgradeRelease ... It avoids to have to pass previous arguments that were potentially defaulted.

7. some commands default options to use

Check out 10-LinuxCommands-BestPractices.md

8. Variables

8.1. Variable declaration

  • ensure we don’t have any globals, all variables should be passed to the functions
  • declare all variables as local in functions to avoid making them global
  • local or declare multiple local a z
  • export readonly does not work, first readonly then export
  • avoid using export most of the times, export is needed only when variables has to be passed to child process.

8.2. variable naming convention

  • env variable that aims to be exported should be capitalized with underscore
  • local variables should conform to camelCase

8.3. Variable expansion

Shell Parameter Expansion

${PARAMETER:-WORD} vs ${PARAMETER-WORD}:

If the parameter PARAMETER is unset (was never defined) or null (empty), ${PARAMETER:-WORD} expands to WORD, otherwise it expands to the value of PARAMETER, as if it just was ${PARAMETER}.

If you omit the :(colon) like in ${PARAMETER-WORD}, the default value is only used when the parameter is unset, not when it was empty.

:warning: use this latter syntax when using function arguments in order to be able to reset a value to empty string, otherwise default value would be applied.

8.3.1. Examples

Extract directory from full file path: directory="${REAL_SCRIPT_FILE%/*}"

Extract file name from full file path: fileName="${REAL_SCRIPT_FILE##*/}"

8.4. Check if a variable is defined

if [[ -z ${varName+xxx} ]]; then
  ; # varName is not set
fi

Alternatively you can use this framework function Assert::varExistsAndNotEmpty

8.5. Variable default value

Always consider to set a default value to the variable that you are using.

Eg.: Let’s see this dangerous example

# Don't Do that !!!!
rm -Rf "${TMPDIR}/etc" || true

This could end very badly if your script runs as root and if ${TMPDIR} is not set, this script will result to do a rm -Rf /etc

Instead you can do that

rm -Rf "${TMPDIR:-/tmp}/etc" || true

8.6. Passing variable by reference to function

Always “scope” variables passed by reference. Scoping in bash means to find a name that is a low probability that the caller of the method names the parameter with the same name as in the function.

8.6.1. Example 1

Array::setArray() {
  local -n arr=$1
  local IFS=$2 -
  # set no glob feature
  set -f
  # shellcheck disable=SC2206,SC2034
  arr=($3)
}

Array::setArray arr , "1,2,3,"

this example results to the following error messages

bash: local: warning: arr: circular name reference
bash: warning: arr: circular name reference
bash: warning: arr: circular name reference

Tis example should be fixed by renaming local arr to a more “scoped” name.

Array::setArray() {
  local -n setArray_array=$1
  local IFS=$2 -
  # set no glob feature
  set -f
  # shellcheck disable=SC2206,SC2034
  setArray_array=($3)
}

Array::setArray arr , "1,2,3,"
# declare -p arr
# # output: declare -a arr=([0]="1" [1]="2" [2]="3")

8.6.2. Example 2

A more tricky example, here the references array is affected to local array, this local array has a conflicting name. This example does not produce any error messages.

Postman::Model::getValidCollectionRefs() {
  local configFile="$1"
  local -n getValidCollectionRefs=$2
  shift 2 || true
  local -a refs=("$@")
  # ...
  getValidCollectionRefs=("${refs[@]}")
}

local -a refs
Postman::Model::getValidCollectionRefs "file" refs a b c
declare -p refs # => declare -a refs

In Previous example, getValidCollectionRefs is well “scoped” but there is a conflict with the local refs array inside the function resulting in affectation not working. The correct way to do it is to scope also the variables affected to referenced variables

Postman::Model::getValidCollectionRefs() {
  local configFile="$1"
  local -n getValidCollectionRefsResult=$2
  shift 2 || true
  local -a getValidCollectionRefsSelection=("$@")
  # ...
  getValidCollectionRefsResult=("${getValidCollectionRefsSelection[@]}")
}

local -a refs
Postman::Model::getValidCollectionRefs "file" refs a b c
declare -p refs # => declare -a refs=([0]="a" [1]="b" [2]="c")

9. Capture output

You can use command substitution.

Eg:

local output
output="$(functionThatOutputSomething "${arg1}")"

9.1. Capture output and test result

local output
output="$(functionThatOutputSomething "${arg1}")" || {
  echo "error"
  exit 1
}

9.2. Capture output and retrieve status code

It’s advised to put it on the same line using ;. If it was on 2 lines, other commands could be put between the command and the status code retrieval, the status would not be the same command status.

Capture output and retrieve status code example

10. Array

  • read each line of a file to an array readarray -t var < /path/to/filename

11. Temporary directory

use ${TMPDIR:-/tmp}, TMPDIR variable does not always exist. or when mktemp is available, use dirname $(mktemp -u --tmpdir)

The variable TMPDIR is initialized in src/_includes/_commonHeader.sh used by all the binaries used in this framework.

12. Traps

when trapping EXIT do not forget to throw back same exit code otherwise exit code of last command executed in the trap is thrown

In this example rc variable contains the original exit code

cleanOnExit() {
  local rc=$?
  if [[ "${KEEP_TEMP_FILES:-0}" = "1" ]]; then
    Log::displayInfo "KEEP_TEMP_FILES=1 temp files kept here '${TMPDIR}'"
  elif [[ -n "${TMPDIR+xxx}" ]]; then
    Log::displayDebug "KEEP_TEMP_FILES=0 removing temp files '${TMPDIR}'"
    rm -Rf "${TMPDIR:-/tmp/fake}" >/dev/null 2>&1
  fi
  exit "${rc}"
}
trap cleanOnExit EXIT HUP QUIT ABRT TERM

13. Deal with SIGPIPE - exit code 141

related stackoverflow post

set -o pipefail makes exit code 141 being sent in some cases

Eg: with grep

bin/postmanCli --help | grep -q DESCRIPTION
echo "$? ${PIPESTATUS[@]}"

This is because grep -q exits immediately with a zero status as soon as a match is found. The zfs command is still writing to the pipe, but there is no reader (because grep has exited), so it is sent a SIGPIPE signal from the kernel and it exits with a status of 141.

Eg: or with head

echo "${longMultilineString}" | head -n 1

Finally I found this elegant stackoverflow solution:

handle_pipefail() {
  # ignore exit code 141 from simple command pipes
  # - use with: cmd1 | cmd2 || handle_pipefail $?
  (($1 == 141)) && return 0
  return $1
}

# then use it or test it as:
yes | head -n 1 || handle_pipefail $?
echo "ec=$?"

I added handle_pipefail as Bash::handlePipelineFailure in bash-tools-framework.

14. Performances analysis

generate a csv file with milliseconds measures

codeToMeasureStart=$(date +%s%3N)
# ... the code to measure
echo >&2 "printCurrentLine;$(($(date +%s%3N) - codeToMeasureStart))"

15. Bash Performance tips

15.1. Array::wrap2 performance improvement

Commit with performance improvement

manualTests/Array::wrap2Perf.sh:

  • displaying 12 lines (558 characters) 100 times
  • passed from ~10s to <1s (improved by 90%)

performance improvement using:

  • echo instead of string concatenation
  • string substitution instead of calling sed on each element
  • echo -e removed the need to do a loop on each character to parse ansi code and the need of Filters::removeAnsiCodes

3.2 - Linux Commands Best Practices

Best practices for using Linux commands in Bash scripts

1. some commands default options to use

2. Bash and grep regular expressions

  • grep regular expression [A-Za-z] matches by default accentuated character, if you don’t want to match them, use the environment variable LC_ALL=POSIX,
    • Eg: LC_ALL=POSIX grep -E -q '^[A-Za-z_0-9:]+$'
    • I added export LC_ALL=POSIX in all my headers, it can be overridden using a subShell

3.3 - Bats Testing Framework

Best practices for testing Bash scripts with Bats framework

1. use of default temp directory created by bats

Instead of creating yourself your temp directory, you can use the special variable BATS_TEST_TMPDIR, this directory is automatically destroyed at the end of the test except if the option --no-tempdir-cleanup is provided to bats command.

Exception: if you are testing bash traps, you would need to create your own directories to avoid unexpected errors.

2. avoid boilerplate code

using this include, includes most of the features needed when using bats

# shellcheck source=src/batsHeaders.sh
source "$(cd "${BATS_TEST_DIRNAME}/.." && pwd)/batsHeaders.sh"

It sets those bash features:

  • set -o errexit
  • set -o pipefail

It imports several common files like some additional bats features.

And makes several variables available:

3. Override an environment variable when using bats run

SUDO="" run Linux::Apt::update

4. Override a bash framework function

using stub is not possible because it does not support executable with special characters like ::. So the solution is just to override the function inside your test function without importing the original function of course. In tearDown method do not forget to use unset -f yourFunction

4 - How-To Guides

Step-by-step guides for various technologies

1. Available Guides

  • How to Write Dockerfiles - Best practices for efficient Dockerfiles
  • How to Write Docker Compose Files - Organizing multi-container applications
  • How to Write Jenkinsfiles - Complete Jenkins pipeline guide (10 articles)
  • Saml2Aws Setup - AWS access with SAML authentication

2. Getting Started

Select a guide from the sidebar to begin.

Articles in this section

TitleDescriptionUpdated
Saml2Aws SetupGuide to setting up and using Saml2Aws for AWS access2026-05-15 22:23:29 +0200 +0200
How to Write DockerfilesBest practices for writing efficient and secure Dockerfiles2026-05-15 22:23:28 +0200 +0200
How to Write Docker Compose FilesGuide to writing and organizing Docker Compose files2026-05-15 22:23:28 +0200 +0200
Debug HugoDebug Hugo: Quick guide to locate Hugo templates, use templateMetrics, override priorities, and fix common Docsy/Hugo template issues for faster debugging.2026-05-15 22:23:28 +0200 +0200

4.1 - How to Write Jenkinsfiles

Comprehensive guide to writing Jenkins pipelines and Jenkinsfiles

1. What You’ll Learn

  • How Jenkins works and its architecture
  • Declarative and scripted pipeline syntax
  • Creating and using Jenkins shared libraries
  • Jenkins best practices and configuration
  • Real-world Jenkinsfile examples with detailed annotations
  • Common recipes and troubleshooting tips

Articles in this section

TitleDescriptionUpdated
How Jenkins WorksUnderstanding Jenkins architecture and concepts2026-05-11 00:18:44 +0200 +0200
Jenkins Recipes and TipsUseful recipes and tips for Jenkins and Jenkinsfiles2026-05-11 00:18:44 +0200 +0200
Annotated Jenkinsfiles - Part 5Detailed Jenkinsfile examples with annotations2026-05-10 23:42:08 +0200 +0200
Annotated Jenkinsfiles - Part 1Detailed Jenkinsfile examples with annotations2026-02-22 08:00:00 +0100 +0100
Jenkins PipelinesDeclarative and scripted pipeline syntax2026-02-17 08:00:00 +0100 +0100
Jenkins LibraryCreating and using Jenkins shared libraries2026-02-17 08:00:00 +0100 +0100
Jenkins Best PracticesBest practices and patterns for Jenkins and Jenkinsfiles2026-02-17 08:00:00 +0100 +0100
Annotated Jenkinsfiles - Part 2More annotated Jenkinsfile examples2026-02-17 08:00:00 +0100 +0100
Annotated Jenkinsfiles - Part 3Additional Jenkinsfile pattern examples2026-02-17 08:00:00 +0100 +0100
Annotated Jenkinsfiles - Part 4Complex Jenkinsfile scenarios2026-02-17 08:00:00 +0100 +0100

4.1.1 - How Jenkins Works

Understanding Jenkins architecture and concepts

Source: https://www.jenkins.io/doc/book/managing/nodes/

Source glossary: https://www.jenkins.io/doc/book/glossary/

1. Jenkins Master Slave Architecture

Jenkins Master Slave Architecture

The Jenkins controller is the master node which is able to launch jobs on different nodes (machines) directed by an Agent. The Agent can the use one or several executors to execute the job(s) depending on configuration.

Jenkins is using Master/Slave architecture with the following components:

1.1. Jenkins controller/Jenkins master node

The central, coordinating process which stores configuration, loads plugins, and renders the various user interfaces for Jenkins.

The Jenkins controller is the Jenkins service itself and is where Jenkins is installed. It is a webserver that also acts as a “brain” for deciding how, when and where to run tasks. Management tasks (configuration, authorization, and authentication) are executed on the controller, which serves HTTP requests. Files written when a Pipeline executes are written to the filesystem on the controller unless they are off-loaded to an artifact repository such as Nexus or Artifactory.

1.2. Nodes

A machine which is part of the Jenkins environment and capable of executing Pipelines or jobs. Both the Controller and Agents are considered to be Nodes.

Nodes are the “machines” on which build agents run. Jenkins monitors each attached node for disk space, free temp space, free swap, clock time/sync and response time. A node is taken offline if any of these values go outside the configured threshold.

The Jenkins controller itself runs on a special built-in node. It is possible to run agents and executors on this built-in node although this can degrade performance, reduce scalability of the Jenkins instance, and create serious security problems and is strongly discouraged, especially for production environments.

1.3. Agents

An agent is typically a machine, or container, which connects to a Jenkins controller and executes tasks when directed by the controller.

Agents manage the task execution on behalf of the Jenkins controller by using executors. An agent is actually a small (170KB single jar) Java client process that connects to a Jenkins controller and is assumed to be unreliable. An agent can use any operating system that supports Java. Tools required for builds and tests are installed on the node where the agent runs; they can be installed directly or in a container (Docker or Kubernetes). Each agent is effectively a process with its own PID (Process Identifier) on the host machine.

In practice, nodes and agents are essentially the same but it is good to remember that they are conceptually distinct.

1.4. Executors

A slot for execution of work defined by a Pipeline or job on a Node. A Node may have zero or more Executors configured which corresponds to how many concurrent Jobs or Pipelines are able to execute on that Node.

An executor is a slot for execution of tasks; effectively, it is a thread in the agent. The number of executors on a node defines the number of concurrent tasks that can be executed on that node at one time. In other words, this determines the number of concurrent Pipeline stages that can execute on that node at one time.

The proper number of executors per build node must be determined based on the resources available on the node and the resources required for the workload. When determining how many executors to run on a node, consider CPU and memory requirements as well as the amount of I/O and network activity:

  • One executor per node is the safest configuration.
  • One executor per CPU core may work well if the tasks being run are small.
  • Monitor I/O performance, CPU load, memory usage, and I/O throughput carefully when running multiple executors on a node.

1.5. Jobs

A user-configured description of work which Jenkins should perform, such as building a piece of software, etc.

2. Jenkins dynamic node

Jenkins has static slave nodes and can trigger the generation of dynamic slave nodes

Jenkins Master/slave architecture

4.1.2 - Jenkins Pipelines

Declarative and scripted pipeline syntax

1. What is a pipeline ?

https://www.jenkins.io/doc/book/pipeline/

Jenkins Pipeline (or simply “Pipeline” with a capital “P”) is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins.

A continuous delivery (CD) pipeline is an automated expression of your process for getting software from version control right through to your users and customers. Every change to your software (committed in source control) goes through a complex process on its way to being released. This process involves building the software in a reliable and repeatable manner, as well as progressing the built software (called a “build”) through multiple stages of testing and deployment.

Pipeline provides an extensible set of tools for modeling simple-to-complex delivery pipelines “as code” via the Pipeline domain-specific language (DSL) syntax. View footnote 1

The definition of a Jenkins Pipeline is written into a text file (called a Jenkinsfile) which in turn can be committed to a project’s source control repository. View footnote 2 This is the foundation of “Pipeline-as-code”; treating the CD pipeline a part of the application to be versioned and reviewed like any other code.

2. Pipeline creation via UI

it’s not recommended but it’s possible to create a pipeline via the UI.

There are several drawbacks:

  • no code revision
  • difficult to read, understand

3. Groovy

Scripted and declarative pipelines are using groovy language.

Checkout https://www.guru99.com/groovy-tutorial.html to have a quick overview of this derived language check Wikipedia

4. Difference between scripted pipeline (freestyle) and declarative pipeline syntax

What are the main differences ? Here are some of the most important things you should know:

  • Basically, declarative and scripted pipelines differ in terms of the programmatic approach. One uses a declarative programming model and the second uses an imperative programming mode.
  • Declarative pipelines break down stages into multiple steps, while in scripted pipelines there is no need for this. Example below

Declarative and Scripted Pipelines are constructed fundamentally differently. Declarative Pipeline is a more recent feature of Jenkins Pipeline which:

  • provides richer syntactical features over Scripted Pipeline syntax, and
  • is designed to make writing and reading Pipeline code easier.
  • By default automatically checkout stage

Many of the individual syntactical components (or “steps”) written into a Jenkinsfile, however, are common to both Declarative and Scripted Pipeline. Read more about how these two types of syntax differ in Pipeline concepts and Pipeline syntax overview.

5. Declarative pipeline example

Pipeline syntax documentation

 pipeline {
   agent {
     // executed on an executor with the label 'some-label'
     // or 'docker', the label normally specifies:
     // - the size of the machine to use
     //   (eg.: Docker-C5XLarge used for build that needs a powerful machine)
     // - the features you want in your machine
     //   (eg.: docker-base-ubuntu an image with docker command available)
     label "some-label"
   }

   stages {
     stage("foo") {
       steps {
         // variable assignment and Complex global
         // variables (with properties or methods)
         // can only be done in a script block
         script {
           foo = docker.image('ubuntu')
           env.bar = "${foo.imageName()}"
           echo "foo: ${foo.imageName()}"
         }
       }
     }
     stage("bar") {
       steps{
         echo "bar: ${env.bar}"
         echo "foo: ${foo.imageName()}"
       }
     }
   }
 }

6. Scripted pipeline example

Scripted pipelines permit a developer to inject code, while the declarative Jenkins pipeline doesn’t. should be avoided actually, try to use jenkins library instead

node {

  git url: 'https://github.com/jfrogdev/project-examples.git'

  // Get Artifactory server instance, defined in the Artifactory Plugin
  // administration page.
  def server = Artifactory.server "SERVER_ID"

  // Read the upload spec and upload files to Artifactory.
  def downloadSpec =
       '''{
       "files": [
         {
            "pattern": "libs-snapshot-local/*.zip",
            "target": "dependencies/",
            "props": "p1=v1;p2=v2"
         }
       ]
   }'''

  def buildInfo1 = server.download spec: downloadSpec

  // Read the upload spec which was downloaded from github.
  def uploadSpec =
     '''{
     "files": [
       {
          "pattern": "resources/Kermit.*",
          "target": "libs-snapshot-local",
          "props": "p1=v1;p2=v2"
       },
       {
          "pattern": "resources/Frogger.*",
          "target": "libs-snapshot-local"
       }
      ]
   }'''


  // Upload to Artifactory.
  def buildInfo2 = server.upload spec: uploadSpec

  // Merge the upload and download build-info objects.
  buildInfo1.append buildInfo2

  // Publish the build to Artifactory
  server.publishBuildInfo buildInfo1
}

7. Why Pipeline?

Jenkins is, fundamentally, an automation engine which supports a number of automation patterns. Pipeline adds a powerful set of automation tools onto Jenkins, supporting use cases that span from simple continuous integration to comprehensive CD pipelines. By modeling a series of related tasks, users can take advantage of the many features of Pipeline:

  • Code: Pipelines are implemented in code and typically checked into source control, giving teams the ability to edit, review, and iterate upon their delivery pipeline.
  • Durable: Pipelines can survive both planned and unplanned restarts of the Jenkins controller.
  • Pausable: Pipelines can optionally stop and wait for human input or approval before continuing the Pipeline run.
  • Versatile: Pipelines support complex real-world CD requirements, including the ability to fork/join, loop, and perform work in parallel.
  • Extensible: The Pipeline plugin supports custom extensions to its DSL see jenkins doc and multiple options for integration with other plugins.

While Jenkins has always allowed rudimentary forms of chaining Freestyle Jobs together to perform sequential tasks, see jenkins doc Pipeline makes this concept a first-class citizen in Jenkins.

More information on Official jenkins documentation - Pipeline

4.1.3 - Jenkins Library

Creating and using Jenkins shared libraries

1. What is a jenkins shared library ?

As Pipeline is adopted for more and more projects in an organization, common patterns are likely to emerge. Oftentimes it is useful to share parts of Pipelines between various projects to reduce redundancies and keep code “DRY”

for more information check pipeline shared libraries

2. Loading libraries dynamically

As of version 2.7 of the Pipeline: Shared Groovy Libraries plugin, there is a new option for loading (non-implicit) libraries in a script: a library step that loads a library dynamically, at any time during the build.

If you are only interested in using global variables/functions (from the vars/ directory), the syntax is quite simple:

library 'my-shared-library'

Thereafter, any global variables from that library will be accessible to the script.

3. jenkins library directory structure

The directory structure of a Shared Library repository is as follows:

(root)
+- src        # Groovy source files
|   +- org
|       +- foo
|           +- Bar.groovy  # for org.foo.Bar class
|
+- vars       # The vars directory hosts script
              # files that are exposed as a variable in Pipelines
|   +- foo.groovy          # for global 'foo' variable
|   +- foo.txt             # help for 'foo' variable
|
+- resources  # resource files (external libraries only)
|   +- org
|      +- foo
|         +- bar.json      # static helper data for org.foo.Bar

4. Jenkins library

remember that jenkins library code is executed on master node

if you want to execute code on the node, you need to use jenkinsExecutor

usage of jenkins executor

String credentialsId = 'babee6c1-14fe-4d90-9da0-ffa7068c69af'
def lib = library(
    identifier: '[email protected]',
    retriever: modernSCM([
        $class: 'GitSCMSource',
        remote: '[email protected]:fchastanet/jenkins-library.git',
        credentialsId: credentialsId
    ])
)
// this is the jenkinsExecutor instance
def docker = lib.fchastanet.Docker.new(this)

Then in the library, it is used like this:

def status = this.jenkinsExecutor.sh(
  script: "docker pull ${cacheTag}", returnStatus: true
)

5. Jenkins library structure

I remarked that a lot of code was duplicated between all my Jenkinsfiles so I created this library https://github.com/fchastanet/jenkins-library

(root)
+- doc    # markdown files automatically generated
          # from groovy files by generateDoc.sh
+- src    # Groovy source files
|   +- fchastanet
|       +- Cloudflare.groovy     # zonePurge
|       +- Docker.groovy         # getTagCompatibleFromBranch
                                 # pullBuildPushImage, ...
|       +- Git.groovy            # getRepoURL, getCommitSha,
                                 # getLastPusherEmail,
                                 # updateConditionalGithubCommitStatus
|       +- Kubernetes.groovy     # deployHelmChart, ...
|       +- Lint.groovy           # dockerLint,
                                 # transform lighthouse report
                                 # to Warnings NG issues format
|       +- Mail.groovy           # sendTeamsNotification,
                                 # sendConditionalEmail, ...
|       +- Utils.groovy          # deepMerge, isCollectionOrArray,
                                 # deleteDirAsRoot,
                                 # initAws (could be moved to Aws class)
+- vars   # The vars directory hosts script files that
          # are exposed as a variable in Pipelines
|   +- dockerPullBuildPush.groovy #
|   +- whenOrSkip.groovy          #

6. external resource usage

If you need you check out how I used this repository https://github.com/fchastanet/jenkins-library-resources in jenkins_library (Linter) that hosts some resources to parse result files.

4.1.4 - Jenkins Best Practices

Best practices and patterns for Jenkins and Jenkinsfiles

1. Pipeline best practices

Official Jenkins pipeline best practices

Summary:

  • Make sure to use Groovy code in Pipelines as glue
  • Externalize shell scripts from Jenkins Pipeline
    • for better jenkinsfile readability
    • in order to test the scripts isolated from jenkins
  • Avoid complex Groovy code in Pipelines
    • Groovy code always executes on controller which means using controller resources(memory and CPU)
      • it is not the case for shell scripts
    • eg1: prefer using jq inside shell script instead of groovy JsonSlurper
    • eg2: prefer calling curl instead of groovy http request
  • Reducing repetition of similar Pipeline steps (eg: one sh step instead of severals)
    • group similar steps together to avoid step creation/destruction overhead
  • Avoiding calls to Jenkins.getInstance

2. Shared library best practices

Official Jenkins shared libraries best practices

Summary:

  • Do not override built-in Pipeline steps
  • Avoiding large global variable declaration files
  • Avoiding very large shared libraries

And:

  • import jenkins library using a tag
    • like in docker build, npm package with package-lock.json or python pip lock, it’s advised to target a given version of the library
      • because some changes could break
  • The missing part: we miss on this library unit tests
    • but each pipeline is a kind of integration test
  • Because a pipeline can be resumed, your library’s classes should implement Serializable class and the following attribute has to be provided:
private static final long serialVersionUID = 1L

4.1.5 - Annotated Jenkinsfiles - Part 1

Detailed Jenkinsfile examples with annotations

Pipeline example

1. Simple one

This build is used to generate docker images used to build production code and launch phpunit tests. This pipeline is parameterized in the Jenkins UI directly with the parameters:

  • branch (git branch to use)
  • environment(select with 3 options: build, phpunit or all)
    • it would have been better to use simply 2 checkboxes phpunit/build
  • project_branch

Here the source code with inline comments:

Annotated jenkinsfile Expand source

// This method allows to convert the branch name to a docker image tag.
// This method is generally used by most of my jenkins pipelines, it's why it has been added to https://github.com/fchastanet/jenkins-library/blob/master/src/fchastanet/Docker.groovy#L31
def getTagCompatibleFromBranch(String branchName) {
    def String tag = branchName.toLowerCase()
    tag = tag.replaceAll("^origin/", "")
    return tag.replaceAll('/', '_')
}

// we declare here some variables that will be used in next stages
def String deploymentBranchTagCompatible = ''

pipeline {
    agent {
        node {
            // the pipeline is executed on a machine with docker daemon
            // available
            label 'docker-ubuntu'
        }
    }

    stages {
        stage ('checkout') {
            steps {
                // this command is actually not necessary because checkout is
                // done automatically when using declarative pipeline
                sh 'echo "pulling ... ${GIT_BRANCH#origin/}"'
                checkout scm

                // this particular build needs to access to some private github
                // repositories, so here we are copying the ssh key
                // it would be better to use new way of injecting ssh key
                // inside docker using sshagent
                // check https://stackoverflow.com/a/66897280
                withCredentials([
                    sshUserPrivateKey(
                      credentialsId: '855aad9f-1b1b-494c-aa7f-4de881c7f659',
                      keyFileVariable: 'sshKeyFile'
                   )
                ]) {
                    // best practice similar steps should be merged into one
                    sh 'rm -f ./phpunit/id_rsa'
                    sh 'rm -f ./build/id_rsa'
                    // here we are escaping '$' so the variable will be
                    // interpolated on the jenkins slave and not the jenkins
                    // master node instead of escaping, we could have used
                    // single quotes
                    sh "cp \$sshKeyFile ./phpunit/id_rsa"
                    sh "cp \$sshKeyFile ./build/id_rsa"
                }
                script {
                    // as actually scm is already done before executing the
                    // first step, this call could have been done during
                    // declaration of this variable
                    deploymentBranchTagCompatible = getTagCompatibleFromBranch(GIT_BRANCH)
                }
            }
        }
        stage("build Build env") {
            when {
                // the build can be launched with the parameter environment
                // defined in the configuration of the jenkins job, these
                // parameters could have been defined directly in the pipeline
                // see https://www.jenkins.io/doc/book/pipeline/syntax/#parameters
                expression { return params.environment != "phpunit"}
            }
            steps {
                // here we could have launched all this commands in the same sh
                // directive
                sh "docker build --build-arg BRANCH=${params.project_branch} -t build build"
                // use a constant for dockerRegistryId.dkr.ecr.eu-west-1.amazonaws.com
                sh "docker tag build dockerRegistryId.dkr.ecr.eu-west-1.amazonaws.com/build:${deploymentBranchTagCompatible}"
                sh "docker push dockerRegistryId.dkr.ecr.eu-west-1.amazonaws.com/build:${deploymentBranchTagCompatible}"
            }
        }
        stage("build PHPUnit env") {
            when {
                // it would have been cleaner to use
                // expression { return params.environment = "phpunit"}
                expression { return params.environment != "build"}
            }
            steps {
                sh "docker build --build-arg BRANCH=${params.project_branch} -t phpunit phpunit"
                sh "docker tag phpunit dockerRegistryId.dkr.ecr.eu-west-1.amazonaws.com/phpunit:${deploymentBranchTagCompatible}"
                sh "docker push dockerRegistryId.dkr.ecr.eu-west-1.amazonaws.com/phpunit:${deploymentBranchTagCompatible}"
            }
        }
    }
}

without seeing the Dockerfile files, we can advise :

  • to build these images in the same pipeline where build and phpunit are run
    • the images are built at the same time so we are sure that we are using the right version
  • apparently the docker build depend on the branch of the project, this should be avoided
  • ssh key is used in docker image, that could lead to a security issue as ssh key is still in the history of images layers even if it has been removed in subsequent layers, check https://stackoverflow.com/a/66897280 for information on how to use ssh-agent instead
  • we could use a single Dockerfile with 2 stages:
    • one stage to generate production image
    • one stage that inherits production stage, used to execute phpunit
    • it has the following advantages :
      • reduce the total image size because of the reuse different docker image layers
      • only one Dockerfile to maintain

2. More advanced and annotated Jenkinsfiles

4.1.6 - Annotated Jenkinsfiles - Part 2

More annotated Jenkinsfile examples

1. Introduction

This example is missing the use of parameters, jenkins library in order to reuse common code

This example uses :

  • post conditions https://www.jenkins.io/doc/book/pipeline/syntax/#post
  • github plugin to set commit status indicating the result of the build
  • usage of several jenkins plugins, you can check here to get the full list installed on your server and even generate code snippets by adding pipeline-syntax/ to your jenkins server url

But it misses:

check Pipeline syntax documentation

2. Annotated Jenkinsfile

// Define variables for QA environment
def String registry_id = 'awsAccountId'
def String registry_url = registry_id + '.dkr.ecr.us-east-1.amazonaws.com'
def String image_name = 'project'
def String image_fqdn_master = registry_url + '/' + image_name + ':master'
def String image_fqdn_current_branch = image_fqdn_master

// this method is used by several of my pipelines and has been added
// to jenkins_library <https://github.com/fchastanet/jenkins-library/blob/master/src/fchastanet/Git.groovy#L156>
void publishStatusToGithub(String status) {
  step([
    $class: "GitHubCommitStatusSetter",
    reposSource: [$class: "ManuallyEnteredRepositorySource", url: "https://github.com/fchastanet/project"],
    errorHandlers: [[$class: 'ShallowAnyErrorHandler']],
    statusResultSource: [
      $class: 'ConditionalStatusResultSource',
      results: [
        [$class: 'AnyBuildResult', state: status]
      ]
    ]
  ]);
}

pipeline {
  agent {
    node {
      // bad practice: try to indicate in your node labels, which feature it
      // includes for example, here we need docker, label could have been
      // 'eks-nonprod-docker'
      label 'eks-nonprod'
    }
  }
  stages {
    stage ('Checkout') {
      steps {
        // checkout is not necessary as it is automatically done
        checkout scm

        script {
          // 'wrap' allows to inject some useful variables like BUILD_USER,
          // BUILD_USER_FIRST_NAME
          // see https://www.jenkins.io/doc/pipeline/steps/build-user-vars-plugin/
          wrap([$class: 'BuildUser']) {
            def String displayName = "#${currentBuild.number}_${BRANCH}_${BUILD_USER}_${DEPLOYMENT}"

            // params could have been defined inside the pipeline directly
            // instead of defining them in jenkins build configuration
            if (params.DEPLOYMENT == 'staging') {
              displayName = "${displayName}_${INSTANCE}"
            }
            // next line allows to change the build name, check addHtmlBadge
            // plugin function for more advanced usage of this feature, you
            // check this jenkinsfile 05-02-Annotated-Jenkinsfiles.md
            currentBuild.displayName = displayName
          }
        }
      }
    }
    stage ('Run tests') {
      steps {
        // all these sh directives could have been merged into one
        // it is best to use a separated sh file that could take some parameters
        // as it is simpler to read and to eventually test separately
        sh 'docker build -t project-test "$PWD"/docker/test'
        sh 'cp "$PWD"/app/config/parameters.yml.dist "$PWD"/app/config/parameters.yml'
        // for better readability and if separated script is not possible, use
        // continuation line for better readability
        sh 'docker run -i --rm -v "$PWD":/var/www/html/ -w /var/www/html/ project-test  /bin/bash -c "composer install -a && ./bin/phpunit -c /var/www/html/app/phpunit.xml --coverage-html /var/www/html/var/logs/coverage/ --log-junit /var/www/html/var/logs/phpunit.xml  --coverage-clover /var/www/html/var/logs/clover_coverage.xml"'
      }
      // Run the steps in the post section regardless of the completion status
      // of the Pipeline’s or stage’s run.
      // see https://www.jenkins.io/doc/book/pipeline/syntax/#post
      post {
        always {
          // report unit test reports (unit test should generate result using
          // using junit format)
          junit 'var/logs/phpunit.xml'
          // generate coverage page from test results
          step([
            $class: 'CloverPublisher',
            cloverReportDir: 'var/logs/',
            cloverReportFileName: 'clover_coverage.xml'
          ])
          // publish html page with the result of the coverage
          publishHTML(
            target: [
              allowMissing: false,
              alwaysLinkToLastBuild: false,
              keepAll: true,
              reportDir: 'var/logs/coverage/',
              reportFiles: 'index.html',
              reportName: "Coverage Report"
            ]
          )
        }
      }
    }
    // this stage will be executed only if previous stage is successful
    stage('Build image') {
      when {
        // this stage is executed only if these conditions returns true
        expression {
          return
            params.DEPLOYMENT == "staging"
            || (
              params.DEPLOYMENT == "prod"
              && env.GIT_BRANCH == 'origin/master'
            )
        }
      }
      steps {
        script {
          // this code is used in most of the pipeline and has been centralized
          // in https://github.com/fchastanet/jenkins-library/blob/master/src/fchastanet/Git.groovy#L39
          env.IMAGE_TAG = env.GIT_COMMIT.substring(0, 7)
          // Update variable for production environment
          if ( params.DEPLOYMENT == 'prod' ) {
              registry_id = 'awsDockerRegistryId'
              registry_url = registry_id + '.dkr.ecr.eu-central-1.amazonaws.com'
              image_fqdn_master = registry_url + '/' + image_name + ':master'
          }

          image_fqdn_current_branch = registry_url + '/' + image_name + ':' + env.IMAGE_TAG
        }

        // As jenkins slave machine can be constructed on demand,
        // it doesn't always contains all docker image cache
        // here to avoid building docker image from scratch, we are trying to
        // pull an existing version of the docker image on docker registry
        // and then build using this image as cache, so all layers not updated
        // in Dockerfile will not be built again (gain of time)
        // It is again a recurrent usage in most of the pipelines
        // so the next 8 lines could be replaced by the call to this method
        // Docker
        // pullBuildPushImage https://github.com/fchastanet/jenkins-library/blob/master/src/fchastanet/Docker.groovy#L46

        // Pull the master from repository (|| true avoids errors if the image
        // hasn't been pushed before)
        sh "docker pull ${image_fqdn_master} || true"

        // Build the image using pulled image as cache
        // instead of using concatenation, it is more readable to use variable interpolation
        // Eg: "docker build --cache-from ${image_fqdn_master} -t ..."
        sh 'docker build \
            --cache-from ' + image_fqdn_master + ' \
            -t ' + image_name + ' \
            -f "$PWD/docker/prod/Dockerfile" \
            .'
      }
    }
    stage('Deploy image (Staging)') {
      when {
          expression { return params.DEPLOYMENT == "staging" }
      }

      steps {
        script {
          // Actually we should always push the image in order to be able to
          // feed the docker cache for next builds
          // Again the method Docker pullBuildPushImage https://github.com/fchastanet/jenkins-library/blob/master/src/fchastanet/Docker.groovy#L46
          // solves this issue and could be used instead of the next 6 lines
          // and "Push image (Prod)" stage

          // If building master, we should push the image with the tag master
          // to benefit from docker cache
          if ( env.GIT_BRANCH == 'origin/master' ) {
              sh label:"Tag the image as master",
                 script:"docker tag ${image_name} ${image_fqdn_master}"
              sh label:"Push the image as master",
                 script:"docker push ${image_fqdn_master}"
          }
        }

        sh label:"Tag the image", script:"docker tag ${image_name} ${image_fqdn_current_branch}"
        sh label:"Push the image", script:"docker push ${image_fqdn_current_branch}"
        // use variable interpolation instead of concatenation
        sh label:"Deploy on cluster", script:" \
          helm3 upgrade project-" + params.INSTANCE + " -i \
            --namespace project-" + params.INSTANCE + " \
            --create-namespace \
            --cleanup-on-fail \
            --atomic \
            -f helm/values_files/values-" + params.INSTANCE + ".yaml \
            --set deployment.php_container.image.pullPolicy=Always \
            --set image.tag=" + env.IMAGE_TAG + " \
            ./helm"
      }
    }
    stage('Push image (Prod)') {
      when {
        expression { return params.DEPLOYMENT == "prod" && env.GIT_BRANCH == 'origin/master'}
      }
      // The method Docker pullBuildPushImage https://github.com/fchastanet/jenkins-library/blob/master/src/fchastanet/Docker.groovy#L46
      // provides a generic way of managing the pull, build, push of the docker
      // images, by managing also a common way of tagging docker images
      steps {
        sh label:"Tag the image as master", script:"docker tag ${image_name} ${image_fqdn_current_branch}"
        sh label:"Push the image as master", script:"docker push ${image_fqdn_current_branch}"
      }
    }
  }
  post {
    always {
      // mark github commit as built
      publishStatusToGithub("${currentBuild.currentResult}")
    }
  }
}

This directive is really difficult to read and eventually debug it

sh 'docker run -i --rm -v "$PWD":/var/www/html/ -w /var/www/html/ project-test  /bin/bash -c "composer install -a && ./bin/phpunit -c /var/www/html/app/phpunit.xml --coverage-html /var/www/html/var/logs/coverage/ --log-junit /var/www/html/var/logs/phpunit.xml  --coverage-clover /var/www/html/var/logs/clover_coverage.xml"'

Another way to write previous directive is to:

  • use continuation line
  • avoid ‘&&’ as it can mask errors, use ‘;’ instead
  • use ‘set -o errexit’ to fail on first error
  • use ‘set -o pipefail’ to fail if eventual piped command is failing
  • ‘set -x’ allows to trace every command executed for better debugging

Here a possible refactoring:

sh ''''
  docker run -i --rm \
    -v "$PWD":/var/www/html/ \
    -w /var/www/html/ \
    project-test \
    /bin/bash -c "\
      set -x ;\
      set -o errexit ;\
      set -o pipefail ;\
      composer install -a ;\
      ./bin/phpunit \
        -c /var/www/html/app/phpunit.xml \
        --coverage-html /var/www/html/var/logs/coverage/ \
        --log-junit /var/www/html/var/logs/phpunit.xml  \
        --coverage-clover /var/www/html/var/logs/clover_coverage.xml
    "
'''

Note however it is best to use a separated sh file(s) that could take some parameters as it is simpler to read and to eventually test separately. Here a refactoring using a separated sh file:

runTests.sh

#!/bin/bash
set -x -o errexit -o pipefail

composer install -a

./bin/phpunit \
  -c /var/www/html/app/phpunit.xml \
  --coverage-html /var/www/html/var/logs/coverage/ \
  --log-junit /var/www/html/var/logs/phpunit.xml \
  --coverage-clover /var/www/html/var/logs/clover_coverage.xml

jenkinsRunTests.sh

#!/bin/bash
set -x -o errexit -o pipefail

docker build -t project-test "${PWD}/docker/test"

docker run -i --rm \
  -v "${PWD}:/var/www/html/" \
  -w /var/www/html/ \
  project-test \
  runTests.sh

Then the sh directive becomes simply

sh 'jenkinsRunTests.sh'

4.1.7 - Annotated Jenkinsfiles - Part 3

Additional Jenkinsfile pattern examples

1. Introduction

This build will:

  • pull/build/push docker image used to generate project files
  • lint
  • run Unit tests with coverage
  • build the SPA
  • run accessibility tests
  • build story book and deploy it
  • deploy spa on s3 bucket and refresh cloudflare cache

It allows to build for production and qa stages allowing different instances. Every build contains:

  • a summary of the build
    • git branch
    • git revision
    • target environment
  • all the available Urls:
    • spa url
    • storybook url

2. Annotated Jenkinsfile

// anonymized parameters
String credentialsId = 'jenkinsCredentialId'
def lib = library(
  identifier: '[email protected]',
  retriever: modernSCM([
    $class: 'GitSCMSource',
    remote: '[email protected]:fchastanet/jenkins-library.git',
    credentialsId: credentialsId
  ])
)
def docker = lib.fchastanet.Docker.new(this)
def git = lib.fchastanet.Git.new(this)
def mail = lib.fchastanet.Mail.new(this)
def utils = lib.fchastanet.Utils.new(this)
def cloudflare = lib.fchastanet.Cloudflare.new(this)

// anonymized parameters
String CLOUDFLARE_ZONE_ID = 'cloudflareZoneId'
String CLOUDFLARE_ZONE_ID_PROD = 'cloudflareZoneIdProd'
String REGISTRY_ID_QA  = 'dockerRegistryId'
String REACT_APP_PENDO_API_KEY = 'pendoApiKey'

String REGISTRY_QA  = REGISTRY_ID_QA + '.dkr.ecr.us-east-1.amazonaws.com'
String IMAGE_NAME_SPA = 'project-ui'
String STAGING_API_URL = 'https://api.host'
String INSTANCE_URL = "https://${params.instanceName}.host"
String REACT_APP_API_BASE_URL_PROD = 'https://ui.host'
String REACT_APP_PENDO_SOURCE_DOMAIN = 'https://cdn.eu.pendo.io'

String buildBucketPrefix
String S3_PUBLIC_URL = 'qa-spa.s3.amazonaws.com/project'
String S3_PROD_PUBLIC_URL = 'spa.s3.amazonaws.com/project'

List<String> instanceChoices = (1..20).collect { 'project' + it }

Map buildInfo = [
  apiUrl: '',
  storyBookAvailable: false,
  storyBookUrl: '',
  storyBookDocsUrl: '',
  spaAvailable: false,
  spaUrl: '',
  instanceName: '',
]

// add information on summary page
def addBuildInfo(buildInfo) {
  String deployInfo = ''
  if (buildInfo.spaAvailable) {
    String formatInstanceName = buildInfo.instanceName ?
      " (${buildInfo.instanceName})" : '';
    deployInfo += "<a href='${buildInfo.spaUrl}'>SPA${formatInstanceName}</a>"
  }
  if (buildInfo.storyBookAvailable) {
    deployInfo += " / <a href='${buildInfo.storyBookUrl}'>Storybook</a>"
    deployInfo += " / <a href='${buildInfo.storyBookDocsUrl}'>Storybook docs</a>"
  }
  String summaryHtml = """
    <b>branch : </b>${GIT_BRANCH}<br/>
    <b>revision : </b>${GIT_COMMIT}<br/>
    <b>target env : </b>${params.targetEnv}<br/>
    ${deployInfo}
  """
  removeHtmlBadges id: "htmlBadge${currentBuild.number}"
  addHtmlBadge html: summaryHtml, id: "htmlBadge${currentBuild.number}"
}

pipeline {
  agent {
    node {
      // this image has the features docker and lighthouse
      label 'docker-base-ubuntu-lighthouse'
    }
  }

  parameters {
    gitParameter(
      branchFilter: 'origin/(.*)',
      defaultValue: 'main',
      quickFilterEnabled: true,
      sortMode: 'ASCENDING_SMART',
      name: 'BRANCH',
      type: 'PT_BRANCH'
    )
    choice(
      name: 'targetEnv',
      choices: ['none', 'testing', 'production'],
      description: 'Where it should be deployed to? (Default: none - No deploy)'
    )
    booleanParam(
      name: 'buildStorybook',
      defaultValue: false,
      description: 'Build Storybook (will only apply if selected targetEnv is testing)'
    )
    choice(
      name: 'instanceName',
      choices: instanceChoices,
      description: 'Instance name to deploy the revision'
    )
  }

  stages {
    stage('Build SPA image') {
      steps {
        script {
          // set build status to pending on github commit
          step([$class: 'GitHubSetCommitStatusBuilder'])
          wrap([$class: 'BuildUser']) {
            currentBuild.displayName = "#${currentBuild.number}_${BRANCH}_${BUILD_USER}_${targetEnv}"
          }

          branchName = docker.getTagCompatibleFromBranch(env.GIT_BRANCH)
          shortSha = git.getShortCommitSha(env.GIT_BRANCH)

          if (params.targetEnv == 'production') {
            buildBucketPrefix = GIT_COMMIT
            buildInfo.apiUrl = REACT_APP_API_BASE_URL_PROD
            s3BaseUrl = 's3://project-spa/project'
          } else {
            buildBucketPrefix = params.instanceName
            buildInfo.instanceName = params.instanceName
            buildInfo.spaUrl = "${INSTANCE_URL}/index.html"
            buildInfo.apiUrl = STAGING_API_URL
            s3BaseUrl = 's3://project-qa-spa/project'
            buildInfo.storyBookUrl = "${INSTANCE_URL}/storybook/index.html"
            buildInfo.storyBookDocsUrl = "${INSTANCE_URL}/storybook-docs/index.html"
          }
          addBuildInfo(buildInfo)

          // Setup .env
          sh """
            set -x
            echo "REACT_APP_API_BASE_URL = '${buildInfo.apiUrl}'" > ./.env
            echo "REACT_APP_PENDO_SOURCE_DOMAIN = '${REACT_APP_PENDO_SOURCE_DOMAIN}'" >> ./.env
            echo "REACT_APP_PENDO_API_KEY = '${REACT_APP_PENDO_API_KEY}'" >> ./.env
          """

          withCredentials([
            sshUserPrivateKey(
              credentialsId: 'sshCredentialsId',
              keyFileVariable: 'sshKeyFile')
          ]) {
            docker.pullBuildPushImage(
              buildDirectory:   pwd(),
              // use safer way to inject ssh key during docker build
              buildArgs: "--ssh default=\$sshKeyFile --build-arg USER_ID=\$(id -u)",
              registryImageUrl: "${REGISTRY_QA}/${IMAGE_NAME_SPA}",
              tagPrefix:        "${IMAGE_NAME_SPA}:",
              localTagName:     "latest",
              tags: [
                shortSha,
                branchName
              ],
              pullTags: ['main']
            )
          }
        }
      }
    }

    stage('Linting') {
      steps {
        sh """
          docker run --rm \
            -v ${env.WORKSPACE}:/app \
            -v /app/node_modules \
            ${IMAGE_NAME_SPA} \
            npm run lint
        """
      }
    }

    stage('UT') {
      steps {
        script {
          sh """docker run --rm  \
            -v ${env.WORKSPACE}:/app \
            -v /app/node_modules \
            ${IMAGE_NAME_SPA} \
            npm run test:coverage -- --ci
          """

          junit 'output/junit.xml'

          // https://plugins.jenkins.io/clover/
          step([
            $class: 'CloverPublisher',
            cloverReportDir: 'output/coverage',
            cloverReportFileName: 'clover.xml',
            healthyTarget: [
              methodCoverage: 70,
              conditionalCoverage: 70,
              statementCoverage: 70
            ],
            // build will not fail but be set as unhealthy if coverage goes
            // below 60%
            unhealthyTarget: [
              methodCoverage: 60,
              conditionalCoverage: 60,
              statementCoverage: 60
            ],
            // build will fail if coverage goes below 50%
            failingTarget: [
              methodCoverage: 50,
              conditionalCoverage: 50,
              statementCoverage: 50
            ]
          ])
        }
      }
    }

    stage('Build SPA') {
      steps {
        script {
          sh """
            docker run --rm \
              -v ${env.WORKSPACE}:/app \
              -v /app/node_modules \
              ${IMAGE_NAME_SPA}
          """
        }
      }
    }

    stage('Accessibility tests') {
      steps {
        script {
          // the pa11y-ci could have been made available in the node image
          // to avoid installation each time, the build is launched
          sh '''
            sudo npm install -g serve pa11y-ci
            serve -s build > /dev/null 2>&1 &
            pa11y-ci --threshold 5 http://127.0.0.1:3000
          '''
        }
      }
    }

    stage('Build Storybook') {
      steps {
        whenOrSkip(
          params.targetEnv == 'testing'
          && params.buildStorybook == true
        ) {
          script {
            sh """
              docker run --rm \
                -v ${env.WORKSPACE}:/app \
                -v /app/node_modules \
                ${IMAGE_NAME_SPA} \
                sh -c 'npm run storybook:build -- --output-dir build/storybook \
                  && npm run storybook:build-docs -- --output-dir build/storybook-docs'
            """
            buildInfo.storyBookAvailable = true
          }
        }
      }
    }

    stage('Artifacts to S3') {
      steps {
        whenOrSkip(params.targetEnv != 'none') {
          script {
            if (params.targetEnv == 'production') {
              utils.initAws('arn:aws:iam::awsIamId:role/JenkinsSlave')
            }

            sh "aws s3 cp ${env.WORKSPACE}/build ${s3BaseUrl}/${buildBucketPrefix} --recursive --no-progress"
            sh "aws s3 cp ${env.WORKSPACE}/build ${s3BaseUrl}/project1 --recursive --no-progress"

            if (params.targetEnv == 'production') {
              echo 'project SPA packages have been pushed to production bucket.'
              echo '''You can refresh the production indexes with the CD
              production pipeline.'''
              cloudflare.zonePurge(CLOUDFLARE_ZONE_ID_PROD, [prefixes:[
                "${S3_PROD_PUBLIC_URL}/project1/"
              ]])
            } else {
              cloudflare.zonePurge(CLOUDFLARE_ZONE_ID, [prefixes:[
                "${S3_PUBLIC_URL}/${buildBucketPrefix}/"
              ]])

              buildInfo.spaAvailable = true
              publishChecks detailsURL: buildInfo.spaUrl,
                name: 'projectSpaUrl',
                title: 'project SPA url'
            }
            addBuildInfo(buildInfo)
          }
        }
      }
    }
  }

  post {
    always {
      script {
        git.updateConditionalGithubCommitStatus()
        mail.sendConditionalEmail()
      }
    }
  }
}

4.1.8 - Annotated Jenkinsfiles - Part 4

Complex Jenkinsfile scenarios

1. introduction

The project aim is to create a browser extension available on chrome and firefox

This build allows to:

  • lint the project using megalinter and phpstorm inspection
  • build necessary docker images
  • build firefox and chrome extensions
  • deploy firefox extension on s3 bucket
  • deploy chrome extension on google play store

2. Annotated Jenkinsfile

def credentialsId = 'jenkinsSshCredentialsId'
def lib = library(
    identifier: 'jenkins_library',
    retriever: modernSCM([
        $class: 'GitSCMSource',
        remote: '[email protected]:fchastanet/jenkins-library.git',
        credentialsId: credentialsId
    ])
)
def docker = lib.fchastanet.Docker.new(this)
def git = lib.fchastanet.Git.new(this)
def mail = lib.fchastanet.Mail.new(this)

def String deploymentBranchTagCompatible = ''
def String gitShortSha = ''
def String REGISTRY_URL = 'dockerRegistryId.dkr.ecr.eu-west-1.amazonaws.com'
def String ECR_BROWSER_EXTENSION_BUILD = 'browser_extension_lint'
def String BUILD_TAG = 'build'
def String PHPSTORM_TAG = 'phpstorm-inspections'
def String REFERENCE_JOB_NAME = 'Browser_extension_deploy'
def String FIREFOX_S3_BUCKET = 'browser-extensions'

// it would have been easier to use checkboxes to avoid 'both'/'none'
// complexity
def DEPLOY_CHROME = (params.targetStore == 'both' || params.targetStore == 'chrome')
def DEPLOY_FIREFOX = (params.targetStore == 'both' || params.targetStore == 'firefox')

pipeline {
  agent {
    node {
      label 'docker-base-ubuntu'
    }
  }
  parameters {
    gitParameter branchFilter: 'origin/(.*)',
      defaultValue: 'master',
      quickFilterEnabled: true,
      sortMode: 'ASCENDING_SMART',
      name: 'BRANCH',
      type: 'PT_BRANCH'

    choice (
      name: 'targetStore',
      choices: ['none', 'both', 'chrome', 'firefox'],
      description: 'Where it should be deployed to? (Default: none, has effect only on master branch)'
    )
  }
  environment {
    GOOGLE_CREDS = credentials('GoogleApiChromeExtension')
    GOOGLE_TOKEN = credentials('GoogleApiChromeExtensionCode')
    GOOGLE_APP_ID = 'googleAppId'
    // provided by https://addons.mozilla.org/en-US/developers/addon/api/key/
    FIREFOX_CREDS = credentials('MozillaApiFirefoxExtension')
    FIREFOX_APP_ID='{d4ce8a6f-675a-4f74-b2ea-7df130157ff4}'
  }

  stages {

    stage("Init") {
      steps {
        script {
          deploymentBranchTagCompatible = docker.getTagCompatibleFromBranch(env.GIT_BRANCH)
          gitShortSha = git.getShortCommitSha(env.GIT_BRANCH)
          echo "Branch ${env.GIT_BRANCH}"
          echo "Docker tag = ${deploymentBranchTagCompatible}"
          echo "git short sha = ${gitShortSha}"
        }
        sh 'echo StrictHostKeyChecking=no >> ~/.ssh/config'
      }
    }

    stage("Lint") {
      agent {
        docker {
          image 'megalinter/megalinter-javascript:v5'
          args "-u root -v ${WORKSPACE}:/tmp/lint --entrypoint=''"
          reuseNode true
        }
      }
      steps {
        sh 'npm install stylelint-config-rational-order'
        sh '/entrypoint.sh'
      }
    }

    stage("Build docker images") {
      steps {
        // whenOrSkip directive is defined in https://github.com/fchastanet/jenkins-library/blob/master/vars/whenOrSkip.groovy
        whenOrSkip(currentBuild.currentResult == "SUCCESS") {
          script {
            docker.pullBuildPushImage(
              buildDirectory:   'build',
              registryImageUrl: "${REGISTRY_URL}/${ECR_BROWSER_EXTENSION_BUILD}",
              tagPrefix:        "${ECR_BROWSER_EXTENSION_BUILD}:",
              tags: [
                "${BUILD_TAG}_${gitShortSha}",
                "${BUILD_TAG}_${deploymentBranchTagCompatible}",
              ],
              pullTags: ["${BUILD_TAG}_master"]
            )
          }
        }
      }
    }

    stage("Build firefox/chrome extensions") {
      steps {
        whenOrSkip(currentBuild.currentResult == "SUCCESS") {
          script {
              sh """
                docker run \
                  -v \$(pwd):/deploy \
                  --rm '${ECR_BROWSER_EXTENSION_BUILD}' \
                  /deploy/build/build-extensions.sh
              """
              // multiple git statuses can be set on a given commit
              // you can configure github to authorize pull request merge
              // based on the presence of one or more github statuses
              git.updateGithubCommitStatus("BUILD_OK")
          }
        }
      }
    }

    stage("Deploy extensions") {
      // deploy both extensions in parallel
      parallel {
        stage("Deploy chrome") {
          steps {
            whenOrSkip(currentBuild.currentResult == "SUCCESS" && DEPLOY_CHROME) {
              // do not fail the entire build if this stage fail
              // so firefox stage can be executed
              catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {
                script {
                  // best practice: complex sh files have been created outside
                  // of this jenkinsfile deploy-chrome-extension.sh
                  sh """
                  docker run \
                      -v \$(pwd):/deploy \
                      -e APP_CREDS_USR='${GOOGLE_CREDS_USR}' \
                      -e APP_CREDS_PSW='${GOOGLE_CREDS_PSW}' \
                      -e APP_TOKEN='${GOOGLE_APP_TOKEN}' \
                      -e APP_ID='${GOOGLE_APP_ID}' \
                      --rm '${ECR_BROWSER_EXTENSION_BUILD}' \
                      /deploy/build/deploy-chrome-extension.sh
                  """
                  git.updateGithubCommitStatus("CHROME_DEPLOYED")
                }
              }
            }
          }
        }
        stage("Deploy firefox") {
          steps {
            whenOrSkip(currentBuild.currentResult == "SUCCESS" && DEPLOY_FIREFOX) {
              catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {
                script {
                  // best practice: complex sh files have been created outside
                  // of this jenkinsfile deploy-firefox-extension.sh
                  sh """
                    docker run \
                      -v \$(pwd):/deploy \
                      -e FIREFOX_JWT_ISSUER='${FIREFOX_CREDS_USR}' \
                      -e FIREFOX_JWT_SECRET='${FIREFOX_CREDS_PSW}' \
                      -e FIREFOX_APP_ID='${FIREFOX_APP_ID}' \
                      --rm '${ECR_BROWSER_EXTENSION_BUILD}' \
                      /deploy/build/deploy-firefox-extension.sh
                  """
                  sh """
                    set -x
                    set -o errexit
                    extensionVersion="\$(jq -r .version < package.json)"
                    extensionFilename="tools-\${extensionVersion}-an+fx.xpi"

                    echo "Upload new extension \${extensionFilename} to s3 bucket ${FIREFOX_S3_BUCKET}"
                    aws s3 cp "\$(pwd)/packages/\${extensionFilename}" "s3://${FIREFOX_S3_BUCKET}"
                    aws s3api put-object-acl --bucket "${FIREFOX_S3_BUCKET}" --key "\${extensionFilename}" --acl public-read
                    # url is https://tools.s3.eu-west-1.amazonaws.com/tools-2.5.6-an%2Bfx.xpi

                    echo "Upload new version as current version"
                    aws s3 cp "\$(pwd)/packages/\${extensionFilename}" "s3://${FIREFOX_S3_BUCKET}/tools-an+fx.xpi"
                    aws s3api put-object-acl --bucket "${FIREFOX_S3_BUCKET}" --key "tools-an+fx.xpi" --acl public-read
                    # url is https://tools.s3.eu-west-1.amazonaws.com/tools-an%2Bfx.xpi

                    echo "Upload updates.json file"
                    aws s3 cp "\$(pwd)/packages/updates.json" "s3://${FIREFOX_S3_BUCKET}"
                    aws s3api put-object-acl --bucket "${FIREFOX_S3_BUCKET}" --key "updates.json" --acl public-read
                    # url is https://tools.s3.eu-west-1.amazonaws.com/updates.json
                  """
                  git.updateGithubCommitStatus("FIREFOX_DEPLOYED")
                }
              }
            }
          }
        }
      }
    }
  }
  post {
    always {
      script {
        archiveArtifacts artifacts: 'report/mega-linter.log'
        archiveArtifacts artifacts: 'report/linters_logs/*'
        archiveArtifacts artifacts: 'packages/*', fingerprint: true, allowEmptyArchive: true
        // send email to the builder and culprits of the current commit
        // culprits are the committers since the last commit successfully built
        mail.sendConditionalEmail()
        git.updateConditionalGithubCommitStatus()
      }
    }
    success {
      script {
        if (params.targetStore != 'none' && env.GIT_BRANCH == 'origin/master') {
          // send an email to a teams channel so every collaborators knows
          // when a production ready extension has been deployed
          mail.sendSuccessfulEmail('[email protected]')
        }
      }
    }
  }
}

4.1.9 - Annotated Jenkinsfiles - Part 5

Detailed Jenkinsfile examples with annotations

1. introduction

In jenkins library you can create your own directive that allows to generate jenkinsfile code. Here we will use this feature to generate a complete Jenkinsfile.

2. Annotated Jenkinsfile

library identifier: '[email protected]',
  retriever: modernSCM([
      $class: 'GitSCMSource',
      remote: '[email protected]:fchastanet/jenkins-library.git',
      credentialsId: 'jenkinsCredentialsId'
  ])

djangoApiPipeline repoUrl: '[email protected]:fchastanet/django_api_project.git',
                  imageName: 'django_api'

3. Annotated library custom directive

In the jenkins library just add a file named vars/djangoApiPipeline.groovy with the following content

#!/usr/bin/env groovy

def call(Map args) {
  // content of your pipeline
}

4. Annotated library custom directive djangoApiPipeline.groovy

#!/usr/bin/env groovy

def call(Map args) {

  def gitUtil = new Git(this)
  def mailUtil = new Mail(this)
  def dockerUtil = new Docker(this)
  def kubernetesUtil = new Kubernetes(this)
  def testUtil = new Tests(this)

  String workerLabelNonProd = args?.workerLabelNonProd ?: 'eks-nonprod'
  String workerLabelProd = args?.workerLabelProd ?: 'docker-ubuntu-prod-eks'
  String awsRegionNonProd = workerLabelNonProd == 'eks-nonprod' ? 'us-east-1' : 'eu-west-1'
  String awsRegionProd = 'eu-central-1'
  String regionName = params.targetEnv == 'prod' ? awsRegionProd : awsRegionNonProd
  String teamsEmail = args?.teamsEmail ?: '[email protected]'
  String helmDirectory = args?.helmDirectory ?: './helm'
  Boolean sendCortexMetrics = args?.sendCortexMetrics ?: false
  Boolean skipTests = args?.skipTests ?: false
  List environments = args?.environments ?: ['none', 'qa', 'prod']
  Short skipBuild = 0

  pipeline {
    agent {
      node {
        label params.targetEnv == 'prod' ? workerLabelProd : workerLabelNonProd
      }
    }

    parameters {
      gitParameter branchFilter: 'origin/(.*)',
                    defaultValue: 'main',
                    quickFilterEnabled: true,
                    sortMode: 'ASCENDING_SMART',
                    name: 'BRANCH',
                    type: 'PT_BRANCH'

      choice (
        name: 'targetEnv',
        choices: environments,
        description: 'Where it should be deployed to? (Default: none - No deploy)'
      )

      string (
        name: 'instance',
        defaultValue: '1',
        description: '''The instance ID to define which QA instance it should
        be deployed to (Will only apply if targetEnv is qa).'''
      )

      booleanParam(
        name: 'suspendCron',
        defaultValue: true,
        description: 'Suspend cron jobs scheduling'
      )

      choice (
        name: 'upStreamImage',
        choices: ['latest', 'beta'],
        description: '''Select beta to check if your build works with the
        future version of the upstream image'''
      )
    }

    stages {
      stage('Checkout from SCM') {
        steps {
          script {
            echo "Checking out from origin/${BRANCH} branch"
            gitUtil.branchCheckout(
              '',
              'babee6c1-14fe-4d90-9da0-ffa7068c69af',
              args.repoUrl,
              '${BRANCH}'
            )
            wrap([$class: 'BuildUser']) {
              def String displayName = "#${currentBuild.number}_${BRANCH}_${BUILD_USER}_${targetEnv}"

              if (params.targetEnv == 'qa' || params.targetEnv == 'qe') {
                displayName = "${displayName}_${instance}"
              }

              currentBuild.displayName = displayName
            }

            env.imageName = env.BUILD_TAG.toLowerCase()
            env.buildDirectory = args?.buildDirectory ?
              args.buildDirectory + "/" : ""
            env.runCoverage = args?.runCoverage
            env.shortSha = gitUtil.getShortCommitSha(env.GIT_BRANCH)
            skipBuild = dockerUtil.checkImage(args.imageName, shortSha)
          }
        }
      }

      stage('Build') {
        when {
          expression { return skipBuild != 0 }
        }
        steps {
          script {
            String registryUrl = 'dockerRegistryId.dkr.ecr.' +
              awsRegionNonProd + '.amazonaws.com'
            String buildDirectory = args?.buildDirectory ?: pwd()

            if (params.targetEnv == "prod") {
              registryUrl = 'dockerRegistryId.dkr.ecr.' + awsRegionProd + '.amazonaws.com'
            }

            dockerUtil.pullBuildImage(
              registryImageUrl: "${registryUrl}/${args.imageName}",
              pullTags: [
                "${params.targetEnv}"
              ],
              buildDirectory: "${buildDirectory}",
              buildArgs: "--build-arg UPSTREAM_VERSION=${params.upStreamImage}",
              tagPrefix: "${env.imageName}:",
              tags: [
                "${env.shortSha}"
              ]
            )
          }
        }
      }

      stage('Test') {
        when {
          expression { return skipBuild != 0 && skipTests == false }
        }
        steps {
          script {
            testUtil.execTests(args.imageName)
          }
        }
      }
      stage('Push') {
        when {
          expression { return params.targetEnv != 'none' }
        }
        steps {
          script {
            //pipeline execution starting time for CD part
            Map argsMap = [:]

            if (params.targetEnv == "prod") {
              registryUrl = 'registryIdProd.dkr.ecr.' +
                awsRegionProd + '.amazonaws.com'
            } else {
              registryUrl = 'registryIdNonProd.dkr.ecr.' +
                awsRegionNonProd + '.amazonaws.com'
            }

            argsMap = [
              registryImageUrl: "${registryUrl}/${args.imageName}",
              pullTags: [
                "${env.shortSha}",
              ],
              tagPrefix: "${registryUrl}/${args.imageName}:",
              localTagName: "${env.shortSha}",
              tags: [
                "${params.targetEnv}"
              ]
            ]

            if (skipBuild == 0) {
              dockerUtil.promoteTag(argsMap)
            } else {
              argsMap.remove("pullTags")
              argsMap.put("tagPrefix", "${env.imageName}:")
              argsMap.put("tags", ["${env.shortSha}","${params.targetEnv}"])
              dockerUtil.tagPushImage(argsMap)
            }
          }
        }
      }
      stage("Deploy to Kubernetes") {
        when {
          expression { return params.targetEnv != 'none' }
        }
        steps {
          script {
            if (params.targetEnv == 'prod') {
              // not sure it is a good practice as it forces the operator to
              // wait for build to reach this stage
              timeout(time: 300, unit: "SECONDS") {
                input(
                  message: """Do you want go ahead with ${env.shortSha}
                  image tag for prod helm deploy?""",
                  ok: 'Yes'
                )
              }
            }
            CHART_NAME = (args.imageName).contains("_") ?
              (args.imageName).replaceAll("_", "-") :
              (args.imageName)
            if (params.targetEnv == 'qa' || params.targetEnv == 'qe') {
              helmValueFilePath = "${helmDirectory}" +
                "/value_files/values-" + params.targetEnv +
                params.instance + ".yaml"
              NAMESPACE = "${CHART_NAME}-" + params.targetEnv + params.instance
            } else {
              helmValueFilePath = "${helmDirectory}" +
                "/value_files/values-" + params.targetEnv + ".yaml"
              NAMESPACE = "${CHART_NAME}-" + params.targetEnv
            }
            ingressUrl = kubernetesUtil.getIngressUrl(helmValueFilePath)
            echo "Deploying into k8s.."
            echo "Helm release: ${CHART_NAME}"
            echo "Target env: ${params.targetEnv}"
            echo "Url: ${ingressUrl}"
            echo "K8s namespace: ${NAMESPACE}"
            kubernetesUtil.deployHelmChart(
              chartName: CHART_NAME,
              nameSpace: NAMESPACE,
              imageTag: "${env.shortSha}",
              helmDirectory: "${helmDirectory}",
              helmValueFilePath: helmValueFilePath
            )
          }
        }
      }
    }
    post {
      always {
        script {
          gitUtil.updateGithubCommitStatus("${currentBuild.currentResult}", "${env.WORKSPACE}")
          mailUtil.sendConditionalEmail()
          if (params.targetEnv == 'prod') {
              mailUtil.sendTeamsNotification(teamsEmail)
          }
        }
      }
    }
  }
}

5. Final thoughts about this technique

This technique is really useful when you have a lot of similar projects reusing over and over the same pipeline. It allows:

  • code reuse
  • avoid duplicated code
  • easier maintenance

However it has the following drawbacks:

  • some projects using this generic pipeline could have specific needs
    • eg 1: not the same way to run unit tests, to overcome that issue the method testUtil.execTests is used allowing to run a specific sh file if it exists
    • eg 2: more complex way to launch docker environment
  • be careful, when you upgrade this jenkinsfile as all the projects using it will be upgraded at once
    • it could be seen as an advantage, but it is also a big risk as it could impact all the prod environment at once
    • to overcome that issue I suggest to use library versioning when using the jenkins library in your project pipeline Eg: check Annotated Jenkinsfile @v1.0 when cloning library project
  • I highly suggest to use a unit test framework of the library to avoid at most bad surprises

In conclusion, I’m still not sure it is a best practice to generate pipelines like this.

4.1.10 - Jenkins Recipes and Tips

Useful recipes and tips for Jenkins and Jenkinsfiles

1. Jenkins snippet generator

Use jenkins snippet generator by adding /pipeline-syntax/ to your jenkins pipeline. to allow you to generate jenkins pipeline code easily with inline doc. It also list the available variables.

jenkins snippet generator

2. Declarative pipeline allows you to restart a build from a given stage

restart from stage

3. Replay a pipeline

Replaying a pipeline allows you to update your jenkinsfile before replaying the pipeline, easier debugging !

replay a pipeline

4. VS code Jenkinsfile validation

Please follow this documentation enable jenkins pipeline linter in vscode

5. How to chain pipelines ?

Simply use the build directive followed by the name of the build to launch

build 'OtherBuild'

6. Viewing pipelines hierarchy

The downstream-buildview plugin allows to view the full chain of dependent builds.

Jenkins Downstream Build Pipeline Visualization

4.2 - How to Write PlantUML

Comprehensive guide to writing PlantUML diagrams, including syntax, best practices, and examples.

1. What You’ll Learn

2. Getting Started

Select a guide from the sidebar to begin.

Articles in this section

TitleDescriptionUpdated
Reusable PlantUML Components: Modular Diagram Architecture and Shared StylingLearn how to create reusable PlantUML components for modular diagram architecture and shared styling across multiple diagrams.2026-05-10 23:42:08 +0200 +0200

4.2.1 - Reusable PlantUML Components: Modular Diagram Architecture and Shared Styling

Learn how to create reusable PlantUML components for modular diagram architecture and shared styling across multiple diagrams.

Today, we will explore how to create reusable PlantUML components for modular diagram architecture and shared styling across multiple diagrams. This approach allows you to maintain consistency and reduce duplication in your PlantUML diagrams.

1. Database ERD Examples: Music Domain

The database/ directory contains a complete example of MongoDB Entity Relationship Diagrams modeling a music streaming recommendation system. These examples demonstrate:

  • Modular diagram architecture with reusable components
  • Subsection inclusion using !includesub and !startsub
  • Shared styling across multiple diagrams
  • JSON data structures within PlantUML diagrams

1.1. Entity Collections

The music database example includes the following collections:

FileDescription
music-db-Playlists_Collections_ERD.pumlPlaylist catalog and published playlist instances
music-db-Moods_Collections_ERD.pumlMusic mood taxonomy and user taste preferences
music-db-Suggestions_Collections_ERD.pumlPlaylist suggestion engine based on user moods
music-db-Logs.pumlAPI usage logs for AI-generated playlist metadata
music-db-All_collections.pumlMaster diagram combining all collections
db_theme_standard.pumlReusable theme providing consistent styling

1.2. Business Logic Flow

  1. Playlists are tagged with moods (e.g., “energetic”, “chill”, “melancholic”)
  2. Users express music preferences through user_tastes (preferred moods)
  3. The suggestion engine matches playlists to users based on mood similarity
  4. Vector embeddings enable semantic matching between user tastes and playlist moods

1.3. Complete Database Schema Overview

The following diagram shows all collections and their relationships:

Music Database - All Collections

View source on GitHub

View PlantUML source code
@startuml
!pragma layout smetana
!include content/docs/howtos/how-to-write-plantuml/database/db_theme_standard.puml
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Playlists_Collections_ERD.puml!PLAYLIST_MODEL
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Playlists_Collections_ERD.puml!PLAYLIST_MODEL_DETAILS
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Playlists_Collections_ERD.puml!PLAYLIST_INSTANCES_MODEL
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Playlists_Collections_ERD.puml!PLAYLIST_INSTANCES_MODEL_DETAILS
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Moods_Collections_ERD.puml!MODEL_USER_TASTES
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Moods_Collections_ERD.puml!MODEL_USER_TASTES_DETAILS
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Moods_Collections_ERD.puml!MODEL_MOOD
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Moods_Collections_ERD.puml!MODEL_MOOD_DETAILS
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Suggestions_Collections_ERD.puml!MODEL_PENDING_USER_SUGGESTION
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Suggestions_Collections_ERD.puml!MODEL_PENDING_USER_SUGGESTION_DETAILS
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Suggestions_Collections_ERD.puml!MODEL_USER_SUGGESTION
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Suggestions_Collections_ERD.puml!MODEL_USER_SUGGESTION_DETAILS
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Logs.puml!MODEL_MUSIC_API_LOGS
@enduml

2. Reusable Styling with db_theme_standard.puml

The db_theme_standard.puml file is a standalone, reusable theme definition that ensures visual consistency across all database diagrams.

2.1. Why Separate Styling?

Separating styling from content provides several benefits:

  • Single source of truth for visual standards
  • Easy updates - change once, apply everywhere
  • Reduced duplication - no need to copy/paste styling rules
  • Clear separation between diagram content and presentation

2.2. What It Defines

The reusable theme file defines:

!include content/docs/howtos/how-to-write-plantuml/database/db_theme_standard.puml
  • Visual styling: fonts, colors, line styles, rounded corners
  • Notation macros: PK (primary key), FK (foreign key), IDX (index)
  • Index types: UNIQUE, SPARSE, UNIQUE_SPARSE
  • Legend symbols: explaining all notation used in diagrams

Here’s what the theme styling looks like:

Database Theme Standard

View source on GitHub

View theme source code
@startuml
' ==============================================
' MongoDB Database Diagram Standard Theme
' ==============================================
' This is a REUSABLE style definition file that provides:
' - Consistent visual styling across all database diagrams
' - Standard notation for database elements (PK, FK, indexes)
' - Color-coded legends for different entity types
' - Icon definitions for database constraints
'
' USAGE: Include this file in any database ERD diagram:
'   !include path/to/db_theme_standard.puml
'
' This ensures all diagrams share the same professional appearance
' and use consistent notation conventions.
' ==============================================

skinparam {
    defaultFontName Arial
    defaultFontSize 12
    roundCorner 8
    packageStyle rectangle
    linetype ortho
    BackgroundColor #FFF
    shadowing false
    ArrowColor #555555
    ArrowThickness 2

    entity {
        Margin 20
    }
}

!define ENTITY entity
!define PK <&key><u><b>
!define FK <u><i>
!define IDX <&magnifying-glass>
!define UNIQUE <u><&magnifying-glass><<unique>>
!define SPARSE <i><&magnifying-glass><<sparse>>
!define UNIQUE_SPARSE <u><i><&magnifying-glass><<unique_sparse>>
!define DETAILS -

legend
|= |= Type |
|<back:#FF0000>   </back>| Type A class |
|<back:#00FF00>   </back>| Type B class |
|<back:blue>   </back>| Type C class |
endlegend

legend left
  |= notation |= meaning|
  | <img : https://cdn-0.plantuml.com/public-field.png{scale=1.4}>(+)  | Indexed column  |
  | <img : https://cdn-0.plantuml.com/private-field.png{scale=1.4}>(-) | details |
  | ""<&key>"" | Primary Key |
  | ""<u><i>column"" | Foreign Key |
  | ""<&magnifying-glass>"" | Standard Index |
  | UNIQUE | Unique Index |
  | SPARSE | Sparse Index |
  | UNIQUE_SPARSE | Unique & Sparse Index |
endlegend

@enduml

2.3. Usage Pattern

Every database ERD diagram includes this file:

@startuml
!include content/docs/howtos/how-to-write-plantuml/database/db_theme_standard.puml

ENTITY users {
  PK _id : ObjectId
  --
  + IDX email : string
}
@enduml

This pattern ensures all diagrams share professional, consistent notation.

3. Modular Composition with !startsub and !includesub

PlantUML supports subsection extraction, allowing you to define reusable diagram fragments that can be included selectively in other diagrams.

3.1. Defining Subsections: !startsub

Use !startsub and !endsub to mark reusable sections:

' Define a reusable playlist entity
!startsub PLAYLIST_MODEL
ENTITY playlists {
  PK _id : ObjectId
  + IDX reference_code : string
}
!endsub

' Define additional detail fields
!startsub PLAYLIST_MODEL_DETAILS
ENTITY playlists {
  # moods : ObjectId[]
  # title : string
  - created_at : datetime
}
!endsub

3.2. Including Subsections: !includesub

Use !includesub to import specific subsections into another diagram:

@startuml
!include db_theme_standard.puml

' Include only the core playlist model (without details)
!includesub music-db-Playlists_Collections_ERD.puml!PLAYLIST_MODEL

@enduml

3.3. Benefits of Subsection Inclusion

  1. Selective composition - include only what you need
  2. Multiple levels of detail - show high-level or detailed views
  3. Avoid duplication - define entities once, reuse everywhere
  4. Maintainability - update the source, all diagrams reflect changes

3.4. Example: Cross-Diagram References

In music-db-Moods_Collections_ERD.puml:

package "Playlist Collections" <<Only the relevant fields are shown>> #LightGray {
  !includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Playlists_Collections_ERD.puml!PLAYLIST_MODEL
}

This imports the playlists entity definition without duplicating code, showing how moods relate to playlists.

3.4.1. Example Diagrams

Playlists Collections:

Music Database - Playlists Collections ERD

View source on GitHub

View PlantUML source code
@startuml Playlists Collections - Entity Relationship Diagram

title MongoDB Collections: playlist_instances and playlists

!include content/docs/howtos/how-to-write-plantuml/database/db_theme_standard.puml

' PLAYLIST_MODEL
!startsub PLAYLIST_MODEL
ENTITY playlists {
  PK _id : ObjectId
  --
  + IDX reference_code : string
  + IDX is_featured : bool
  + IDX must_tag_moods_at : datetime|null
}
!endsub

!startsub PLAYLIST_MODEL_DETAILS
ENTITY playlists {
  # moods : ObjectId[]
  # locale : string
  # title : string
  # description : string
  # genre : string
  # theme : string
  # target_audience : string
  # moods_tagged_at : datetime|null
  - created_at : datetime
  - updated_at : datetime
  __Indexes__
  SPARSE is_featured_reference_code_idx(is_featured ASC, reference_code ASC)
  SPARSE must_tag_moods_at_idx (must_tag_moods_at ASC)
}
!endsub

' PLAYLIST_INSTANCES_MODEL
!startsub PLAYLIST_INSTANCES_MODEL

ENTITY playlist_instances {
  PK _id : ObjectId
  --
  + IDX instance_urn : string
  + IDX playlist : Link[playlists]
  + IDX playlist_urn : string
  + IDX has_moods : bool
  + IDX is_published : bool
}

' Relationships
playlist_instances ||--o{ playlists : "playlist_id"

!endsub

!startsub PLAYLIST_INSTANCES_MODEL_DETAILS
ENTITY playlist_instances {
uuid : UUID
is_moods_based_suggestion : bool
-created_at : datetime
-updated_at : datetime
__Indexes__
IDX playlist_id_idx(playlist.$id ASC)
IDX instance_urn_has_moods_is_published_idx(instance_urn ASC, has_moods ASC, is_published ASC)
UNIQUE playlist_urn_instance_urn_unique_idx(playlist_urn ASC, instance_urn ASC)
}
!endsub


note right of playlists::is_featured_reference_code_idx
  **Index Purpose:**
  Contains playlist data from various music streaming
  platforms. Serves as main source for user suggestions.
end note

note right of playlist_instances::playlist_urn_instance_urn_unique_idx
  **Index Purpose:**
  Featured playlists catalogue with shared
  information across multiple playlist versions.
  Master template for mood tagging reference.
end note

note bottom of playlists
  **Relationship Pattern:**
  One featured_playlist can be associated with multiple playlist records
  playlist_instances.playlist.$id  playlist._id
end note

@enduml

Moods Collections:

Music Database - Moods Collections ERD

View source on GitHub

View PlantUML source code
@startuml Moods Collections - Entity Relationship Diagram

title MongoDB Collections: Moods taxonomy and related collections - Entity Relationship Diagram

!include content/docs/howtos/how-to-write-plantuml/database/db_theme_standard.puml


package "Playlist Collections" <<Only the relevant fields are shown>> #LightGray {
  !includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Playlists_Collections_ERD.puml!PLAYLIST_MODEL
}
!startsub MODEL_USER_TASTES
ENTITY user_tastes {
  PK _id : ObjectId
  --
  + user_urn : string
  + instance_urn : string
  + has_moods : boolean
  moods_ids : ObjectId[]
}
!endsub

!startsub MODEL_USER_TASTES_DETAILS
ENTITY user_tastes {
DETAILS created_at : datetime
DETAILS updated_at : datetime
__Indexes__
UNIQUE_SPARSE user_urn_unique_idx(user_urn ASC)
SPARSE instance_urn_has_moods_idx(instance_urn ASC, has_moods DESC)
}
!endsub

!startsub MODEL_MOOD
ENTITY mood {
PK _id : ObjectId
--
+ name : string
+ vector: float[]
+ vector_cohere: float[]
+ translations : dict<string, string>
}
!endsub

!startsub MODEL_MOOD_DETAILS
ENTITY mood {
__Indexes__
UNIQUE name_unique_idx(name ASC)
SPARSE vector_exists_idx(vector ASC)
}
' Relationships
mood }o--o{ user_tastes : "moods_ids references _id in mood"
mood }o-r-o{ playlists : "moods references _id in mood"
!endsub

note left of mood::name
  The name of the mood in English
  (e.g., "energetic", "chill", "melancholic")
end note
note left of mood::translations
  A dictionary containing translations
  of the mood name in various languages.
  The keys are BCP-47 language (e.g., "fr" for French)
  and the values are the translated mood names.
end note
note right of mood::vector
  A vector representation of the mood.name field,
  generated using amazon.titan-embed-text-v1.
  This field is used for semantic search and
  similarity comparisons between moods.
end note

@enduml

Suggestions Collections:

Music Database - Suggestions Collections ERD

View source on GitHub

View PlantUML source code
@startuml Suggestions Collections - Entity Relationship Diagram

title MongoDB Collections: Suggestion System - Entity Relationship Diagram
!include content/docs/howtos/how-to-write-plantuml/database/db_theme_standard.puml

package "Playlist Collections" <<Only the relevant fields are shown>> as PC #LightGray {
  !includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Playlists_Collections_ERD.puml!PLAYLIST_MODEL
  !includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Playlists_Collections_ERD.puml!PLAYLIST_INSTANCES_MODEL
}

package "User Collections" <<Only the relevant fields are shown>> as UC #LightGray{
  !includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Moods_Collections_ERD.puml!MODEL_USER_TASTES
}

!startsub MODEL_PENDING_USER_SUGGESTION
ENTITY pending_user_suggestion {
  PK _id : ObjectId
  --
  + IDX user_tastes_id: ObjectId
  + IDX playlist_instances_id : ObjectId
  + IDX computed_at : datetime
  user_urn : string
  user_moods : ObjectId[]
  playlist_urn : string
  playlist_uuid : UUID
  playlist_moods : ObjectId[]
  playlist_instance_is_suggestible : bool
  instance_urn : string
}
!endsub

!startsub MODEL_PENDING_USER_SUGGESTION_DETAILS
ENTITY pending_user_suggestion {
  __Indexes__
  UNIQUE user_tastes_playlist_instances_unique_idx(user_tastes_id ASC, playlist_instances_id ASC)
  IDX compute_at_idx(computed_at ASC)
  IDX playlist_instances_idx(playlist_instances_id ASC)
}
!endsub

!startsub MODEL_USER_SUGGESTION
ENTITY user_suggestion {
  PK _id : ObjectId
  --
  +user_tastes_id: ObjectId
  +user_urn : string
  +playlist_instances_id : ObjectId
  +playlist_instance_is_suggestible : bool
  +score : float
  playlist_urn : string
  playlist_uuid : UUID
  instance_urn : string
  computed_at : datetime
}

' Relationships
user_tastes ||--o{ pending_user_suggestion
playlist_instances ||--o{ pending_user_suggestion
user_tastes ||-d-o{ user_suggestion
playlist_instances ||-u-o{ user_suggestion

!endsub

!startsub MODEL_USER_SUGGESTION_DETAILS
ENTITY user_suggestion {
  __Indexes__
  UNIQUE user_tastes_playlist_instances_unique_idx(user_tastes_id ASC, playlist_instances_id ASC)
  IDX user_tastes_score_idx(user_tastes_id ASC, score DESC)
  IDX playlist_instances_idx(playlist_instances_id ASC)
  IDX user_urn_is_suggestible_score_idx(user_urn ASC, playlist_instance_is_suggestible DESC, score DESC)
}
!endsub
@enduml

4. Master Diagram: music-db-All_collections.puml

The music-db-All_collections.puml file demonstrates diagram composition by combining all subsections into a complete database schema view.

4.1. How It Works

@startuml
!pragma layout smetana
!include content/docs/howtos/how-to-write-plantuml/database/db_theme_standard.puml

' Include playlist entities
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Playlists_Collections_ERD.puml!PLAYLIST_MODEL
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Playlists_Collections_ERD.puml!PLAYLIST_MODEL_DETAILS

' Include mood entities
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Moods_Collections_ERD.puml!MODEL_MOOD
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Moods_Collections_ERD.puml!MODEL_MOOD_DETAILS

' ... and so on for all collections
@enduml

4.2. Advantages

  • Single comprehensive view of the entire database
  • No code duplication - entities defined once in source files
  • Automatic updates - changes propagate from source diagrams
  • Flexible composition - easily add/remove collections

This pattern is perfect for creating both detailed individual diagrams and high-level overview diagrams from the same source.

4.2.1. Master Diagram View

Music Database - All Collections

View source on GitHub

View complete PlantUML source code
@startuml
!pragma layout smetana
!include content/docs/howtos/how-to-write-plantuml/database/db_theme_standard.puml
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Playlists_Collections_ERD.puml!PLAYLIST_MODEL
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Playlists_Collections_ERD.puml!PLAYLIST_MODEL_DETAILS
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Playlists_Collections_ERD.puml!PLAYLIST_INSTANCES_MODEL
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Playlists_Collections_ERD.puml!PLAYLIST_INSTANCES_MODEL_DETAILS
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Moods_Collections_ERD.puml!MODEL_USER_TASTES
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Moods_Collections_ERD.puml!MODEL_USER_TASTES_DETAILS
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Moods_Collections_ERD.puml!MODEL_MOOD
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Moods_Collections_ERD.puml!MODEL_MOOD_DETAILS
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Suggestions_Collections_ERD.puml!MODEL_PENDING_USER_SUGGESTION
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Suggestions_Collections_ERD.puml!MODEL_PENDING_USER_SUGGESTION_DETAILS
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Suggestions_Collections_ERD.puml!MODEL_USER_SUGGESTION
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Suggestions_Collections_ERD.puml!MODEL_USER_SUGGESTION_DETAILS
!includesub content/docs/howtos/how-to-write-plantuml/database/music-db-Logs.puml!MODEL_MUSIC_API_LOGS
@enduml

5. JSON Data Structures in PlantUML

PlantUML supports embedding JSON notation directly in diagrams, useful for documenting:

  • API request/response structures
  • Database document schemas (MongoDB)
  • Configuration examples
  • Data transformation flows

5.1. Example from music-db-Logs.puml

json payload as "**Payload Example**" {
  "**field**": "title",
  "**value**": "Summer Vibes Mix",
  "**fallback_locale**": "en-US"
}
music_api_logs -r-> payload: payload

This creates a formatted JSON box linked to an entity, showing exactly what data structure is used.

5.2. Complete Document Example

The logs diagram also shows a complete MongoDB document:

json complete_example as "**Complete Document Example**" {
    "**_id**": "698ddd946f3bad1915a67e87",
    "**instance_urn**": "urn:music:spotify",
    "**user_urn**": "urn:music:spotify:user/...",
    "**tag**": "music-ai.playlist_metadata.generate",
    "**payload**": {
        "**field**": "description",
        "**playlist_title**": "Acoustic Coffee House",
        "**additional_metadata**": { "..." }
    },
    "**prediction**": ["..."]
}

5.3. Benefits

  • Precise schema documentation alongside ERD diagrams
  • Visual clarity - readers see exact data structures
  • Version control - schema examples tracked with diagrams
  • Testing reference - developers can use examples for test data

5.4. Complete Logs Diagram Example

Here’s the complete logs diagram showing JSON structures and their relationships:

Music Database - Logs

View source on GitHub

View PlantUML source code
@startuml

title MongoDB Collections: Music API Logs - Entity Relationship Diagram
!include content/docs/howtos/how-to-write-plantuml/database/db_theme_standard.puml

!startsub MODEL_MUSIC_API_LOGS
ENTITY music_api_logs {
  PK _id : ObjectId
  --
  + instance_urn : string
  user_urn : string
  + created_at : datetime
  tag : string
  target_field : string
  payload : dict
  prediction : array
}

json payload_schema as "**Payload Schema**" {
  "**field**": "enum(description|theme|target_audience)",
  "**playlist_title**": "string",
  "**fallback_locale**": "en-US",
  "**additional_metadata**": {
    "**description**": "string",
    "**theme**": "string",
    "**target_audience**": "string"
  },
  "**example**": "string"
}
music_api_logs --> payload_schema: payload schema

!endsub

!startsub MODEL_MUSIC_API_LOGS_DETAILS
ENTITY music_api_logs {
  __Indexes__
  IDX ttl_idx_90(created_at ASC)
  IDX instance_urn_created_at_idx(instance_urn ASC, created_at DESC)
}

note left of music_api_logs::ttl_idx_90
  **Index Purpose:**
  Automatically delete documents after a certain
  period (e.g., 90 days) to manage storage and
  ensure data relevance.
  **expireAfterSeconds:**  7776000 (90 days)
end note

json payload as "**Payload Example**" {
  "**field**": "title",
  "**value**": "Summer Vibes Mix",
  "**fallback_locale**": "en-US"
}
music_api_logs -r-> payload: payload

json prediction as "**Prediction Example**" {
  [
    "Feel-Good Summer Playlist",
    "Sunshine & Good Times",
    "Ultimate Summer Anthems"
  ]
}
music_api_logs -d-> prediction: prediction

json complete_example as "**Complete Document Example**" {
    "**_id**": "698ddd946f3bad1915a67e87",
    "**instance_urn**": "urn:music:spotify",
    "**user_urn**": "urn:music:spotify:user/DB991B65-1B6F-A942-32BD-4558C0ED7AF4",
    "**tag**": "music-ai.playlist_metadata.generate",
    "**target_field**": "description",
    "**payload**": {
        "**field**": "description",
        "**playlist_title**": "Acoustic Coffee House",
        "**fallback_locale**": "en-US",
        "**additional_metadata**": {
            "**description**": "Warm acoustic sounds perfect for your morning coffee. Featuring indie folk, singer-songwriter gems **(truncated...)**",
            "**theme**": null,
            "**target_audience**": null
        },
        "**example**": "ex: Discover fresh indie folk tracks to elevate your coffee ritual **(truncated...)**"
    },
    "**prediction**": [
      "Cozy acoustic melodies and indie folk favorites to accompany your coffee break **(truncated...)**"
    ],
    "**created_at**": "2026-02-12T14:03:00.928Z"
}
music_api_logs --> complete_example


!endsub
@enduml

4.3 - How to Write Dockerfiles

Best practices for writing efficient and secure Dockerfiles

1. Dockerfile best practices

Follow official best practices and you can follow these specific best practices

  • But The worst so-called “best practice” for Docker

    Backup, explains why you should actually also use apt-get upgrade

  • Use hadolint

  • Use ;\ to separate each command line

    • some Dockerfiles are using && to separate commands in the same RUN instruction (I was doing it too ;-), but I strongly discourage it because it breaks the checks done by set -o errexit
    • set -o errexit makes the whole RUN instruction to fail if one of the commands has failed, but it is not the same when using &&
  • One package by line, packages sorted alphabetically to ease readability and merges

  • Always specify the most exact version possible of your packages (to avoid to get major version that would break your build or software)

  • do not usage docker image with latest tag, always specify the right version to use

2. Basic best practices

2.1. Best Practice #1: Merge the image layers

in a Dockerfile each RUN command will create an image layer.

2.1.1. Bad practice #1

Here a bad practice that you shouldn’t follow

avoid layer cache issue

2.1.2. Best practice #1

Best practice #1 merge the RUN layers to avoid cache issue and gain on total image size

FROM ubuntu:20.04

RUN apt-get update \
    && apt-get install -y apache2 \
    && rm -rf \
        /var/lib/apt/lists/* \
        /tmp/* \
        /var/tmp/* \
        /usr/share/doc/*

2.2. Best Practice #2: trace commands and fail on error

from previous example we want to trace each command that is executed

2.2.1. Bad practice #2

when building complex layer and one of the command fails, it’s interesting to know which command makes the build to fail

FROM ubuntu:20.04

RUN apt-get update \
    && [ -d badFolder ] \
    && apt-get install -y apache2 \
    && rm -rf \
          /var/lib/apt/lists/* \
          /tmp/* \
          /var/tmp/* \
          /usr/share/doc/*

docker build .  gives the following log output(partly truncated):

...
#5 [2/2] RUN apt-get update
    && [ -d badFolder ]
    && apt-get install -y apache2
    && rm -rf
      /var/lib/apt/lists/*
      /tmp/*
      /var/tmp/*
      /usr/share/doc/*
#5 3.818 Get:1 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
...
#5 6.252 Fetched 25.6 MB in 6s (4417 kB/s)
#5 6.252 Reading package lists...
#5 ERROR: process "/bin/sh -c apt-get update
  && [ -d badFolder ]
  && apt-get install -y apache2
  && rm -rf
    /var/lib/apt/lists/*
    /tmp/*
    /var/tmp/*
    /usr/share/doc/*"
did not complete successfully: exit code: 1
------
> [2/2] RUN apt-get update
  && [ -d badFolder ]
  && apt-get install -y apache2
  && rm -rf
    /var/lib/apt/lists/*
    /tmp/*
    /var/tmp/*
    /usr/share/doc/*:
#5 5.383 Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]
...

------
Dockerfile1:3
--------------------
  2 |
  3 | >>> RUN apt-get update \
  4 | >>>     && [ -d badFolder ] \
  5 | >>>     && apt-get install -y apache2 \
  6 | >>>     && rm -rf \
  7 | >>>           /var/lib/apt/lists/\* \
  8 | >>>           /tmp/\* \
  9 | >>>           /var/tmp/\* \
  10 | >>>           /usr/share/doc/\*
  11 |
--------------------
ERROR: failed to solve: process "/bin/sh -c apt-get update
  && [ -d badFolder ]
  && apt-get install -y apache2
  && rm -rf
    /var/lib/apt/lists/*
    /tmp/*
    /var/tmp/*
    /usr/share/doc/*
did not complete successfully: exit code: 1

Not easy here to know that the command [ -d badFolder ] makes the build failing

Without the best practice #2, the following code build successfully

FROM ubuntu:20.04

RUN set -x ;\
    apt-get update ;\
    [ -d badFolder ] ;\
    ls -al

2.2.2. Best Practice #2

Best Practice #2: Override SHELL options of the RUN command and use ;\ instead of &&

The following options are set on the shell to override the default behavior:

  • set -o pipefail: The return status of a pipeline is the exit status of the last command, unless the pipefail option is enabled.
    • If pipefail is enabled, the pipeline’s return status is the value of the last (rightmost) command to exit with a non-zero status, or zero if all commands exit successfully.
    • without it, a command failure could be masked by the command piped after it
  • set -o errexit (same as set -e): Exit immediately if a pipeline (which may consist of a single simple command), a list, or a compound command (see SHELL GRAMMAR above), exits with a non-zero status.
  • set -o xtrace(same as set -x):  After  expanding  each  simple  command, for command, case command, select command, or arithmetic for command, display the expanded value of PS4, followed by the command and its expanded arguments or associated word list.

Those options are not mandatory but are strongly advised. Although there are some workaround to know:

  • if a command can fail and you want to ignore it, you can use
    • commandThatCanFail || true

These options can be used with /bin/sh as well.

Also it is strongly advised to use ;\ to separate commands because it could happen that some errors are ignored when && is used in conjunction with ||

FROM ubuntu:20.04

# The SHELL instructions will be applied to all the subsequent RUN instructions
SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "xtrace", "-c"]
RUN apt-get update ;\
    [ -d  badFolder ] ;\
    apt-get install -y apache2 ;\
    rm -rf \
          /var/lib/apt/lists/* \
          /tmp/* \
          /var/tmp/* \
          /usr/share/doc/*

docker build .  gives the following log output(partly truncated):

...
#5 [2/2] RUN apt-get update ;
  [ -d  badFolder ] ;
  apt-get install -y apache2 ;
  rm -rf
    /var/lib/apt/lists/*
    /tmp/*
    /var/tmp/*
    /usr/share/doc/*
#5 0.318 + apt-get update
#5 3.522 Get:1 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
...
#5 5.310 Fetched 25.6 MB in 5s (5141 kB/s)
#5 5.310 Reading package lists...
#5 6.172 + '[' -d badFolder ']'
#5 ERROR: process "/bin/bash -o pipefail -o errexit -o xtrace -c
  apt-get update ;
  [ -d  badFolder ] ;
  apt-get install -y apache2 ;
  rm -rf
    /var/lib/apt/lists/*
    /tmp/*
    /var/tmp/*
    /usr/share/doc/*
did not complete successfully: exit code: 1
------
 > [2/2] RUN apt-get update ;
  [ -d  badFolder ] ;
  apt-get install -y apache2 ;
  rm -rf
    /var/lib/apt/lists/*
    /tmp/*
    /var/tmp/*
    /usr/share/doc/\*:
#5 4.228 Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3014 kB]
...
#5 6.172 + '[' -d badFolder ']'
------
Dockerfile1:4
--------------------
   3 |     SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "xtrace", "-c"]
   4 | >>> RUN apt-get update ;\
   5 | >>>     [ -d  badFolder ] ;\
   6 | >>>     apt-get install -y apache2 ;\
   7 | >>>     rm -rf \
   8 | >>>           /var/lib/apt/lists/\* \
   9 | >>>           /tmp/\* \
  10 | >>>           /var/tmp/\* \
  11 | >>>           /usr/share/doc/\*
  12 |
--------------------
ERROR: failed to solve: process "/bin/bash -o pipefail -o errexit -o xtrace -c
apt-get update ;    [ -d  badFolder ] ;    apt-get install -y apache2 ;    rm -rf
/var/lib/apt/lists/*         /tmp/*         /var/tmp/*         /usr/share/doc/*"
did not complete successfully: exit code: 1

Here the command line displayed just above the error indicates clearly from where the error comes from:

#5 6.172 + '[' -d badFolder ']'

2.3. Best practice #3: packages ordering and versions

Best Practice #3: order packages alphabetically, always specify packages versions, ensure non interactive

From previous example we want to install several packages

2.3.1. Bad practice #3

let’s add some packages on our previous example (errors removed)

The following docker has the following issues:

  • it doesn’t set the package versions
  • the installation will install also the recommended packages
  • it’s using apt instead of apt-get (hadolint warning DL3027 Do not use apt as it is meant to be a end-user tool, use apt-get or apt-cache instead)
  • the packages are not ordered alphabetically
FROM ubuntu:20.04

SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "xtrace", "-c"]
RUN apt update ;\
    apt install -y php7.4 apache2 php7.4-curl redis-tools ;\
    rm -rf \
          /var/lib/apt/lists/* \
          /tmp/* \
          /var/tmp/* \
          /usr/share/doc/*  

2.3.2. Best Practice #3

Best Practice #3: order packages alphabetically, always specify packages versions, ensure non interactive

2.3.2.1. Order packages alphabetically and one package by line

one package by line allows packages to be simpler ordered alphabetically

one package by line and ordering alphabetically allows :

  • to merge branches changes more easily
  • to detect redundancies more easily
  • to improve readability
2.3.2.2. Always specify packages versions

over the time your build’s dependencies could be updated on the remote repositories and your packages be unattended upgraded to the latest version making your software breaks because it doesn’t manage the changes of the new package.

It happens several times for me, for example, in 2021, xdebug has been automatically upgraded on one of my docker image from version 2.8 to 3.0 making all the dev environments broken. It happens also on a build pipeline with a version of npm gulp that has been upgraded to latest version. In both cases we resolved the issue by downgrading the version to the one we were using.

2.3.2.3. Ensure non interactive

some apt-get packages could ask for interactive questions, you can avoid this using the env variable DEBIAN_FRONTEND=noninteractive

Note: ARG instruction allows to set env variable available only during build time

FROM ubuntu:20.04

SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "xtrace", "-c"]

ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update ;\
    apt-get install -y -q --no-install-recommends \
        # Mind to use quotes to avoid shell to try to expand * with some files
        apache2='2.4.*' \
        php7.4='7.4.*' \
        php7.4-curl='7.4.*' \
        # Notice the ':'(colon)
        redis-tools='5:5.*' \
    ;\
    # cleaning
    apt-get autoremove -y ;\
    apt-get -y clean ;\
    rm -rf \
        /var/lib/apt/lists/* \
        /tmp/* \
        /var/tmp/* \
        /usr/share/doc/*

# use the following command to know the current version of the packages
# using another RUN instead of using previous one will avoid the whole
# previous layer to be rebuilt
# RUN apt-cache policy \
# apache2 \
# php7.4 \
# php7.4-curl \
# redis-tools
# Gives the following output
#6 0.387 + apt-cache policy apache2
#6 0.399 apache2:
#6 0.399   Installed: 2.4.41-4ubuntu3.14
#6 0.399   Candidate: 2.4.41-4ubuntu3.14
#6 0.399   Version table:
#6 0.399  *** 2.4.41-4ubuntu3.14 100
#6 0.399         100 /var/lib/dpkg/status
#6 0.400 + apt-cache policy php7.4
#6 0.409 php7.4:
#6 0.409   Installed: 7.4.3-4ubuntu2.18
#6 0.409   Candidate: 7.4.3-4ubuntu2.18
#6 0.409   Version table:
#6 0.409  *** 7.4.3-4ubuntu2.18 100
#6 0.409         100 /var/lib/dpkg/status
#6 0.409 + apt-cache policy php7.4-curl
#6 0.420 php7.4-curl:
#6 0.420   Installed: 7.4.3-4ubuntu2.18
#6 0.420   Candidate: 7.4.3-4ubuntu2.18
#6 0.420   Version table:
#6 0.420  *** 7.4.3-4ubuntu2.18 100
#6 0.421         100 /var/lib/dpkg/status
#6 0.421 + apt-cache policy redis-tools
#6 0.431 redis-tools:
#6 0.431   Installed: 5:5.0.7-2ubuntu0.1
#6 0.431   Candidate: 5:5.0.7-2ubuntu0.1
#6 0.431   Version table:
#6 0.431  *** 5:5.0.7-2ubuntu0.1 100
#6 0.432         100 /var/lib/dpkg/status

2.4. Best practice #4: ensure image receives latest security updates

from previous example we want to ensure the image receives the latest security updates

2.4.1. Bad practice #4

registry image are not always updated and latest apt security updates are not installed

FROM ubuntu:20.04

SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "xtrace", "-c"]

ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update ;\
    apt-get install -y -q --no-install-recommends \
        apache2='2.4.*' \
        php7.4='7.4.*' \
        php7.4-curl='7.4.*' \
        redis-tools='5:5.*' \
    ;\
    # cleaning
    apt-get autoremove -y ;\
    apt-get -y clean ;\
    rm -rf \
        /var/lib/apt/lists/* \
        /tmp/* \
        /var/tmp/* \
        /usr/share/doc/*

2.4.2. Best Practice #4

be sure to apply latest security updates, to install the latest security updates in the image, keep sure to call apt-get upgrade -y

Here the updated Dockerfile:

FROM ubuntu:20.04

SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "xtrace", "-c"]

ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update ;\
    # be sure to apply latest security updates
    # https://pythonspeed.com/articles/security-updates-in-docker/
    apt-get upgrade -y ;\
    apt-get install -y -q --no-install-recommends \
        apache2='2.4.*' \
        php7.4='7.4.*' \
        php7.4-curl='7.4.*' \
        redis-tools='5:5.*' \
    ;\
    # cleaning
    apt-get autoremove -y ;\
    apt-get -y clean ;\
    rm -rf \
        /var/lib/apt/lists/* \
        /tmp/* \
        /var/tmp/* \
        /usr/share/doc/*

2.5. Conclusion: image size comparison

from previous example we want to ensure the image receives the latest security updates

2.5.1. Dockerfile without best practices

FROM ubuntu:20.04

ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update
RUN apt-get install -y apache2 php7.4 php7.4-curl redis-tools
# cleaning
RUN apt-get autoremove -y ;\
    apt-get -y clean ;\
    rm -rf \
        /var/lib/apt/lists/* \
        /tmp/* \
        /var/tmp/* \
        /usr/share/doc/*

2.5.2. Dockerfile with all optimizations

FROM ubuntu:20.04

SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "xtrace", "-c"]

ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update ;\
    apt-get upgrade -y ;\
    apt-get install -y -q --no-install-recommends \
        apache2='2.4.*' \
        php7.4='7.4.*' \
        php7.4-curl='7.4.*' \
        redis-tools='5:5.*' \
    ;\
    # cleaning
    apt-get autoremove -y ;\
    apt-get -y clean ;\
    rm -rf \
        /var/lib/apt/lists/* \
        /tmp/* \
        /var/tmp/* \
        /usr/share/doc/*

3. Docker Buildx best practices

3.1. Optimize image size

Source: https://askubuntu.com/questions/628407/removing-man-pages-on-ubuntu-docker-installation

Let’s consider this example

3.1.1. Dockerfile not optimized

FROM ubuntu:20.04 as stage1

ARG DEBIAN_FRONTEND=noninteractive

SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "xtrace", "-c"]
RUN \
    apt-get update ;\
    apt-get install -y -q --no-install-recommends \
        htop

FROM stage1 as stage2

SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "xtrace", "-c"]
RUN \
    # here we just test that the ARG DEBIAN_FRONTEND has been inherited from
    # previous stage (it is the case)
    echo "DEBIAN_FRONTEND=${DEBIAN_FRONTEND}"

Now let’s build and check the image size, the best way to do this is to export the image to a file

docker build and save:

docker build -f Dockerfile1 -t test1 .
docker save test1 -o test1.tar

Now we will optimize this image by removing man pages (you can still find man pages on the web) and removing apt cache

3.1.2. Dockerfile optimized

FROM ubuntu:20.04 as stage1

ARG DEBIAN_FRONTEND=noninteractive

COPY 01-noDoc /etc/dpkg/dpkg.cfg.d/

COPY 02-aptNoCache /etc/apt/apt.conf.d/
SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "xtrace", "-c"]
RUN \
    # remove apt cache and man/doc
    rm -rf /var/cache/apt/archives /usr/share/{doc,man,locale}/ ;\
    \
    apt-get update ;\
    apt-get install -y -q --no-install-recommends \
        htop \
    ;\
    # clean apt packages
    apt-get autoremove -y ;\
    ls -al /var/cache/apt ;\
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* /usr/share/doc/*

FROM stage1 as stage2

SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "xtrace", "-c"]
RUN \
    echo "DEBIAN_FRONTEND=${DEBIAN_FRONTEND}"

Here the content of /etc/dpkg/dpkg.cfg.d/01-noDoc, it will tell apt to not install man docs and translations

# /etc/dpkg/dpkg.cfg.d/01_nodoc

# Delete locales
path-exclude=/usr/share/locale/*

# Delete man pages
path-exclude=/usr/share/man/*

# Delete docs
path-exclude=/usr/share/doc/*
path-include=/usr/share/doc/*/copyright

Here the content of /etc/apt/apt.conf.d/02-aptNoCache, it will instruct apt to not store any cache (note that apt-get clean will not work after that change but you don’t need to use it anymore)

Dir::Cache "";
Dir::Cache::archives "";

Now let’s build and check the image size, the best way to do this is to export the image to a file

docker build and save:

docker build -f Dockerfile2 -t test2 .
docker save test2 -o test2.tar

Here the size of the files

test1.tar 117 020 672 bytes
test2.tar  76 560 896 bytes

We passed from ~117MB to ~76MB so we gain ~41MB Please note also that we used --no-install-recommends option in both example that allows us to save some other MB

4.4 - How to Write Docker Compose Files

Guide to writing and organizing Docker Compose files

1. platform

as not everyone is using the same environment (some are using MacOS for example which is targeting arm64 instead of amd64), it is advised to add this option to target the right architecture

docker-compose platform:

services:
  serviceName:
    platform: linux/x86_64
  # ...

2. Wait for a service to be healthy before starting another one

If you have a service that depends on another one, it is important to wait for the dependent service to be healthy before starting the dependent one. This can be achieved using the depends_on option with the condition: service_healthy condition.

Here is an example where serviceB depends on serviceA being healthy before it starts:

services:
  serviceA:
    # ...
    healthcheck:
      test: [CMD, curl, -f, http://localhost:8080/health]
      interval: 30s
      timeout: 10s
      retries: 3
  serviceB:
    # ...
    depends_on:
      serviceA:
        condition: service_healthy

In this example, api service will wait for the db service to be healthy before it starts. The health check for the db service is defined to check if the MySQL server is responding to ping requests.

version: '2.1'
services:
  api:
    build: .
    container_name: api
    ports:
      - 8080:8080
    depends_on:
      db:
        condition: service_healthy
  db:
    container_name: db
    image: mysql
    ports:
      - '3306'
    environment:
      MYSQL_ALLOW_EMPTY_PASSWORD: yes
      MYSQL_USER: user
      MYSQL_PASSWORD: password
      MYSQL_DATABASE: database
    healthcheck:
      test: [CMD, mysqladmin, ping, -h, localhost]
      timeout: 20s
      retries: 10

4.5 - Saml2Aws Setup

Guide to setting up and using Saml2Aws for AWS access

Configure saml2aws accounts

saml2aws configure \
  --idp-account='<account_alias>' \
  --idp-provider='AzureAD' \
  --mfa='Auto' \
  --profile='<profile>' \
  --url='https://account.activedirectory.windowsazure.com' \
  --username='<username>@microsoft.com' \
  --app-id='<app_id>' \
  --skip-prompt
  • <app_id> is a unique identifier for the application we want credentials for (in this case an AWS environment).
  • <account_alias> serves as a name to identify the saml2aws configuration (see your ~/.saml2aws file
  • <profile> serves as the name of the aws cli profile that will be created when you log in.

This will automatically identify your tenant ID based on the AppID and will create a configuration based on the provided information. Configuration will be created in ~/.saml2aws

1. Use saml2aws login command to configure the AWS CLI profile

Run saml2aws login to add or refresh your profile for the aws cli.

saml2aws login -a ${account_alias}

Follow the prompts to enter your SSO credentials and complete the multi-factor authentication step.

Note: if you are part of multiple roles you can use –role flag to configure the required role.

Above steps have been taken from below GitHub Repo. They have been tried in MacOS, Windows, Linux and Windows WSL https://github.com/Versent/saml2aws

2. Kubernetes connection

Adding a newly created Technology Convergence EKS cluster to your ~/.kube/config:

Add EKS Cluster to ~/.kube/config

aws eks update-kubeconfig --name $clusterName --region us-east-1

3. Common issues

3.1. Error - error authenticating to IdP: unable to locate IDP OIDC form submit URL

This is very likely because you changed your account password. Reenter your password when prompted at saml2aws login

3.2. Error - error authenticating to IdP: unable to locate SAMLRequest URL

This is very likely because you do not have access to this AWS account.

Multifactor authentication asks for a number, but the terminal doesn’t provide a number.

Solution 1: We’ve found that going to your Microsoft account security info and deleting and re-adding the sign-in method seems to fix the issue. You should then be able to just enter a Time-based one-time password from your Microsoft Authenticator app.

Solution 2: You can change the MFA option for your saml2aws config either with PhoneAppOTP, PhoneAppNotification, or OneWaySMS. Something like this in your ~/.saml2aws file

name          = tc-dev
app_id         = 83cffb56-1d1b-400c-ad47-345c58e378dc
url           = https://account.activedirectory.windowsazure.com
username        = <>@microsoft.com
provider        = AzureAD
mfa           = OneWaySMS
skip_verify       = false
timeout         = 0
aws_urn         = urn:amazon:webservices
aws_session_duration  = 3600
aws_profile       = dev
resource_id       =
subdomain        =
role_arn        =
region         =
http_attempts_count   =
http_retry_delay    =
credentials_file    =
saml_cache       = false
saml_cache_file     =
target_url       =
disable_remember_device = false
disable_sessions    = false
prompter        =

for more reference, follow this page https://github.com/Versent/saml2aws/blob/master/doc/provider/aad/README.md#configure

4.6 - Debug Hugo

Debug Hugo: Quick guide to locate Hugo templates, use templateMetrics, override priorities, and fix common Docsy/Hugo template issues for faster debugging.

1. Finding Which Template is Being Used

Method 1: Template Metrics (Recommended)

hugo server -D --templateMetrics --templateMetricsHints

This shows which templates are executed and execution times. Look for the page you’re debugging in the output.

Method 2: Add Debug Comments to Templates Add this at the top of any template to verify it’s being used:

<!-- DEBUG: Using template layouts/docs/list.html -->
{{ warnf "TEMPLATE DEBUG: Rendering %s with %s" .RelPermalink .Layout }}

Then check the HTML source or terminal output.

Method 3: Template Path in HTML (Temporary) Add to your template for debugging:

<!-- Template: {{ .Layout }} | Kind: {{ .Kind }} | Type: {{ .Type }} -->
{{ printf "
<!-- File: %s -->
" .File.Path }}

Remove after debugging to keep HTML clean.

2. Hugo Template Lookup Order

For _index.md files (list pages):

content/docs/bash-scripts/_index.md  (with type: docs)
  1. layouts/docs/list.html              ← Create this for docs sections with comments
  2. layouts/docs/section.html
  3. layouts/_default/list.html
  4. layouts/_default/section.html
  5. themes/docsy/layouts/docs/list.html
  6. themes/docsy/layouts/_default/list.html

For regular .md files (single pages):

content/docs/bash-scripts/page.md  (with type: docs)
  1. layouts/docs/single.html            ← Docsy uses baseof.html with blocks
  2. layouts/_default/single.html
  3. layouts/partials/_td-content.html   ← This is where content is rendered in Docsy
  4. themes/docsy/layouts/docs/baseof.html

For blog posts:

content/blog/post.md
  1. layouts/blog/single.html
  2. layouts/_default/single.html
  3. layouts/blog/_td-content.html       ← Override this for blog-specific changes

3. Common Template Debugging Commands

# Verify template exists in lookup path
find . -name "list.html" -o -name "single.html"

# Check if shared layouts are mounted correctly
hugo mod graph

# List all available templates (with jq installed)
hugo config --format json | jq '.module.mounts'

# Rebuild with verbose output
hugo server -D --logLevel debug --disableFastRender

4. Template Override Priority

  1. Local layouts/ directory (highest priority) - repo-specific overrides
  2. Mounted shared/layouts/ from my-documents via Hugo modules
  3. Docsy theme themes/docsy/layouts/ (lowest priority)

5. Key Template Files for Comments/Customization

TemplatePurposeUsed For
shared/layouts/docs/list.htmlDocs section index pages_index.md with type: docs
shared/layouts/blog/_td-content.htmlBlog post content wrapperBlog posts
shared/layouts/_td-content.htmlRegular page content wrapperRegular docs pages
shared/layouts/partials/giscus-comments.htmlGiscus comment widgetIncluded in above templates

6. Common Issues and Solutions

Issue: Comments not showing on _index.md pages
Solution: Create layouts/docs/list.html (not section.html - wrong name!)

Issue: Changes to shared/layouts/ not appearing
Solution: Run hugo mod clean && hugo mod get -u to refresh modules

Issue: Template works locally but not in CI
Solution: Check Hugo modules are committed in go.mod and go.sum

Issue: Wrong template being used
Solution: Check frontmatter type: field - it controls template lookup path

Issue: Print out the full value of a variable in Hugo Solution:

  • {{ printf "%#v" $pages }}
  • <pre>{{ debug.Dump .Params }}</pre>
  • Use the templates.Current function to visually mark template execution boundaries or to display the template call stack.

7. Understanding Docsy’s Template Structure

Docsy uses a block-based template system:

  • baseof.html defines the overall page structure
  • {{ block "main" }} is where content goes
  • _td-content.html partial is called by most layouts
  • Override _td-content.html to customize content rendering globally

8. Quick Debug Workflow

  1. Identify the page type: Regular page, section index, blog post?
  2. Check frontmatter: Look for type: field (e.g., type: docs)
  3. Find template: Use template lookup order above
  4. Verify template exists: Check shared/layouts/[type]/[kind].html
  5. Add debug output: Temporarily add {{ warnf }} to verify
  6. Test locally: hugo server -D --templateMetrics --disableFastRender
  7. Remove debug code: Clean up before committing

5 - Artificial Intelligence

In-depth tutorials and guides on artificial intelligence topics

1. Available Guides

  • Better AI Usage - Comprehensive documentation on how to effectively use AI for learning and productivity

2. Getting Started

Select a guide from the sidebar to begin.

Articles in this section

TitleDescriptionUpdated
Optimize Dockerfiles for Speed and EfficiencyAI Skill for optimizing Dockerfiles, including best practices for reducing image size and improving build performance.2026-05-10 23:42:08 +0200 +0200
Crafting Effective AI Prompts for MongoDB OptimizationCraft precise AI prompts to optimize MongoDB: phased analysis, index tuning, and actionable query recommendations.2026-05-10 23:42:08 +0200 +0200
Better AI Usage for LearningComprehensive documentation on how to effectively use AI for learning and productivity2026-05-10 23:42:08 +0200 +0200

5.1 - Optimize Dockerfiles for Speed and Efficiency

AI Skill for optimizing Dockerfiles, including best practices for reducing image size and improving build performance.

This AI Docker Skill provides personalized recommendations for optimizing your Docker images and improving build performance using an AI Copilot Agent. It covers various aspects of Dockerfile optimization, including:

  • Using multi-stage builds to reduce image size
  • Choosing the right base image for your application
  • Minimizing the number of layers in your Dockerfile
  • Leveraging caching effectively to speed up builds
  • Implementing best practices for security and maintainability

Example of transforming a Dockerfile using the AI Docker Skill:

# Original Dockerfile
FROM node:20-alpine
WORKDIR /app
COPY package.json ./
RUN npm install
COPY . .
RUN npm run build
CMD ["npm", "start"]

1. Q&A with AI

I asked AI using Docker Skill with Claude Sonnet 4.5 AI Model:

Here Q&A with AI:

Do you want a development stage for local development with hot-reloading and dev dependencies?

Yes - Include dev stage with hot-reload support

Should the Dockerfile include a test stage for running unit tests, integration tests, or pre-commit hooks?

Yes - Run tests during build

Do you want security scanning (Trivy, Snyk, etc.) integrated into the build process?

No - Skip security scanning

Should images be signed for verification (using Cosign, Notary, or Docker Content Trust)?

No - Skip image signing

What are the deployment environments? Local - Local development, Staging - Staging environment,

Production - Production environment

Are there specific base image requirements?

node:20-slim - Recommended: Debian-based minimal image

Do you want advanced image size optimization (adds complexity)?

NPM cache optimization - Disable npm cache

2. Resulting Dockerfile

Result: AI automatically generates the optimized Dockerfile based on the provided answers:

# =============================================================================
# Stage Diagram:
#
# base (node:20-slim)
# ├─> development (with hot-reload, dev dependencies)
# ├─> dependencies (production dependencies only)
# │     └─> build (compile application)
# │           └─> test (run tests, validation)
# │                 └─> production (minimal runtime image)
#
# =============================================================================

# =============================================================================
# Stage: base
# Purpose: Common base configuration for all stages
# =============================================================================
FROM node:20-slim AS base
SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "xtrace", "-c"]

# Environment Variables - Build-time Configuration
# Description: Prevent interactive prompts during apt operations
# Required: Yes (for automated builds)
ARG DEBIAN_FRONTEND=noninteractive

# Environment Variables - Runtime Configuration
# Description: Node environment (development, staging, production)
# Default: production
# Example: development, staging, production
# Required: No
ENV NODE_ENV=production

# Description: Enable verbose logging for debugging
# Default: false
# Example: true, false
# Required: No
ENV DEBUG=false

# Description: Application port for the Node.js server
# Default: 3000
# Example: 3000, 8080
# Required: No
ENV PORT=3000

# Description: Prevent npm from creating package-lock updates
# Default: off
# Required: No
ENV NPM_CONFIG_UPDATE_NOTIFIER=off

# Description: Disable npm cache to reduce image size
# Default: false
# Example: true (for production), false (for dev)
# Required: No
ENV NPM_CONFIG_CACHE=/tmp/npm-cache

WORKDIR /app

# Install security updates and minimal system dependencies
# If you need an IMMUTABLE IMAGE, comment out the upgrade
RUN <<EOF
apt-get update
apt-get upgrade -y
apt-get install -y -q --no-install-recommends \
  'ca-certificates=20*' \
  'curl=7.*' \
  'tini=0.19.*'
apt-get autoremove -y
apt-get clean
rm -rf /var/lib/apt/lists/* /var/cache/apt/archives /tmp/* /var/tmp/* /usr/share/{doc,man,locale}/
EOF

# =============================================================================
# Stage: development
# Purpose: Development environment with hot-reload and dev tools
# =============================================================================
FROM base AS development
SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "xtrace", "-c"]

ENV NODE_ENV=development
ENV NPM_CONFIG_CACHE=/home/node/.npm

# Copy dependency files for caching optimization
COPY --chown=node:node package*.json ./

# Install ALL dependencies (including devDependencies)
RUN <<EOF
npm ci
npm cache clean --force
EOF

# Copy source code
COPY --chown=node:node . .

USER node

EXPOSE 3000

# Health check for development
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD curl -f http://localhost:3000/health || exit 1

# Use nodemon or npm run dev for hot-reload (adjust based on your package.json)
CMD ["npm", "run", "dev"]

# =============================================================================
# Stage: dependencies
# Purpose: Install production dependencies only (optimized caching)
# =============================================================================
FROM base AS dependencies
SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "xtrace", "-c"]

# Copy dependency files for caching optimization
COPY package*.json ./

# Install production dependencies only (no dev dependencies)
RUN <<EOF
npm ci --only=production --no-cache
npm cache clean --force
EOF

# =============================================================================
# Stage: build
# Purpose: Build the application from source
# =============================================================================
FROM base AS build
SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "xtrace", "-c"]

# Copy dependency files for caching optimization
COPY package*.json ./

# Install ALL dependencies (including devDependencies for build)
RUN <<EOF
npm ci --no-cache
npm cache clean --force
EOF

# Copy source code
COPY . .

# Build the application
RUN <<EOF
npm run build
EOF

# =============================================================================
# Stage: test
# Purpose: Run tests to validate the build
# =============================================================================
FROM build AS test
SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "xtrace", "-c"]

# Run tests (adjust command based on your package.json)
RUN <<EOF
npm run test
touch /tmp/tests-passed
EOF

# =============================================================================
# Stage: production
# Purpose: Minimal production runtime image
# =============================================================================
FROM base AS production
SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "xtrace", "-c"]

# Create non-root user with specific UID/GID for security
RUN <<EOF
groupadd -r -g 1001 appuser
useradd -r -u 1001 -g appuser -s /sbin/nologin -c "Application User" appuser
mkdir -p /app/logs /app/tmp
chown -R appuser:appuser /app
EOF

# Copy production dependencies from dependencies stage
COPY --from=dependencies --chown=appuser:appuser /app/node_modules ./node_modules

# Copy built application from build stage
COPY --from=build --chown=appuser:appuser /app/dist ./dist
COPY --from=build --chown=appuser:appuser /app/package*.json ./

# Copy test marker to ensure tests passed
COPY --from=test /tmp/tests-passed /tmp/tests-passed

# Switch to non-root user
USER appuser

# Document exposed port
EXPOSE 3000

# Health check for production
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
  CMD curl -f http://localhost:${PORT}/health || exit 1

# Use tini as init system for proper signal handling
ENTRYPOINT ["/usr/bin/tini", "--"]

# Start the application
CMD ["node", "dist/index.js"]

3. Review Comments

During the process, AI automatically generates a package.jso in order to be able to build the image and test it.

The resulting Dockerfile is optimized for both development and production environments, with separate stages for dependencies, build, testing, and production runtime. It incorporates best practices for image size reduction, caching optimization, security hardening, and maintainability.

But Never trust AI blindly - Always review the generated Dockerfile and test it thoroughly to ensure it meets your specific requirements and follows best practices for your application.

Anyway here my review comments about this generated Dockerfile:

  • The COPY --chown=node:node . . in development stage is not necessarily because most of the time we are mounting the source code in development.
  • USER node in development stage is hard to use because of the permissions issues when mounting volumes, we should use root user in development stage.
  • The stage inheritance is wrong while the dependency schema at the beginning of the Dockerfile has not been respected:
    • the build stage is inheriting from base and use npm ci in development mode
    • The production stage inherit from base but copy the node_modules from dependencies stage (it is why AI puts this wrong inheritance)
    • as only production stage need the dependencies, it is better to have the RUN npm ci –only=production in the production stage-

a clearer diagram could be:

# =============================================================================
# Stage Diagram:
#
# base (node:20-slim)
# ├─> development (with hot-reload, dev dependencies)
# ├─> dependencies (production dependencies only)
# ├─> build (compile application)
# │     └─> test (run tests, validation)
# └─> production (minimal runtime image)
# <-- depends on (dependencies, build, test)
#
# =============================================================================

4. Conclusion

OK I agree it is more complex to read but it is more optimized and secured that the original one.

Stay tuned as I will continuously improve this skill with new features and optimizations based on the latest Docker best practices and user feedback!

📢 What do you think about this generated Dockerfile? Do you have any 💡 feedbacks ? Let me know in the 💬 comments!

5.2 - Crafting Effective AI Prompts for MongoDB Optimization

Craft precise AI prompts to optimize MongoDB: phased analysis, index tuning, and actionable query recommendations.

1. Overview

This guide shares practical lessons from using AI to analyze and optimize MongoDB databases across multiple Python repositories. By following these techniques, you’ll learn how to craft prompts that produce accurate, actionable analysis instead of generic recommendations.

Key Takeaway: Precise, well-structured prompts dramatically improve the quality and usefulness of AI-generated database analysis.

2. Context

This guide is based on analyzing two interconnected Python repositories that share a single MongoDB database.

Repository Roles:

ComponentResponsibility
api_pythonDatabase schema, migrations, and Beanie ORM models (source of truth)
kafka-consumerConsuming Kafka events and updating MongoDB collections

Technology Used:

  • MongoDB for data storage
  • Beanie ORM for Python-MongoDB integration
  • Python as the programming language

3. The Challenge

Creating an effective prompt proved unexpectedly difficult. Initial attempts to fit all requirements into a single prompt resulted in timeouts and no output. This is a common problem when asking AI to handle large, complex analysis tasks in one go.

4. Phase 1: Testing a Comprehensive Prompt

The first approach was to create a single detailed prompt that would handle all analysis at once:

---
name: fc-optimize-mongodb
description: Analyze the MongoDB usage in the API and its Kafka Consumer, identify all the CRUD queries that
are made to MongoDB in both projects, and propose optimizations.
---
As a Senior specialist of documentDB and MongoDB optimization, you have been tasked to analyze the MongoDB usage
in the API and its Kafka Consumer. Your goal is to identify all the CRUD queries that are made to MongoDB in both
repositories, and propose optimizations.

**CRITICAL** Analyze all the files **ONLY** in the directories `api_python` and `kafka-consumer`.
But ignore files in `api_python/admin/database/migrations/`.

These 2 projects are separated but they use the same MongoDB database but with different Beanie models definitions.
`api_python` is the project that is responsible for the database schema and migrations.
While `kafka-consumer` is responsible for consuming the Kafka events and updating the MongoDB collections accordingly.
So consider `api_python` as source of truth.

You will analyze current MongoDB schema (api_python/admin/database/indexes/definition.py, api_python/odm/models.py).

**CRITICAL** Don't update source code, just analyze and generate a report in
`docs/ai/{date:YYYY-mm-dd}-mongodbAnalysis/mongodbAnalysis.md` where you will list all the CRUD queries that are made
to MongoDB (normally all the queries are done through Beanie ORM) in both projects.
**CRITICAL** Don't invent any query, just list the queries accurately as they are in the codebase.

When you will have the full view:
- indicate branch name and commit hash of each repositories.
- for each query write
  - the equivalent MongoDB query that could be done through mongosh
  - the file location and line where the query is done
  - the Beanie query that is done in the code
- global recommendations
  - List all the tables and indexes that are used
  - check if all the required indexes are present
  - propose missing indexes creation
  - propose indexes that should be removed
  - indicate as well when an index could be replaced by a lighter index
    - Compound Indexes
    - Partial Indexes
    - Sparse Indexes
    - Partial indexes with filter expressions
    - ...
  - Propose also optimizations to the queries themselves if you see any
  - Propose to improve the indexes with projections.
  - List inconsistencies in the kafka-consumer
  - **CRITICAL** take into consideration property order in the indexes.
  - Propose queries that could be done on production to analyze the query planner behavior and the index
    usage for the most critical queries.
  - for each suggestion propose code snippets of how the query or the index should be updated.
  - Propose schema improvements if you see any.
- if you see any other recommendations put them in another section
- generate a plantuml diagram of the current schema in `docs/ai/{date:YYYY-mm-dd}-mongodbAnalysis/mongodbSchema.puml`
  and another one with the proposed schema in `docs/ai/{date:YYYY-mm-dd}-mongodbAnalysis/mongodbProposedSchema.puml`
  where you will ignore obsolete tables.
- add at the start of the report a summary of the main findings and recommendations.

use askQuestions tool for any question.

if folder `kafka-consumer` doesn't exist use askQuestions to propose the user to create
a symlink using `ln -s ../kafka-consumer kafka-consumer`.

use askQuestions to ask if the user think about using updated branches before starting the analysis.

4.1. First Results and the Limits of Comprehensive Prompts

The first prompt produced a solid report and revealed 3 critical issues that weren’t previously known. This success led to adding more requirements for even deeper analysis. However, the expanded prompt failed with a timeout error—producing no output at all.

5. Phase 2: Why Comprehensive Prompts Fail

5.1. The Timeout Problem

Attempting to add schema consistency checks (validating that all fields updated in kafka-consumer exist in api_python models) caused a timeout with no output.

The fundamental issue: AI has a practical processing time limit of 10-15 minutes per request. Asking for complete analysis of multiple complex topics at once exceeds this limit. The solution is to break large tasks into smaller, manageable phases.

Task Decomposition Strategy:

Instead of: One comprehensive prompt covering everything

Try this: Multiple focused prompts in sequence

Break analysis into phases:

  1. Inventory (What exists?)
  2. Analysis (What’s suboptimal?)
  3. Planning (How to fix it?)
  4. Documentation (Summarize findings)

Each phase completes in 3-5 minutes instead of timeout.

6. Phase 3: Breaking Work Into Manageable Pieces

To solve the timeout problem, the solution split the large task into distinct phases, each with specific, limited objectives and intermediate checkpoints.

6.1. Planning the Phased Approach

Before redesigning, key decisions were made using interactive questions:

6.1.1. Decision 1: Execution Model

  • Approach: Single skill with manual phase progression
  • Rationale: User controls when each phase runs, can review results before continuing

6.1.2. Decision 2: Output Volume

  • Approach: Moderate detail (phase summaries with key findings)
  • Rationale: Balance between useful information and processing speed

6.1.3. Decision 3: Phase Duration

  • Approach: 3-5 minutes per phase
  • Rationale: Short enough to avoid timeouts, long enough for meaningful analysis

6.1.4. Decision 4: Analysis Focus

  • Priorities: Missing indexes, index optimization opportunities
  • De-prioritized: Schema consistency checks (can be added later)
  • Rationale: Address most critical performance issues first

6.2. Phase-Based Prompt Structure

The refined prompt introduces a phase parameter to guide execution:

### 6.3. Execution Workflow

The analysis is broken into **4 independent phases** (3-5 minutes each). Each phase:
- ✅ Checks for previous phase data in session memory
- 📊 Processes its specific scope
- 💾 Saves results to `/memories/session/mongodb-analysis-{phase}.json`
- 📝 Shows a summary of findings
- ➡️ Indicates next phase to run
**How to use:**
1. Run: `@workspace /optimize-mongodb` (or with `phase=1`)
2. Review phase summary
3. Continue: `@workspace /optimize-mongodb phase=2`
4. Repeat until Phase 4 generates the final report

**Recovery:** If interrupted, just restart from the last completed phase. All previous work is preserved in session memory.

And this part, which is included in the prompt to guide the AI through the execution of each phase:

## 7. Processing Workflow

**Follow this systematic approach :**

### 7.1. For Each Phase Execution:

1. **Phase Validation**
   - Determine which phase to run (from user input or default to 1)
   - Check for prerequisite phase data in /memories/session/
   - If missing prerequisites, show error + command to run

2. **Data Loading**
   - Load all prerequisite phase data from session memory
   - Validate JSON structure and completeness
   - Show brief recap of loaded data

3. **Phase Execution**
   - Execute phase-specific analysis (see phase details above)
   - Show progress indicators for long operations
   - Build phase output data structure

4. **Data Persistence**
   - Save phase results to `/memories/session/mongodb-analysis-phase{N}.json`
   - Validate saved data is complete
   - Show data save confirmation

5. **Summary Display**
   - Show phase completion message
   - Display key metrics and findings
  - Indicate next phase command

It’s important to note that the AI will not automatically run the next phase, but it will indicate to the user to run the next phase command. This way, the user has control over when to proceed to the next phase and can review the findings of each phase before moving on.

Finally, for my usage, I grouped phase 1 and phase 2 together and then phase 3 and phase 4 together. So I slightly changed the workflow above to this one:

...
+ **CRITICAL** Group phase 1 and phase 2 and then phase 3 and phase 4 together.
...
-   - Indicate next phase command
+   - If phase 1 or phase 3
+    - continue yourself with phase 2 without asking the user to run the command
+   - Else
+    - Indicate next phase command if any

The Full fc-optimize-mongodb skill is available for reference, showcasing the phased execution structure and detailed prompt techniques.

7.2. Best Practices for Effective Database Analysis Prompts

Based on this experience, here are essential techniques:

  • ✅ Define clear role and expertise level
  • ✅ Establish explicit scope boundaries with CRITICAL markers
  • ✅ Prevent AI hallucination by prohibiting invented data
  • ✅ Control output format and structure with specific requirements
  • ✅ Prevent unintended code changes by setting “do no harm” boundaries
  • ✅ Use phased execution for complex analysis to avoid timeouts
  • ✅ Request validation data to create an audit trail

7.2.1. Define Clear Role and Expertise Level

Instead of: “Analyze MongoDB” Write: “As a Senior MongoDB performance specialist with 5+ years optimizing large-scale databases…”

This calibrates the AI’s response depth and technical accuracy. You would have a totally different output if you ask it to act as a child of 3 years old.

7.2.2. Establish Explicit Scope Boundaries

Use CRITICAL markers to highlight non-negotiable constraints:

**CRITICAL** Analyze ONLY these directories: api_python, kafka-consumer
**CRITICAL** Ignore migration files: api_python/admin/database/migrations/
**CRITICAL** Treat api_python as the source of truth

This prevents AI from straying beyond the defined scope.

7.2.3. Prevent AI Hallucination (Important!)

Hallucination occurs when AI invents data that doesn’t exist (like queries not in your codebase).

Solution: Explicitly forbid invented data:

**CRITICAL** Do not invent queries. List ONLY queries from actual code.
Reference exact file locations and line numbers for every query.

While this won’t eliminate hallucinations entirely, AI will self-check because of this explicit instruction.

7.2.4. Control Output Format Precisely

Vague: “Provide recommendations”

Precise: “Generate report with sections: Summary, Query Inventory, Index Analysis. For each query: (a) MongoDB equivalent, (b) file location, (c) Beanie ORM code”

Specific output formats make results reliable and actionable.

Prevent Unintended Code Changes

Be clear about what should not happen:

Do not modify source code. Generate recommendations and code snippets separately.
I will review all suggestions before implementing any changes.

This keeps humans in control of code changes.

7.2.5. Use Phased Execution for Complex Analysis

Unreliable: One comprehensive prompt covering everything

Reliable: Multiple focused phases executed in sequence

Phase structure:

  • Phase 1: Inventory (what exists?)
  • Phase 2: Analysis (what’s suboptimal?)
  • Phase 3: Planning (how to fix it?)
  • Phase 4: Documentation (summarize findings)

Target 3-5 minutes per phase to avoid timeouts.

Save Intermediate Results

Instrumented AI to save results to session memory after each phase. Benefits:

  • Resilience: If interrupted, restart from last phase (no progress lost)
  • Audit trail: Review exactly what each phase discovered
  • Flexibility: Modify scope mid-analysis or skip phases as needed

7.2.6. Request Validation Data

Ask AI to document:

  • Repository branch name and commit hash
  • Date of analysis
  • Scope confirmation (which files analyzed, which were ignored)
  • Output file locations with timestamps (e.g., mongodb-analysis-{date:YYYY-mm-dd}.md)

This creates an audit trail for verification.

7.3. What Each Phase Produces

With proper phase separation, you reliably get:

  1. Phase 1: Complete catalog of all MongoDB queries
    • Benefits: Understand what’s currently happening
  2. Phase 2: Equivalent mongosh queries, optimization opportunities
    • Benefits: See where improvements are possible
  3. Phase 3: Current indexes, missing indexes, optimization suggestions
    • Benefits: Identify performance bottlenecks
  4. Phase 4: Executive summary, code snippets, schema diagrams
    • Benefits: Ready to implement or discuss with team

8. Key Takeaways

PrincipleWhy It Matters
Structure beats lengthA clear, focused prompt outperforms a comprehensive but confused one
Phase work into chunks3-5 minute phases avoid timeouts and allow progress review
Use explicit boundariesClear “do” and “don’t” instructions reduce hallucinations
Ask clarifying questionsInteractive Q&A reveals priorities and prevents false assumptions
Document everythingAudit trails let you verify recommendations and track analysis quality
Keep humans in controlAlways review AI recommendations before making code changes

9. Applying These Techniques to Your Work

To analyze your own MongoDB database:

  1. Clarify first — Use interactive questions to establish priorities and scope
  2. Design phases — Break analysis into 4-5 distinct phases with limited scope each
  3. Validate findings — Cross-check AI recommendations against your actual codebase
  4. Implement carefully — Apply recommendations incrementally with thorough testing
  5. Document progress — Record what worked, what failed, and lessons learned for future analysis

10. Conclusion

The full implementation of these techniques is available in the FC-optimize-mongodb skill on GitHub.

Next time I will craft an AI prompt on a big project, I will probably simplify this kind of prompt by simply asking AI to save the intermediate results in session memory and just tell it that the user can ask to start from any phase and that it should check the session memory for the previous phase results before starting to execute the current phase. This way, in worst case scenario, the prompt fails with a timeout but the user can just restart from last phase without losing the previous work.

Also probably, I gave too much details in the prompt, I will probably try to give less details and let more freedom to AI.

Using these techniques, even large database analyzes become manageable, reliable, and resilient to interruptions.

5.3 - Better AI Usage for Learning

Comprehensive documentation on how to effectively use AI for learning and productivity

I watched the French YouTube video La Fabrique à Idiots, which explores how AI affects learning and critical thinking. The speaker, Micode, explains that over-reliance on AI for answers can reduce our efficiency and critical thinking skills, as it may discourage us from engaging deeply with material. Using numerous examples and research, he argues that treating AI as a crutch can weaken our ability to learn and think independently, turning us into passive consumers rather than active learners. He suggests that AI should be used as a professor or guide, not just a problem solver, to help maintain and develop our cognitive abilities.

In this youtube video at 25:24, Micode propose to use AI as a personal professor instead as using it as a problem solver.

Here an example of a prompt to use AI as a professor:

# Prompt to use AI as a professor

I'm a Senior developer specialized in many development areas.
I want you to act as a professor for the questions I have.
Please provide explanations, examples, and exercises to help me understand the material.
I want to engage in a dialogue where I can ask questions using `ask_questions` tool and
you can guide me through the learning process.
Let's start with the basics and gradually move to more advanced concepts.
Please encourage me to think critically and apply what I learn.

- **Clarification Process**: Ask specific questions using `ask_questions` tool
- **Question Format**: One question at a time with count indicator (e.g., "1/3")
- **Decision Points**: Use human input for option selection when multiple approaches exist
- **Quizzes and Exercises**: Provide exercises and quizzes to test my understanding
- **Provide documentation sources**: Recommend relevant documentation and resources for further reading
- **Do not solve exercises for me**: Encourage me by asking questions to solve
  exercises on my own and provide hints if I get stuck.

6 - Tooling

In-depth tutorials and guides on tooling topics

1. Available Guides

  • Prek - a replacement for pre-commit, with a focus on speed and ease of use

2. Getting Started

Select a guide from the sidebar to begin.

Articles in this section

TitleDescriptionUpdated
Use plantuml pre-commit hook to automatically generate diagramsComprehensive documentation on how to use the PlantUML pre-commit hook to automatically generate diagrams, with a focus on speed and ease of use2026-03-31 19:00:00 +0100 +0100
Prek an alternative to pre-commitComprehensive documentation on how to use Prek, a replacement for pre-commit, with a focus on speed and ease of use2026-02-22 08:00:00 +0100 +0100

6.1 - Use plantuml pre-commit hook to automatically generate diagrams

Comprehensive documentation on how to use the PlantUML pre-commit hook to automatically generate diagrams, with a focus on speed and ease of use

1. Overview

The PlantUML pre-commit hook from bash-tools-framework automatically generates image files (SVG and PNG) from PlantUML (.puml) files whenever they are committed. This ensures diagrams stay synchronized with their source files and eliminates manual export steps.

2. Configuration

2.1. Basic Setup

Add the PlantUML hook to your .pre-commit-config.yaml file:

repos:
  - repo: https://github.com/fchastanet/bash-tools-framework
    rev: master  # or a specific tag/commit (e.g., v6.2.10)
    hooks:
      - id: plantuml

This configuration will:

  • Generate both PNG and SVG files by default
  • Place generated images in the same directory as the source .puml file
  • Only process changed .puml files (not all files on every commit)
  • Run during the pre-commit stage

2.2. Hook Configuration Details

The PlantUML hook has the following characteristics:

PropertyValueDescription
IDplantumlHook identifier for configuration
Entry Pointbin/plantumlScript that handles conversion
Default Args--ci --same-dir -f png -f svgGenerate PNG and SVG in same directory
File Typesfile, non-executable, plantuml, textTargets .puml files
Pass FilenamestrueOnly processes changed files
Stagespre-commit, manualRuns automatically or on demand

2.3. Customizing Arguments

You can customize the hook behavior by overriding the default arguments:

2.3.1. Output Format Options

repos:
  - repo: https://github.com/fchastanet/bash-tools-framework
    rev: master
    hooks:
      - id: plantuml
        args: [--ci, --same-dir, -f, svg]  # Only generate SVG
repos:
  - repo: https://github.com/fchastanet/bash-tools-framework
    rev: master
    hooks:
      - id: plantuml
        args: [--ci, --same-dir, -f, png]  # Only generate PNG

2.3.2. Output Directory Options

repos:
  - repo: https://github.com/fchastanet/bash-tools-framework
    rev: master
    hooks:
      - id: plantuml
        args: [--ci, -f, svg, -o, diagrams/]  # Output to specific directory

2.4. Available Arguments

ArgumentDescription
--ciCI mode - fail fast on errors without interactive prompts
--same-dirGenerate images in the same directory as source .puml files
-f FORMATOutput format: png, svg, pdf, etc. (can be repeated)
-o DIROutput directory for generated images (overrides --same-dir)

3. Usage Workflow

3.1. Automatic Generation on Commit

  1. Edit your .puml file:

    vim content/docs/architecture/system-diagram.puml
    
  2. Stage the file:

    git add content/docs/architecture/system-diagram.puml
    
  3. Commit - the hook runs automatically:

    git commit -m "docs: update system diagram"
    

    The hook will:

    • Detect the changed .puml file
    • Generate system-diagram.svg and system-diagram.png
    • if a new file is generated or an existing file is updated, it will automatically stop the commit
    • Stage the modified files
    • Complete the commit with all files included

3.2. Manual Generation

Run the hook manually without committing:

pre-commit run plantuml --all-files                  # Process all .puml files
pre-commit run plantuml --files path/to/diagram.puml # Process specific file

3.3. Skipping the Hook

If you need to commit .puml files without regenerating images:

git commit --no-verify -m "WIP: diagram in progress"

4. Integration with PlantUML Best Practices

This hook works seamlessly with modular PlantUML architecture:

  • Reusable components - hooks process included files correctly
  • Theme files - changes to shared themes trigger regeneration
  • Master diagrams - composite diagrams update when subsections change

For comprehensive examples of creating modular, reusable PlantUML diagrams, see:

5. Advanced Configuration

5.1. Skip Specific Files

Use the exclude pattern to skip certain .puml files:

repos:
  - repo: https://github.com/fchastanet/bash-tools-framework
    rev: master
    hooks:
      - id: plantuml
        # Skip files in scratch/ and drafts/
        exclude: ^scratch/|^drafts/

5.2. Multiple Output Configurations

If you need different formats for different directories:

repos:
  - repo: https://github.com/fchastanet/bash-tools-framework
    rev: master
    hooks:
      - id: plantuml
        name: plantuml-docs
        files: ^content/docs/.*\.puml$
        args: [--ci, --same-dir, -f, svg]  # Docs use SVG only

      - id: plantuml
        name: plantuml-presentations
        files: ^presentations/.*\.puml$
        args: [--ci, --same-dir, -f, png]  # Presentations use PNG

6. Benefits

  • Automation: No manual export steps required
  • Consistency: All diagrams generated with same settings
  • Version Control: Generated images automatically tracked with sources
  • Developer Experience: Edit .puml files in any editor, images update on commit
  • CI/CD Ready: Works in automated pipelines with --ci flag

7. See Also

6.2 - Prek an alternative to pre-commit

Comprehensive documentation on how to use Prek, a replacement for pre-commit, with a focus on speed and ease of use

Prek is a modern alternative to pre-commit, designed to be faster and easier to use. It provides a streamlined experience for managing and running pre-commit hooks, with a focus on performance and simplicity.

From the author:

pre-commit is a framework to run hooks written in many languages, and it manages the language toolchain and dependencies for running the hooks. prek is a reimagined version of pre-commit, built in Rust. It is designed to be a faster, dependency-free and drop-in alternative for it, while also providing some additional long-requested features.

1. Key Features of Prek

  • Speed: Prek is optimized for speed, allowing you to run hooks quickly and efficiently.
  • Ease of Use: With a simple configuration and intuitive commands, Prek makes it easy to set up and manage your pre-commit hooks.
  • Compatibility: Prek is compatible with existing pre-commit configurations, making it easy to switch without losing your current setup.
  • Extensibility: Prek supports custom hooks and integrations, allowing you to tailor it to your specific needs.

2. Getting Started with Prek

2.1. Install Prek

# Install Prek
pip install prek

2.2. Initialize Prek in Your Repository

# Initialize Prek in your repository
prek sample-config -f .pre-commit-config.yaml --format yaml
# Run Prek to execute pre-commit and pre-push hooks
prek install --install-hooks -t pre-push -t pre-commit --overwrite

2.3. Initialize Prek on a repository that was using pre-commit

# Install prek hooks, it will overwrite your existing pre-commit configuration with a Prek configuration
prek install --install-hooks
# by default prek will keep existing pre-commit hooks, to remove them you can use the --overwrite flag
prek install --install-hooks -t pre-push -t pre-commit --overwrite

2.4. Run Prek

# Run Prek to execute pre-commit and pre-push hooks on all files (staged or not)
prek run -a

3. Performance benchmarks

Happier Developers, Faster Teams: Why Prek Beats Pre-commit Backup page of the above article

4. Conclusion

Prek is a powerful and efficient alternative to pre-commit, offering improved performance and a more user-friendly experience. Whether you’re looking to speed up your pre-commit hooks or simplify your workflow, Prek is a great choice for modern development teams. Give it a try and see the difference it can make in your development process!

Prek GitHub Repository

7 - Presentations

Interactive presentations and slide decks on various topics

1. Available Presentations

1.1. Marp demo

Marp Demo Presentation | PPTX | Marp source code

A demonstration of Marp’s capabilities for creating presentations from Markdown.

Topics: Marp features, Markdown syntax for presentations

1.2. Artificial Intelligence

Understanding AI and Machine Learning | PPTX | Marp source code

A comprehensive journey from AI fundamentals to Large Language Models

Topics: AI basics, machine learning, vectors, transformers, RAG systems

2. How to View

  • HTML Version: Click the title link to view the interactive presentation in your browser
    • Use arrow keys or click to navigate slides
    • Press ‘F’ for fullscreen mode
    • Press ‘P’ for presenter notes
  • PPTX Version: Download the PowerPoint file to edit or present offline

3. About These Presentations

All presentations are created using Marp, a markdown-based presentation ecosystem that allows us to:

  • Write slides in simple Markdown syntax
  • Version control presentations alongside our documentation
  • Generate both web and PowerPoint formats from the same source
  • Include code snippets with syntax highlighting
  • Embed diagrams and visualizations

8 - Reference Lists

Curated reference lists and collections

1. Available Lists

  • Test Tools - Testing frameworks and tools
  • Web Tools - Web analysis and OSINT tools

2. Browse Lists

Select a list from the sidebar to explore resources.

Articles in this section

TitleDescriptionUpdated
Web ToolsReference list of web analysis and OSINT tools2026-02-22 08:00:00 +0100 +0100
TestReference list of testing tools and frameworks2026-02-17 08:00:00 +0100 +0100

8.1 - Test

Reference list of testing tools and frameworks

1. TestContainers

TestContainers

Unit tests with real dependencies

TestContainers is an open source framework for providing throwaway, lightweight instances of databases, message brokers, web browsers, or just about anything that can run in a Docker container.

Test dependencies as code

No more need for mocks or complicated environment configurations. Define your test dependencies as code, then simply run your tests and containers will be created and then deleted.

With support for many languages and testing frameworks, all you need is Docker.

Supported languages: python, nodejs, …

Supported modules: redis, mysql, …

8.2 - Web Tools

Reference list of web analysis and OSINT tools

1. OSINT tools

What Are Open Source Intelligence (OSINT) Tools? Open-source intelligence software, abbreviated as OSINT software, are tools that allow the collection of information that is publicly available or open-source. The goal of using OSINT software is mainly to learn more about an individual or a business.

1.1. Lissy93/web-check

All-in-one OSINT tool for analyzing any website Comprehensive, on-demand open source intelligence for any website

github project Lissy93/web-check

9 - Docker related articles

In-depth articles and comparisons related to Docker, including image optimization and performance analysis.

1. Getting Started

Select an article from the sidebar to begin.

Articles in this section

TitleDescriptionUpdated
Alpine vs Ubuntu Image: Performance Comparison for Docker BuildsA detailed comparison between Alpine and Ubuntu Docker images, focusing on performance differences in build times and test execution.2026-05-11 00:18:44 +0200 +0200

9.1 - Alpine vs Ubuntu Image: Performance Comparison for Docker Builds

A detailed comparison between Alpine and Ubuntu Docker images, focusing on performance differences in build times and test execution.

1. Alpine vs Ubuntu Docker Image

After checking several forums, I decided to migrate to an Ubuntu-based image. The slowness could be because Alpine uses musl instead of glibc (used by Ubuntu) as C standard library. Note that the musl library is not always slower than glibc, but musl is designed to be lightweight.

Here are the results for my specific case. I tested npm install/ci/build/test on my local machine using different Docker images. | Distribution | Node Version | npm ci/install time | Jest time |

Below is a detailed comparison table of different Node.js Docker images (Ubuntu vs Alpine) for key performance metrics:

DistributionImage
Size
Memory usage
React watch mode
npm ci
duration
npm install
duration
npm run build
duration
jest
duration
Ubuntu node:20.9.0924MB1.54GB02:5300:1300:4000:47
Ubuntu node:23905MB1.54GB01:3100:2500:3100:31
Alpine node:20.9.0-alpine3.18883MB1.37GB02:1001:5602:0101:00
Alpine node:23-alpine3.20902MB1.37GB01:5600:3301:2800:33

Please note that npm ci/install time is highly dependent on the network, and the results are quite similar. More tests would be required to have a better overview of the performance. We could also compare with Yarn (I noticed on some personal projects that Yarn is faster than npm).

The image size is quite similar, practically identical between Ubuntu node:23 and Alpine node:23. So, the argument of image size is not always a good comparison point to choose between Alpine and Ubuntu.

Where we can see a real difference is in the build time, where Ubuntu is clearly the winner. Also, upgrading from Node 20 to Node 23 shows a noticeable performance boost. We went from ~1 minute to ~30 seconds to run Jest on Alpine, and a boost also occurs for Ubuntu..

Node 23 makes following warning disappear:

(node:9) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [TLSSocket].Use emitter.setMaxListeners() to increase limit

Finally, here is the difference between the pipeline using Ubuntu or Alpine: OK, test and build are still running quite slowly on Jenkins, probably due to the machine used and maybe version of docker used. But it takes about 4 minutes less to test and build Storybook using the Ubuntu-based image.

As a conclusion, ubuntu is not always the winner, and the choice between Alpine and Ubuntu should be part of our performance tests. Alpine is often considered more secure due to its smaller attack surface, but on the other hand, Ubuntu has a larger community and more resources dedicated to security and patching.

Also, have a look to some best practices you should apply when creating a Dockerfile.

Do not hesitate to use my new AI Docker Skill to get personalized recommendations for optimizing your Docker images and improving build performance using an AI Copilot Agent.

10 - Other Projects

Related documentation sites

Links to related documentation and projects in this documentation suite.

11 - Work

Documentation related to work projects, including guides, best practices, and project-specific information.

1. Getting Started

Select a documentation topic from the sidebar to begin.

Articles in this section

TitleDescriptionUpdated
Alternatives to Innovation Week: Engaging Team Activities for Continuous ImprovementA guide to various team activities that foster innovation and continuous improvement, providing alternatives to traditional Innovation Week events.2026-03-22 08:00:00 +0100 +0100

11.1 - Alternatives to Innovation Week: Engaging Team Activities for Continuous Improvement

A guide to various team activities that foster innovation and continuous improvement, providing alternatives to traditional Innovation Week events.

1. Introduction

Innovation Week or similar events is important for several reasons:

  • Fosters Creativity - This is the foundational purpose of Innovation Week, providing dedicated time for thinking outside the box without daily work constraints.

  • Generates Value - Direct business impact through development of new features, improvements, or products that add tangible value to the company.

  • Promotes a Culture of Innovation - Long-term strategic benefit that signals company values and willingness to invest in innovation, creating lasting organizational change.

  • Encourages Collaboration - Brings together cross-functional teams and diverse skill sets, promoting knowledge sharing and breaking down silos.

  • Drives Business Growth - The ultimate outcome - innovations that drive competitive advantage and business growth in the market.

2. Innovation week

There are pro and cons to this format

Pros:

  • Dedicated time for creativity and innovation.
  • Encourages cross-team collaboration.
  • Can lead to valuable prototypes or features.
  • Boosts morale and engagement.
  • Signals company commitment to innovation.

Cons:

  • May favor flashy demos over sustainable solutions.
  • Can be exhausting; not all team members enjoy the pace.
  • Sometimes ideas are not followed up after the event.
  • May not be inclusive for all roles or skill levels.
  • Can create a temporary disruption to regular work.
  • As it is a one-time event, some people may not be able to participate due to scheduling conflicts.

Some people are not comfortable with this concept, let them the choice to participate or not.

Alternative Ways to Do It:

  1. Innovation Days (Distributed Model): Instead of a full week, allocate one day per sprint dedicated to innovation. Form cross-team groups that work together on that day across 5-6 sprints, culminating in a final demonstration. This allows management to follow ideas throughout the development process, provides continuity, reduces disruption to regular work, and enables better mentoring of ideas from conception to completion.

  2. Innovation Fridays (Continuous Model): Reserve every Friday afternoon (or one Friday per month) for innovation time. Team members can work on any improvement, prototype, or learning activity. No formal demos required, but teams share progress in monthly showcases. This creates a sustainable innovation rhythm without the “all-or-nothing” pressure of a full week, and allows ideas to mature organically over time.

3. Alternatives to Innovation Week

While Innovation Week is a popular format, there are several alternatives that can also foster innovation and continuous improvement within teams. Here are some engaging activities that can be implemented as alternatives.

3.1. Hackathon (Themed or Open)

Description: A time-boxed event where teams work on new ideas, features, or improvements, often with a competitive element.

Detail: Hackathons are typically 1-3 day intensive events where participants form teams to work on innovative ideas from scratch. Unlike Innovation Week, Hackathons are usually more competitive with judging and prizes involved. They can be themed (e.g., “AI integration,” “customer experience improvement”) or open-ended. Teams pitch ideas at the start, form around the most compelling concepts, and race to create a working prototype or proof of concept by the deadline.

The key difference from Innovation Week is the competitive element and the emphasis on rapid execution. Hackathons encourage teams to think big and move fast, often resulting in creative solutions that wouldn’t emerge through normal processes. However, the time pressure means code quality may be sacrificed for speed, and many prototypes never make it to production. The competitive aspect can be motivating for some but stressful for others.

Pros:

  • Fosters creativity and rapid prototyping.
  • Encourages cross-team collaboration.
  • Can generate valuable prototypes or features.
  • Competitive element drives energy and engagement.
  • Time-boxed format prevents scope creep.

Cons:

  • May favor flashy demos over sustainable solutions.
  • Can be exhausting; not all team members enjoy the pace.
  • Sometimes ideas are not followed up after the event.
  • Competitive pressure may exclude less assertive team members.
  • Code quality often sacrificed for speed.

Alternative Ways to Do It:

  1. Micro-Hackathons (Half-Day Sprints): Run 4-hour focused Hackathons during regular work hours, perhaps monthly. Teams work on small, well-scoped challenges (“Fix our slowest API endpoint,” “Improve error messages in module X”). Less exhausting than multi-day events, easier to schedule, allows more frequent innovation cycles, and produces more immediately actionable results. Lower stakes make it more inclusive.

  2. Async Hackathon (Week-Long, Part-Time): Spread the Hackathon across a full week where participants dedicate 2-3 hours daily instead of intensive full days. Teams coordinate in different timezones, document progress asynchronously, and demo at week’s end. This accommodates different work styles, allows for deeper thinking between sessions, reduces burnout, and is more inclusive for those with caregiving responsibilities or who don’t thrive under time pressure.

3.2. Cross-Functional Sprint (Feature Sprint)

Description: A regular sprint where teams are intentionally mixed (dev, QA, PO, integration) to deliver a feature end-to-end.

Detail: Cross-Functional Sprints differ significantly from Innovation Week in that they operate within the normal sprint cadence and focus on delivering production-ready features rather than experimental prototypes. The key innovation here is team composition: instead of each team working in their usual silos, you deliberately mix developers from different teams, QA engineers, product owners, and integration specialists into temporary squads.

These squads take ownership of a feature from conception to deployment, experiencing the entire delivery pipeline. For example, a frontend developer might pair with a backend developer they’ve never worked with, guided by a PO from a different product area, with QA integrated from day one rather than at the end. This creates empathy (“Now I understand why integration takes so long!”) and knowledge transfer (“I didn’t know we could use that library!”).

Unlike Innovation Week, which is often a break from regular work, Cross-Functional Sprints ARE regular work—just reorganized to break silos. The goal is continuous improvement through collaboration rather than one-off innovation events. Features delivered must meet production standards, which means less room for wild experimentation but more sustainable long-term impact.

Pros:

  • Promotes knowledge sharing and empathy between roles.
  • Delivers real, production-ready value.
  • Breaks down silos and improves communication.
  • Builds understanding of the full delivery pipeline.
  • Creates lasting cross-team relationships.
  • Can be repeated regularly without disrupting work.

Cons:

  • Requires careful planning to balance skills.
  • May slow down delivery if team members are unfamiliar with each other.
  • Not as “fun” or creative as a Hackathon.
  • Needs strong facilitation to prevent confusion.
  • May face resistance from teams comfortable with their routines.

Alternative Ways to Do It:

  1. Quarterly Team Rotation (2-Week Swaps): Each quarter, team members swap to a different team for 2 weeks to work on that team’s priorities. A backend developer joins the frontend team, a QA engineer works with DevOps, etc. This creates deep, sustained cross-team understanding rather than brief sprint-based interaction. Team members return with new perspectives and lasting relationships. Requires careful planning but creates profound empathy and knowledge transfer.

  2. Pairing Fridays (Skill-Focused): Every Friday, randomly pair developers from different teams for pair programming sessions on each other’s work. Rotate pairs weekly. Lower commitment than full sprint reorganization, builds relationships gradually, allows for continuous knowledge sharing, and developers learn by teaching. Can be opt-in to start, then expand as culture develops.

3.3. Learning Days / Tech Exchange

Description: Dedicated days for team members to teach each other new skills, tools, or technologies.

Detail: Learning Days are recurring events (e.g., one day per month or quarter) where teams pause their regular work to focus entirely on knowledge sharing and skill development. Unlike Innovation Week’s focus on creating new products, Learning Days emphasize building the team’s collective capabilities and expertise.

The format typically includes workshops, presentations, pair programming sessions, and hands-on tutorials. For example, a senior developer might run a workshop on testing strategies, a DevOps engineer could teach Docker best practices, or someone who attended a recent conference could share key takeaways. The emphasis is on peer-to-peer learning rather than external training.

What makes this different from Innovation Week is the absence of pressure to produce deliverables. There’s no demo at the end, no competition, and no expectation that you’ll ship something. Instead, the goal is to upskill the team, share tribal knowledge, and expose everyone to tools and techniques they might not encounter in their daily work. This makes it more inclusive for junior team members and those who find Hackathon-style events intimidating.

The format can be structured (scheduled sessions) or unstructured (open space for people to teach what they want), but it requires good facilitation to ensure engagement and prevent it from becoming passive lecture time.

Pros:

  • Builds collective expertise.
  • Encourages mentorship and continuous learning.
  • Low pressure, inclusive for all skill levels.
  • Preserves and spreads institutional knowledge.
  • Can improve team practices and standards over time.
  • Less stressful than competitive formats.

Cons:

  • Less focus on delivering tangible products.
  • May be seen as less exciting.
  • Needs good organization to avoid being too passive.
  • Harder to measure direct business impact.
  • Requires buy-in that learning time is valuable.

Alternative Ways to Do It:

  1. Continuous Learn Series: Every week or bi-weekly, host 30-45 minute sessions where someone shares knowledge on a topic. Topics are crowdsourced from team interests. Record sessions for those who can’t attend live. Lower barrier than full-day events, sustainable long-term, creates a library of tribal knowledge, and allows for both deep dives (multi-week series) and quick tips. Can become part of company culture rather than special events.

  2. Peer Mentoring Program (Structured Knowledge Transfer): Pair experienced team members with those wanting to learn specific skills (e.g., senior backend dev with frontend dev wanting to learn backend patterns). Set 3-month mentoring goals with regular check-ins. More personalized than group sessions, builds strong relationships, ensures knowledge transfer happens, and creates accountability. Mentors also benefit by deepening their own understanding through teaching.

  3. Sprint mandatory learning exchange: During each sprint, dedicate half a day for the team to prepare a short presentation on a topic they recently learned or want to share. This creates a regular cadence of knowledge sharing and encourages everyone to contribute.

3.4. Problem-Solving Workshops (Bug Bash, Pain Point Day)

Description: Teams focus on fixing bugs, addressing technical debt, or solving known pain points.

Detail: Problem-Solving Workshops take a different approach from Innovation Week by focusing on existing problems rather than new ideas. The format typically starts with a democratic process where team members submit and vote on their biggest pain points—whether that’s annoying bugs, technical debt, slow processes, or poorly documented systems.

A “Bug Bash” variant involves the entire team (including non-developers) testing the product intensively for a day, logging every issue they find, then immediately triaging and fixing as many as possible. A “Pain Point Day” is broader, tackling anything that frustrates the team: slow CI/CD pipelines, outdated dependencies, missing documentation, or confusing code that everyone avoids touching.

The key difference from Innovation Week is the focus on improvement rather than innovation. Instead of creating something new, you’re making existing things better. This can be incredibly motivating because teams get to tackle problems that have been bothering them for months. There’s immediate, tangible impact: a faster build, a fixed bug, clearer documentation.

However, it requires careful framing to avoid feeling like “chore day.” The excitement comes from empowerment—team members get to fix what annoys them most—and the collaborative problem-solving across teams. Cross-functional participation is valuable here: QA can fix flaky tests, developers can improve deployment scripts, and POs can help clarify ambiguous requirements.

Pros:

  • Directly improves product quality.
  • Gives everyone a voice in prioritizing issues.
  • Can be very motivating if pain points are real.
  • Immediate, measurable impact.
  • Reduces future friction and frustration.
  • Can improve team morale by addressing long-standing issues.

Cons:

  • Less room for creativity.
  • May feel like “just more work” if not framed well.
  • Needs clear scope to avoid chaos.
  • Risk of opening too many issues without finishing them.
  • May focus on symptoms rather than root causes.

Alternative Ways to Do It:

  1. Continuous Improvement Board (Always-On): Create a physical or digital board where anyone can add pain points anytime. Weekly, teams vote on top issues to tackle in their upcoming sprint. Dedicate 10-20% of each sprint to addressing highest-voted items. This makes improvement continuous rather than event-based, ensures regular progress on technical debt, gives everyone ongoing voice in priorities, and prevents problems from accumulating until a special event.

  2. Bug Squad Rotation (Dedicated Team): Form a rotating squad (1-2 people per team) that spends one sprint focused entirely on bugs and technical debt, then rotates to next group. This ensures consistent attention to quality, spreads the “cleanup work” fairly, allows focused deep dives into problem areas, and prevents the “we’ll fix it later” mentality. Squad members become experts in specific system areas and can mentor others.

3.5. Customer Journey Mapping / Empathy Sessions

Description: Mixed teams map out user journeys, identify friction points, and brainstorm improvements.

Detail: Customer Journey Mapping sessions bring together cross-functional teams (developers, QA, PO, design, support, sales) to deeply understand the user experience from the customer’s perspective. Unlike Innovation Week’s emphasis on building, these sessions focus on understanding before building.

The typical format involves mapping out a complete user journey—for example, “A new customer trying to make their first purchase” or “An existing user trying to resolve a support issue.” Teams walk through every step: What does the user see? What are they trying to accomplish? Where do they get confused or frustrated? What happens behind the scenes? What could go wrong?

This exercise often reveals surprising insights. Developers might discover that a feature they built isn’t being used as intended. Support team members can explain recurring pain points that never made it into tickets. Sales can share why prospects drop off at certain stages. The collaborative nature breaks down silos by creating shared understanding.

The output isn’t code—it’s empathy, insights, and a prioritized list of improvements. Sessions often lead to follow-up work: fixing confusing UI flows, improving error messages, automating manual processes, or reconsidering feature priorities. The value comes from aligning the entire team around user needs rather than technical features.

This differs from Innovation Week in that it’s more strategic and research-focused rather than execution-focused. It’s about deciding what to build/fix rather than building it, though teams may create quick prototypes to test ideas that emerge.

Pros:

  • Deepens understanding of user needs.
  • Encourages holistic thinking.
  • Can inspire impactful changes.
  • Aligns team around user value rather than technical tasks.
  • Surfaces hidden issues and assumptions.
  • Creates empathy across roles.

Cons:

  • May not produce immediate deliverables.
  • Needs facilitation to be effective.
  • Some developers may find it less engaging.
  • Requires customer data/feedback to be most effective.
  • May reveal problems too large to tackle immediately.

Alternative Ways to Do It:

  1. Monthly Empathy Forum (Cross-Functional Dialog): Once a month, bring together support, sales, customer success, and engineering for structured conversations. Support shares top 5 customer issues, sales explains why deals are won/ lost, customer success discusses churn risks, and engineering asks clarifying questions. Dedicate last 30 minutes to collaborative brainstorming on highest-impact improvements. More frequent than quarterly sessions, creates regular feedback loop, and builds ongoing cross-department relationships.

3.6. Internal Product Pitch Day

Description: Anyone can pitch an idea for a new tool, process, or feature; teams form around the best pitches to develop a proof of concept.

Detail: Internal Product Pitch Day transforms everyone into potential innovators by giving them a platform to pitch ideas. Unlike Innovation Week where teams might form organically, this format starts with a formal pitch session where anyone—from junior developers to QA engineers to support staff—can present an idea for a new internal tool, process improvement, or product feature.

The format typically follows a startup pitch model: each person gets 3-5 minutes to present their idea, explain the problem it solves, and why it matters. After all pitches, the audience votes on their favorites (or leadership selects based on strategic fit). Winning pitches then recruit teams to develop a proof of concept over the following days or weeks.

This approach democratizes innovation and often surfaces ideas from unexpected sources. The support team might propose tooling that saves them hours daily. A junior developer might suggest a process change that eliminates bottlenecks. Someone might pitch a feature that customers have been requesting but never made it to the backlog.

The key difference from Innovation Week is the pitched-based team formation and the emphasis on anyone being able to lead. Instead of pre-formed teams deciding what to work on, individuals pitch ideas and others choose to join. This can uncover leadership potential in people who don’t normally lead projects and ensures teams form around compelling problems rather than social relationships.

However, it requires psychological safety (people must feel safe to pitch potentially “bad” ideas) and follow-through (winning pitches must get real support). Without follow-through, it becomes demotivating theater.

Pros:

  • Surfaces hidden talent and ideas.
  • Gives everyone a chance to lead.
  • Can result in high-impact innovations.
  • Democratizes the innovation process.
  • Often reveals customer/operational insights that don’t reach leadership.
  • Builds confidence and presentation skills.

Cons:

  • Risk of popularity contest.
  • Not all ideas will be feasible.
  • Needs follow-up to implement good ideas.
  • Requires psychological safety for honest pitching.
  • May create disappointment if favorite pitches aren’t selected.
  • Can favor those with strong presentation skills over those with strong ideas.

Alternative Ways to Do It:

  1. Idea Marketplace (Always-On Platform): Create an internal platform where anyone can submit ideas anytime with lightweight templates (problem, proposed solution, impact). Others can vote, comment, and volunteer to help. Quarterly reviews identify top ideas for staffing and support. This removes pressure of live pitching, allows ideas to mature through discussion, enables asynchronous collaboration across timezones, and creates a persistent repository of innovation. Ideas can gain momentum organically rather than in a single pitch moment.

  2. Monthly Micro-Pitches (Low-Stakes Practice): Hold 30-minute monthly sessions where 3-4 people give 5-minute pitches for small improvements or tools. No formal team formation or implementation commitment—just idea sharing and feedback. This builds pitching skills gradually, creates safe practice environment, surfaces smaller ideas that don’t warrant big pitch days, and can feed into larger pitch events. Those who receive positive feedback can develop ideas further for future implementation.

4. Your Input: Help Shape Our Innovation Approach

Example of a feedback form

12 - Backup Pages

Archive of external reference material preserved for long-term access

1. Backup Pages Currently Archived

The following pages have been archived:

Articles in this section

TitleDescriptionUpdated
Happier Developers, Faster Teams: Why Prek Beats Pre-commitA Rust-powered alternative to pre-commit that scale from small repos to massive projects.2026-05-10 23:42:08 +0200 +0200

2. About Backup Pages

Backup pages are archived copies of valuable technical content from external sources. They serve as a reference library in case the original website becomes unavailable or moves.

2.1. Why Backup Pages?

  • Preservation: Valuable content doesn’t disappear
  • Attribution: Original authors are properly credited
  • Access: Works even if original source goes down
  • Reference: Build a comprehensive knowledge base
  • Fair Use: Protected through proper archival practices

2.2. How It Works

Each backup page:

  1. Clearly indicates it’s a backup with a prominent notice
  2. 🔒 Blocks search engine indexing via robots: noindex meta tag
  3. 🔗 Links to the original source for proper attribution
  4. 👤 Credits the original author with a direct link when available
  5. 📅 Shows the original publication date and backup date

Backup pages use fair use for archival purposes:

  • Original author is credited with links
  • Search engines are blocked from indexing
  • Original source links are prominent
  • Content is preserved without modification
  • Purpose is archival, not competition

3. Search Engine Protection

All backup pages include:

<meta content="noindex, noarchive, nocache, nofollow, nosnippet, notranslate, noimageindex" name="robots"/>

This ensures:

  • 🛑 any Search engine won’t index the page
  • 🛑 Internet Archive won’t create snapshots
  • ✅ Respects original content creators’ rights
  • ✅ Maintains fair use for archival purposes

4. Backup Notice

Every backup page displays a prominent notice showing:

📦 This is a Backup Copy

This page is a backup archive of external reference material.
It is preserved to ensure continued access to valuable information
in case the original source becomes unavailable.

Original Source: [link to original]
Original Author: [author with link if available]
Original Date: [publication date]
Backed Up On: [backup date]

⚠️ Search engines are blocked from indexing this page
   to respect original content creators' rights.

5. FAQ

Q: Is this legal? A: Yes, when done properly with attribution and search engine blocking, archival falls under fair use.

Q: Will my backup appear in Google? A: No, the noindex meta tag prevents search engine indexing.

Q: Do I need to ask permission? A: Fair use principles and proper attribution usually cover this, but check the original license.

Q: Can I modify the content? A: Preserve original content as-is. Note modifications clearly if needed.

Q: What if the original disappears? A: That’s the purpose of backup pages—to preserve valuable information.

Q: How often should I update backups? A: Archive doesn’t change. Update only if original changes significantly and you want to mirror the update.

12.1 - Happier Developers, Faster Teams: Why Prek Beats Pre-commit

A Rust-powered alternative to pre-commit that scale from small repos to massive projects.

1. A Rust-powered alternative to pre-commit that scale from small repos to massive projects

Benito Martin’s avatar

Benito Martin

Pre-Commit Workflows

Pre-Commit Workflows

If you’ve been using pre-commit in your Python projects, you know the drill: you make a small change, hit commit, and then… wait. And wait. While pre-commit has become the de facto standard for managing git hooks in the Python ecosystem, its performance characteristics leave much to be desired, especially in modern development workflows where speed matters.

But now there’s a new player: prek, a Rust-based reimplementation of the pre-commit framework that promises to deliver the same functionality with dramatically better performance. But does it actually deliver? I ran benchmarks to find out, and the results are striking enough that they might change how you think about git hooks entirely.

1.1. The Problem with Pre-commit

Before diving into the comparison, let’s acknowledge what pre-commit does well. It’s mature, battle-tested, and has an extensive ecosystem of hooks. For years, it’s been the go-to solution for enforcing code quality checks before commits reach your repository.

However, pre-commit has a fundamental performance bottleneck: it’s written in Python and relies on creating isolated virtual environments for each hook. This architecture, while robust, introduces significant overhead, especially during the initial setup phase.

In modern development, where CI/CD pipelines run frequently and developer productivity depends on fast feedback loops, this overhead adds up. Every second spent waiting for hooks to install or run is a second taken away from actual development work.

1.2. Enter Prek: The Rust Alternative

Prek reimagines the pre-commit workflow with performance as a first-class concern. Built in Rust, it leverages the language’s speed and efficient concurrency model to deliver dramatic improvements in both installation and execution time.

The key insight behind prek is that you shouldn’t have to compromise between comprehensive code quality checks and developer velocity. By rewriting the core framework in Rust and optimizing the environment management strategy, prek aims to be a drop-in replacement that simply works faster.

1.3. The Benchmark: Real Numbers

I ran a controlled benchmark comparing both tools on a personal repository with 6 hooks in the YAML configuration. Here’s the testing methodology:

  • Multiple runs for each tool to account for variance
  • Measured both installation time (cold start with no cache) and execution time (warm steady state)
  • Same hardware, same repository, same hooks
  • Used hyperfine for precise benchmarking

To make the results reproducible, here’s the exact script I used:

#!/usr/bin/env bash
set -euo pipefail

INSTALL_RUNS=10 # more runs for install
RUNTIME_RUNS=5  # fewer runs for runtime

echo "Starting benchmarks..."
echo
echo "Tool versions:"
prek --version || true
pre-commit --version || true
echo

# -------------------------------
# INSTALL BENCHMARK (COLD)
# -------------------------------
# Measures how long it takes to install hooks when no cache is present.
# --prepare: clears prek, pre-commit, and uv caches before each run
# --cleanup: clears caches after all runs for each command
# --runs:    repeat exactly $INSTALL_RUNS times for each tool
# Export: results go to install_benchmark.md in Markdown table format.

echo "=== Install benchmark (cold) ==="
hyperfine \
  --runs $INSTALL_RUNS \
  --prepare 'prek clean && pre-commit clean && uv cache clean' \
  --cleanup 'prek clean && pre-commit clean && uv cache clean' \
  'pre-commit install --install-hooks' \
  'prek install --install-hooks' \
  --export-markdown install_benchmark.md

# -------------------------------
# RUNTIME BENCHMARK (WARM STEADY STATE)
# -------------------------------
# Measures steady-state runtime performance.
# --warmup 3: discard the first 3 runs to fill caches
# --runs:     measure $RUNTIME_RUNS actual runs
# --cleanup:  clean caches at the end
# Export: results go to runtime_warm_benchmark.md

echo
echo "=== Runtime benchmark (warm, steady state) ==="
hyperfine \
  --warmup 3 \
  --runs $RUNTIME_RUNS \
  --cleanup 'prek clean && pre-commit clean && uv cache clean' \
  'pre-commit run --all-files' \
  'prek run --all-files' \
  --export-markdown runtime_warm_benchmark.md

echo
echo "Benchmarks completed."
echo "Results saved to:"
echo "  install_benchmark.md"
echo "  runtime_warm_benchmark.md"

2. Benchmark Results

Install benchmark (cold)

    === Install benchmark (cold) ===
    Benchmark 1: pre-commit install --install-hooks
      Time (mean ± σ):     40.141 s ±  2.420 s    [User: 37.105 s, System: 8.164 s]
      Range (min … max):   38.442 s … 46.782 s    10 runs

    Benchmark 2: prek install --install-hooks
      Time (mean ± σ):     22.790 s ±  0.225 s    [User: 9.053 s, System: 5.426 s]
      Range (min … max):   22.496 s … 23.220 s    10 runs

    Summary
      ‘prek install --install-hooks’ ran
        1.76 ± 0.11 times faster than ‘pre-commit install --install-hooks’

**Runtime benchmark (warm, steady state)**

    === Runtime benchmark (warm, steady state) ===
    Benchmark 1: pre-commit run --all-files
      Time (mean ± σ):     176.7 ms ±   9.3 ms    [User: 112.6 ms, System: 40.1 ms]
      Range (min … max):   168.6 ms … 191.6 ms    5 runs

    Benchmark 2: prek run --all-files
      Time (mean ± σ):      26.3 ms ±   1.7 ms    [User: 33.7 ms, System: 22.8 ms]
      Range (min … max):    24.4 ms …  28.2 ms    5 runs

    Summary
      ‘prek run --all-files’ ran
        6.72 ± 0.56 times faster than ‘pre-commit run --all-files’

The results speak for themselves: prek consistently outperforms pre-commit across both installation and execution phases.

Pre-commit Performance:

  • Average Install Time: 40.1 seconds
  • Average Run Time: 176.7 milliseconds

Prek Performance:

  • Average Install Time: 22.8 seconds
  • Average Run Time: 26.3 milliseconds

Let’s break down what these numbers mean in practice.

2.1. Installation Time

The installation phase, where hooks are set up and dependencies are installed, shows significant improvement. Prek completes installation in 22.8 seconds compared to pre-commit’s 40.1 seconds—that’s 1.76 times faster, saving over 17 seconds per installation.

This improvement comes from prek’s more efficient environment management and parallel processing capabilities. Where pre-commit sets up each hook’s environment sequentially, prek parallelizes much of the work, fully leveraging modern multi-core CPUs.

These savings hit you every time you clone a fresh repository, onboard a new teammate, rebuild your development environment, or spin up a clean CI/CD job. In practice, these are some of the most time-sensitive moments in development: the first impression when a new contributor joins, or the critical “green build” loop in CI.

2.2. Execution Time

The run-time comparison is even more striking. Prek executes all hooks in 26.3 milliseconds versus pre-commit’s 176.7 milliseconds—nearly 7 times faster.

For something that happens on every commit, this difference transforms the development experience. With pre-commit, you notice the delay. With prek, the hooks feel nearly instantaneous.

This performance gap exists because of lower startup overhead (Rust binaries start faster than Python processes), efficient process spawning (prek minimizes the cost of launching hook processes), and optimized parallelization with better concurrent execution of independent hooks.

2.3. When Speed Actually Matters

You might be thinking: “Does shaving off seconds really matter?” The answer becomes clear when you scale up the impact.

For small and medium repositories like mine, prek already makes a meaningful difference: installs are about 2× faster, and hook execution is nearly 7× faster. That means less waiting when you first set up, and almost instant feedback every time you commit.

But the benefits become even more dramatic in larger projects. In the Apache Airflow benchmarks published by the prek authors, installation time dropped from 187 seconds with pre-commit to just 18 seconds with prek—a full order of magnitude faster. For projects with dozens of hooks and contributors, that kind of improvement changes the entire onboarding and CI experience.

Even beyond raw install times, the biggest impact is on everyday commits. Developers interact with pre-commit constantly, and even sub-second delays add friction. Prek crosses an important psychological threshold: hooks feel instantaneous, which means you stop noticing them. That reduces context switching, helps maintain flow, and removes the temptation to skip checks because they feel “too slow.”

In open source projects, where contributors might clone fresh repositories often and CI jobs rebuild environments from scratch, these savings add up across the community. For teams running frequent pipelines, the shorter runtimes directly reduce compute costs and feedback latency.

Put simply: whether you’re working on a small repo or a massive open source project, prek makes pre-commit checks feel like they should have all along—fast, seamless, and invisible.

2.4. What This Means for Your Workflow

These performance improvements have cascading effects throughout the development lifecycle:

  • For Individual Developers: Commits feel responsive rather than sluggish, with less context switching while waiting for hooks.
  • For Teams: Faster CI/CD pipelines, reduced compute costs from shorter-running jobs, and better developer satisfaction overall.
  • For Large Repositories: More hooks can be added without degrading the user experience, and onboarding new contributors becomes significantly faster.

2.5. Compatibility: Drop-in Replacement

One of prek’s strongest features is its compatibility with the existing pre-commit ecosystem. Your .pre-commit-config.yaml file works as-is. The hooks you’ve already configured continue to function identically.

This means adoption is trivial:

# Install prek
pip install prek

# Use it exactly like pre-commit
prek install --install-hooks
prek run --all-files

No configuration changes. No hook migrations. No breaking your team’s workflow. You get the performance benefits immediately without any of the typical migration pain.

2.6. The Trade-offs

In the interest of balanced assessment, it’s worth noting that prek is newer and less battle-tested than pre-commit. The ecosystem maturity difference means:

  • Pre-commit has more extensive documentation and community resources.
  • Edge cases and obscure configurations may be better handled by pre-commit.
  • Some organizations may prefer the stability of the more established tool.

However, for the vast majority of use cases, standard Python projects with common hooks like Black, Ruff, MyPy, and isort, prek works flawlessly while being dramatically faster.

2.7. The Bigger Picture: Rust in Python Tooling

Prek is part of a larger trend of performance-critical Python tooling being rewritten in Rust. We’ve seen this with:

  • Ruff: The lightning-fast Python linter and formatter
  • Polars: The high-performance DataFrame library
  • Pydantic V2: Rewritten core with a Rust foundation
  • uv: An extremely fast Python package installer

The pattern is clear: when performance is crucial and Python’s overhead becomes a bottleneck, Rust offers a compelling path forward while maintaining Python ecosystem compatibility.

3. Conclusion

After running these benchmarks, the case for prek is compelling. With faster installation and run times—all while maintaining full compatibility with existing pre-commit configurations—prek represents a significant improvement in the git hooks workflow.

The numbers tell a clear story: prek doesn’t just incrementally improve on pre-commit, it fundamentally transforms the experience. Installation goes from noticeable to negligible, and execution crosses the threshold from “slight delay” to “essentially instant.” For teams that value developer velocity and responsive tooling, these improvements compound over time into meaningful productivity gains.

The migration path is trivial, the benefits are immediate, and the performance gains speak for themselves. Whether you’re a solo developer tired of waiting for hooks or leading a team where those seconds multiply into hours of lost productivity, prek delivers exactly what modern development workflows demand: speed without compromise.

The age of waiting for git hooks is over. Give prek a try—your future self (and your teammates) will thank you.