Enhancement: Extend CI workflow to test library detection with manually compiled dependencies

[JustCoding247]

Background

This issue is a follow-up to #3404, where we identified that the ModSecurity configure script fails to detect YAJL, LMDB, and PCRE2 libraries when they are installed in lib64 directories or other non-standard paths.

While the library detection fix has been submitted via PR, @airween suggested an important enhancement to extend the CI workflow to better test these scenarios.

Current CI Limitations

The current CI workflow primarily tests library detection using OS package managers, which typically install libraries in standard system paths. However, many users compile libraries manually and install them in custom locations, particularly:

• /usr/local/[library]/lib64/ directories
• Custom prefix paths
• Container environments with non-standard library layouts

Proposed Enhancement

Add new CI jobs that test the configure script's library detection capabilities with manually compiled dependencies, specifically:

1. Manual Library Compilation Testing

• Compile YAJL, LMDB, and PCRE2 from source
• Install them in various custom paths (lib64, custom prefixes)
• Verify configure script correctly detects these libraries

2. Multi-Distribution Testing

Test across various Linux distributions that commonly use lib64:

• Alma Linux 9, 10
• CentOS Stream 9, 10
• Oracle Linux 9, 10
• Rocky Linux 9, 10
• Ubuntu 22.04, 24.04
• Debian 12, 13
• OpenSUSE
• Fedora

3. Container Environment Testing

• Test in containerized environments where libraries are often manually compiled
• Verify compatibility with different base images

Benefits

1. Improved Reliability: Catch library detection issues before they reach users
2. Better Coverage: Test real-world scenarios where users compile dependencies manually
3. Cross-Platform Validation: Ensure compatibility across diverse Linux distributions
4. Regression Prevention: Prevent future breakage of library detection logic

Implementation Approach

1. Create new GitHub Actions workflow jobs
2. Add matrix builds for different distributions and library installation methods
3. Implement automated testing scripts for manual library compilation
4. Add validation steps to ensure configure script works correctly

Related Issues

• #3404 - Original lib64 detection issue
• #3318 - Similar PCRE2 detection problems

---

Note: This enhancement would significantly improve the project's build system reliability and help prevent similar issues in the future. I'm ready to contribute to implementing these CI improvements.

[airween]

Hi @JustCoding247,

Note: This enhancement would significantly improve the project's build system reliability and help prevent similar issues in the future. I'm ready to contribute to implementing these CI improvements.

thanks, feel free to send any contribution.

[JustCoding247]

Hi @airween,

Thank you for your positive response to this enhancement proposal! I'm excited about the opportunity to contribute to improving ModSecurity's CI workflow.

Sharing My CI/CD Experience

Before diving into specific implementation details, I'd like to share my experience with a CI/CD pipeline I've developed for building ModSecurity across multiple Linux distributions. This might provide some inspiration or ideas that could be adapted for the GitHub Actions workflow.

I've been maintaining a Jenkins-based CI/CD system that successfully builds ModSecurity on 12 different Linux distributions, including the ones mentioned in this issue. Through this work, I've encountered and solved many of the challenges related to library detection in different environments, particularly the lib64 issue we discussed in #3404.

My Approach Overview

Here's a brief overview of how my pipeline handles these challenges:

1. Containerized Build Environments: Using Docker to ensure consistent and reproducible builds
2. Pre-compiled Dependencies: Managing 15+ dependency libraries to accelerate builds
3. Automated Patch Application: Including the lib64 detection fix when needed
4. Cross-Distribution Support: Successfully building on RHEL-based and Debian-based systems

Detailed Documentation

I've prepared a comprehensive documentation of my CI/CD pipeline implementation. While it's designed for Jenkins rather than GitHub Actions, many concepts and strategies could potentially be adapted:

---

ModSecurity CI/CD Pipeline Documentation

Overview

This document provides a comprehensive introduction to a complete CI/CD pipeline implementation for the ModSecurity project. The solution leverages Jenkins as the pipeline orchestration tool and implements cross-platform standardized build environments through Docker containerization technology, supporting automated builds of ModSecurity across multiple Linux distributions.

Architecture Design

Overall Architecture

The CI/CD pipeline adopts a layered architecture design, consisting of three core components. The complete directory structure is as follows:

ModSecurity CI/CD
├── git.example.com/DevSecOps/Jenkins/CICD/modsecurity/                      # Jenkins Pipeline Control Layer
│   ├── Jenkinsfile                                                          # Main pipeline definition file
│   ├── stage.upload.script.groovy                                           # File upload stage script
│   ├── build-modsecurity-3-on-*.cicd.sh                                     # Docker container build control scripts
│   ├── Dockerfile                                                           # Build environment image definition
│   ├── README.md
│   └── LICENSE
│
├── git.example.com/DevSecOps/Jenkins/Scripts/modsecurity/                   # Build Script Execution Layer
│    ├── build-modsecurity-3-on-*.build.sh                                   # Actual compilation build scripts
│    ├── README.md
│    └── LICENSE
│
└── docker/images/oss/modsecurity/                                           # Docker Image Environment Layer
    └── keep-modsecurity-up-to-date.sh                                       # Image maintenance and update script

Component Responsibilities

1. Jenkins Pipeline Control Layer (CICD)

   • Responsible for orchestrating and controlling the entire build process
   • Manages build parameters and environment variables
   • Coordinates the execution of various build stages

2. Build Script Execution Layer (Scripts)

   • Contains the actual compilation and build logic
   • Handles dependency library management and version control
   • Implements compilation optimizations and patch applications

3. Docker Image Environment Layer (Images)

   • Provides standardized compilation environments
   • Maintains base images for multiple Linux distributions
   • Ensures consistency and reproducibility of build environments

Core Components Details

1. Docker Image Environment Layer

Script Location: docker/images/oss/modsecurity/keep-modsecurity-up-to-date.sh

Main Functions:

• Maintains base images for multiple Linux distributions
• Automatically updates and optimizes images, installing required compilation dependencies
• Generates single-layer optimized images to reduce image size

Supported Linux Distributions:

• AlmaLinux 9, 10
• CentOS Stream 9, 10
• Oracle Linux 9, 10
• Rocky Linux 9, 10
• Ubuntu 22.04, 24.04
• Debian 12, 13

Workflow:

1. Pull official base images and re-tag them
2. Create and start containers
3. Execute appropriate updates and dependency installations based on distribution type (Debian/RHEL)
4. Install development tools and libraries required for ModSecurity compilation
5. Clean cache and temporary files
6. Export containers as single-layer images
7. Save image files for subsequent use

2. Jenkins Pipeline Control Layer

Repository Location: git.example.com/DevSecOps/Jenkins/CICD/modsecurity

Main Files:

• Jenkinsfile: Main pipeline definition file
• stage.upload.script.groovy: File upload stage script
• build-modsecurity-3-on-*.cicd.sh: Docker container build control scripts (multi-distribution support)
• Dockerfile: Build environment image definition

Pipeline Stages:

1. Checkout Stage

   • Check out CI/CD code repository
   • Check out ModSecurity source code repository
   • Switch to specified version tag

2. Upload Stage

   • Dynamically replace version parameters in scripts
   • Upload build scripts to target server
   • Upload ModSecurity source package
   • Upload pre-compiled packages of all dependencies

3. Configure Stage

   • Execute configure in Docker container
   • Apply necessary patches (e.g., lib64 path fixes)

4. Make Stage

   • Execute parallel compilation

5. MakeTest Stage

   • Run test cases

6. MakeInstall Stage

   • Install compilation artifacts
   • Package binary files

7. Cleanup Stage

   • Clean temporary files and containers

3. Build Script Execution Layer

Repository Location: git.example.com/DevSecOps/Jenkins/Scripts/modsecurity

Core Scripts: build-modsecurity-3-on-*.build.sh (supports multiple distributions)

Main Functions:

• Actually executes ModSecurity compilation and building
• Manages dependency library extraction and configuration
• Handles compilation parameters and optimization options
• Applies necessary patch fixes

Key Features:

1. lib64 Path Compatibility Fix

   • Automatically detects and fixes lib64 directory detection issues in ModSecurity 3.0.14
   • Adds lib64 path support for libraries like YAJL, LMDB, PCRE2

2. Compilation Optimizations

   • Uses secure compilation options: -D_FORTIFY_SOURCE=2
   • Enables performance optimization: -O2 -flto
   • Enhances security protection: -fstack-protector-strong

3. Dependency Management

   • Supports version management for 15+ dependency libraries
   • All dependencies are pre-compiled to accelerate the build process

Workflow

Complete Build Process

graph TD
    A[Start Build] --> B[Prepare Docker Image Environment]
    B --> C[Jenkins Triggers Pipeline]
    C --> D[Checkout Code Repositories]
    D --> E[Upload Build Files to Target Server]
    E --> F[Create Docker Build Container]
    F --> G[Extract Dependencies]
    G --> H[Configure ModSecurity]
    H --> I[Compile ModSecurity]
    I --> J[Run Tests]
    J --> K[Install and Package]
    K --> L[Clean Environment]
    L --> M[Build Complete]

Parameterized Build

The pipeline supports the following parameters:

• MODSECURITY_VERSION: ModSecurity version (e.g., 3.0.14-ubuntu-24.04)
• GIT_BRANCH: Git branch name
• LINUX_IP_ADDRESS: Target build server IP
• LINUX_SSH_PORT: SSH port
• Various dependency library version parameters (LIBSSH2_VERSION, CURL_VERSION, etc.)

Technical Features

1. Containerized Build Environment

• Uses Docker to ensure build environment consistency
• Supports parallel builds across multiple Linux distributions
• Image optimization techniques reduce storage space

2. High Degree of Automation

• Fully automated process from environment preparation to final packaging
• Automatic dependency handling
• Automatic application of necessary patches

3. Strong Scalability

• Easy to add support for new Linux distributions
• Modular design facilitates maintenance and extension
• Parameterized configuration supports different version builds
• Can be extended to Windows, macOS, and other operating systems as needed

4. Security Considerations

• Uses MaskPasswordsBuildWrapper to protect sensitive information
• Enables security hardening options during compilation
• Cleans temporary files during the build process

Usage Guide

Prerequisites

1. Jenkins environment configured
2. Docker environment ready
3. Target build server accessible
4. Git repository access permissions configured

Triggering a Build

1. Create a parameterized build job in Jenkins
2. Configure necessary parameters (version, target server, etc.)
3. Trigger the build job
4. Wait for build completion and retrieve artifacts

Build Artifacts

After build completion, the following artifacts are generated:

• ModSecurity binary package: modsecurity-{version}.{distribution}.binary-{timestamp}.tar.gz
• Complete installation package including all dependencies

Maintenance Recommendations

Regular Updates

1. Base Image Updates

   • Recommend running the keep-modsecurity-up-to-date.sh script monthly
   • Keep system packages and security patches up to date

2. Dependency Version Management

   • Regularly check for new versions of dependencies
   • Test compatibility of new versions

3. Patch Management

   • Track upstream project updates
   • Evaluate whether to update or remove existing patches

Troubleshooting

1. Build Failures

   • Check Jenkins build logs
   • Verify Docker container status
   • Confirm network connectivity

2. Dependency Issues

   • Check if dependency files exist
   • Verify version compatibility
   • Review configure logs

3. Performance Optimization

   • Use parallel compilation to accelerate builds
   • Optimize Docker image size
   • Consider using build cache

Platform Extension Capabilities

Currently Supported Platforms

The CI/CD pipeline currently supports 12 mainstream Linux distributions, covering the most commonly used operating systems in enterprise production environments. These distributions have been thoroughly tested and run stably.

Cross-Platform Extension Potential

The pipeline architecture has excellent extensibility and can be extended to other operating systems based on actual work requirements:

1. Windows Platform

   • Can be implemented through Windows containers or WSL2
   • Adapt to Visual Studio compilation toolchain
   • Support generating Windows native binaries

2. macOS Platform

   • Can be implemented through macOS build agents
   • Adapt to Xcode and Homebrew toolchains
   • Support generating Universal Binary

3. Other Unix-like Systems

   • BSD series like FreeBSD, OpenBSD
   • Commercial Unix like Solaris, AIX
   • Embedded Linux distributions

Extension Implementation Strategy

1. Evaluate Requirements: Decide whether to extend based on actual project needs
2. Environment Adaptation: Adjust compilation environment and dependencies for target platform
3. Script Porting: Modify build scripts to accommodate platform characteristics
4. Test Validation: Ensure build quality on new platforms

Summary

This ModSecurity CI/CD pipeline solution achieves the following through the organic combination of Jenkins, Docker, and automated scripts:

1. Standardization: Unified build environment and process
2. Automation: Fully automated execution throughout the process
3. Repeatability: Ensures consistency of build results
4. Extensibility: Easy to add new platform support, extendable to Windows, macOS, and other systems
5. High Efficiency: Parallel builds and pre-compiled dependencies
6. Wide Coverage: Supports 12 mainstream Linux distributions, can be extended as needed

This solution is not only suitable for building ModSecurity on multiple Linux distributions but also has the capability to extend to other operating systems. Through containerization technology and automated processes, it reduces the maintenance cost of build environments and improves build reliability and flexibility.

Appendix

Related Links

• ModSecurity Official Repository: https://github.com/owasp-modsecurity/ModSecurity
• Jenkins Official Documentation: https://www.jenkins.io/doc/
• Docker Official Documentation: https://docs.docker.com/

Version History

• 2025-06: Initial release
• Supports lib64 path fixes for ModSecurity 3.0.14
• Supports 12 mainstream Linux distributions (6 RHEL series, 6 Debian series)
• Capable of extending to Windows, macOS, and other operating systems

---

How This Might Help GitHub Actions

While my implementation uses Jenkins, several concepts could be adapted for GitHub Actions:

1. Docker-based Build Matrix: Using containerized environments for consistent builds across distributions
2. Dependency Pre-compilation: Caching compiled dependencies to speed up CI runs
3. Automated Patch Application: Applying fixes dynamically based on version/distribution
4. Comprehensive Distribution Coverage: Testing on a wide range of Linux distributions

Potential Collaboration

I'd be happy to:

• Share specific scripts or configurations that might be useful
• Help adapt these concepts to GitHub Actions
• Contribute to implementing the enhanced CI workflow
• Test and validate changes across my existing infrastructure

Key Learnings from My Experience

1. Container-based builds are crucial for reproducibility across different distributions
2. Pre-compiled dependencies can significantly reduce CI time
3. Automated environment setup prevents many common build issues
4. Distribution-specific handling is often necessary (e.g., RHEL vs Debian package managers)

I hope this information provides some useful insights for enhancing ModSecurity's CI workflow. I'm looking forward to collaborating on this improvement and would be glad to answer any questions or provide more specific details about any aspect of this pipeline.

Best regards!

[airween]

@JustCoding247,

thanks for this awesome description.

Please let me review it in details and response then.

Also Cc this comment to @theseion and @fzipi - what do you think guys?

[fzipi]

My 2 cents: let's keep everything in GHA if we can. Adding Jenkins also could make it difficult to maintain.

[JustCoding247]

@airween @fzipi @theseion Thank you for the feedback! I completely understand and agree with keeping everything in GitHub Actions for maintainability.

Let me clarify my approach: I'm not suggesting adding Jenkins or external CI systems. Instead, I'm thinking about whether some automation patterns from my CI/CD framework might be adaptable to GitHub Actions workflows.

I've been working on multi-distribution testing patterns that might be relevant to the scenarios mentioned in this issue. To give everyone a better sense of what I mean, here are some components I've been experimenting with:

• Core CI/CD Framework: https://github.com/osscicd/gcc/tree/release
• Automation Scripts: https://github.com/ossscripts/gcc/tree/release
• Container Images: https://github.com/ossimages/gcc/tree/release

These repositories show some approaches that might have potential benefits:

1. GitHub Actions Integration: The patterns could potentially work as native GHA workflows
2. Multi-project Experience: I've applied similar logic across multiple projects
3. Scenario Coverage: They're designed to handle cases like lib64 paths and manual compilations
4. Self-contained: Once set up, they tend to be low-maintenance

I'm sharing this mainly to see what others think - whether there might be useful ideas here that could inspire improvements to ModSecurity's CI approach. I'd be curious to hear thoughts from the community, especially from @airween after your review.

[fzipi]

Hi @JustCoding247!

1. When I saw this issue, I saw AI helped to create this text. Which is not bad on itself. If you have problems communicating, or helps explain your ideas clearer, that's fine. It is not 100% normal yet in this community.
2. You user has no previous commits or push in github. So it is a new user all over. Again, it is not bad, but it is off.
3. Your Framework, Automation and Container components you have been experimenting are also new in github.

It makes me think about what a great philosopher words: "I don't know Rick, it looks fake".

I feel that this project should be extremely careful about supply chain attacks in general. But I might be completely wrong.

[JustCoding247]

Hi @fzipi

Thank you for your feedback — I completely understand your concerns.

Yes, I’m new to this project and to GitHub in general, and I’ve been using AI to help me express myself more clearly since English is not my first language. I want to assure you that I’m genuinely interested in contributing and learning, not trying to mislead or cause harm.

I also fully agree that projects like this must take supply chain risks seriously. If there’s anything I should do differently to follow the community's standards, I’m happy to adjust.

Thanks again for your time and for helping maintain the quality and security of the project.

I really appreciate the work you and the community have done here.

[fzipi]

Thanks for your clarification, and for taking this seriously.

Let's make good use of your time then. 😄

My starting point will be to try to update your pipelines and use only upstream containers for building. That will give us the best from both worlds.

Do not start with a full matrix initially, just a basic PoC that enables us to iterate and see quick results!

[JustCoding247]

Hi @fzipi,

Thanks for the feedback!

I'm migrating my CICD framework from Jenkins to GHA. If something becomes relevant, I'll share it.

Best regards, JustCoding247

[theseion]

@JustCoding247 This sounds exciting. I'm very much looking forward to a PoC.

I do have one question: do you really think it's necessary to make the configure script more complex by having it examine a bunch of potential installation paths? Wouldn't it be much simpler to give users the ability to explicitly configure those paths? A shell script could take care of this easily and then execute configure / make.