datadog-agent/7.66.0 package update

[octo-sts[bot]]

[octo-sts[bot]]

📦 Build Failed: Missing Dependency

fatal error: 'stdarg.h' file not found

Build Details

Category	Details
Build System	Ninja/Clang (eBPF compilation)
Failure Point	/opt/datadog-agent/embedded/bin/clang-bpf compilation of various source files

Root Cause Analysis 🔍

The 'stdarg.h' header file is missing from the build environment. This is a standard C header file required for variable arguments handling. The Linux kernel headers being used for eBPF compilation depend on this header, but it's not available in the system's include path for the clang-bpf compiler.

---

🔍 Build failure fix suggestions

Found similar build failures that have been fixed in the past and analyzed them to suggest a fix:

Suggested Changes

File: package.yaml

• modification at line around line 190-200 (pipeline section - Build the System Probe step) Original:

      # Build once to correctly setup links/generates. The system-probe we end
      # up using will be part of the multicall below.
      unshare --user --map-root-user \
        invoke -e system-probe.build

Replacement:

      # Build once to correctly setup links/generates. The system-probe we end
      # up using will be part of the multicall below.
      # Add stdarg.h and other clang headers to include path
      mkdir -p /usr/include/clang
      cp -r /opt/clang/lib/clang/12.0.1/include/* /usr/include/
      unshare --user --map-root-user \
        invoke -e system-probe.build

Click to expand fix analysis

Analysis

The current build failure shows that the clang-bpf compiler is unable to find the 'stdarg.h' header file during eBPF compilation. This is a standard C header file that should be included in the build environment. The error is occurring during the compilation of eBPF code using the clang-bpf compiler.

Looking at the Melange YAML file, I can see that the build process downloads and extracts clang 12.0.1 to /opt/clang and adds it to the PATH. It also downloads and extracts Linux kernel headers. However, the build is failing to find the standard C headers needed by the clang compiler during eBPF compilation.

The root issue appears to be that while Linux kernel headers are correctly being linked, the standard C headers that come with clang (including stdarg.h) are not being properly included in the compiler's search path. This is common when using a separate clang installation that needs to be told where to find its own include files.

Click to expand fix explanation

Explanation

The suggested fix addresses the root cause of the build failure, which is that the clang-bpf compiler cannot find the 'stdarg.h' header file during eBPF compilation.

The fix copies the standard C headers from the downloaded clang distribution to the system's include path (/usr/include/), where the compiler will look for them by default. This is a common solution when using a standalone clang installation that is not integrated with the system's header files.

Specifically, clang 12.0.1 is downloaded and extracted to /opt/clang in the build process. This clang installation comes with its own set of header files located in the /opt/clang/lib/clang/12.0.1/include/ directory. These headers include stdarg.h and other essential C headers that are required for compilation.

By copying these headers to /usr/include/, we ensure that when the clang-bpf compiler is invoked to compile eBPF code, it will be able to find all the necessary standard headers, including stdarg.h.

This approach maintains the ephemeral and isolated nature of the build environment while ensuring that the compiler has access to all the necessary include files it needs to successfully compile the eBPF code.

Click to expand alternative approaches

Alternative Approaches

• Instead of copying the headers to /usr/include/, we could modify the compiler flags to include the clang include directory by adding '-I/opt/clang/lib/clang/12.0.1/include' to the compilation command. This would involve modifying the invoke command or the Makefile used by the system-probe build process.
• Another approach would be to create symbolic links from /usr/include/stdarg.h to the corresponding file in the clang installation instead of copying all headers. This would be more selective but might miss other required headers.
• We could also consider using the system's existing clang package (clang-16) instead of downloading clang-12, which would automatically have its headers properly set up. However, this might introduce compatibility issues if the code specifically requires clang-12.

---

Was this comment helpful? Please use 👍 or 👎 reactions on this comment.

[debasishbsws]

https://github.com/chainguard-dev/internal-dev/issues/12556

[sergiodj]

This was an interesting one. It actually involved two separate fixes; the stdarg.h one was the first, but then the build failed again later. Let me leave a brief report here of what was done.

For the stdarg.h failure, it took me some time to understand what was happening. First, I determined that the header is indeed present /usr/include, but that one wasn't really being used because it's GCC specific. I then took a step back to understand how the build was performed, and oh boy, it's complicated. The important part is that we actually download our clang and perform the build with it. So I started looking for the stdarg.h that was actually being used, and found it under /opt/clang/lib/clang/12.0.1/include/stdarg.h. Cool, so now we know that the header exists.

After spending some time looking at the differences between upstream versions 7.65.0 and 7.66.0, I couldn't immediately find anything strange there. This was strange, because 7.65.0 builds fine. So I started comparing the compilation commands, and noticed that the compiler being used differed between these two versions. The former was using clang, which resolved to /opt/clang/bin/clang, and the latter uses /opt/datadog-agent/embedded/bin/clang-bpf. Hmm... I decided to enter a debugging session and check the include paths being used by both compilers (clang -E -Wp,-v -xc /dev/null will offer useful info). Comparing both outputs, I noticed that our clang (i.e., /opt/clang/bin/clang) looks for headers inside /opt/clang/lib/clang/12.0.1/include, which upstream's clang doesn't (of course). With that, I was able to find the commit that introduced the change: https://github.com/DataDog/datadog-agent/commit/494f98eb5db09652258c0e0aac936a35d69d7b62 . Unfortunately I had to patch the system prober because it's now using absolute paths and hardcoding where their clang-bpf is located.

After fixing that first problem, the second one showed up: clang was complaining that MNT_INTERNAL is undefined. But wait, our Linux kernel headers do include mount.h, where that define is located. I started looking around and comparing changes affecting files inside pkg/security/ebpf/c/, and found the following suspicious commit: https://github.com/DataDog/datadog-agent/commit/81ff57c37398943f4e8f486cb51d6ffeb8f901e7. Reverting only the part where <linux/mount.h> was removed was enough to make the build pass.