Undeclared LLVM build dependency (Python 3)

[bb010g]

LLVM's build cycle has a undeclared phony dependency on ~~Git's git (or not? currently building to check) & an undeclared dependency on~~ Python 3 (of the form python${major}${minor}).

Here's a Nix expression for a reproducing derivation:

zig-bootstrap-29-0-repro.nix

{ pkgs ? import pkgsPath { }
, pkgsPath ? # <nixpkgs>
    builtins.fetchTarball {
      url = "https://github.com/NixOS/nixpkgs/archive/95b9c99f6d091273572ff1ec62b97b6ad3f68bdf.tar.gz";
      sha256 = "1qcwr9binkwfayix88aljwssxi5djkwscx0rnwlk1yp9q60rgp3d";
    }
, withUndeclaredInputs ? false
}:
pkgs.callPackage (

{ lib, stdenv, cmake, fetchFromGitHub, python3, writeScriptBin
, hostTriple ? let host = stdenv.hostPlatform.parsed; in {
    cpu = host.cpu.name;
    os = host.kernel.name;
    abi = host.abi.name;
  }
, withUndeclaredInputs ? true
}:

stdenv.mkDerivation rec {
  pname = "zig-bootstrapped";
  version = "0.6.0+2020-04-16";

  src = fetchFromGitHub {
    owner = "ziglang";
    repo = "zig-bootstrap";
    rev = "2bd6bea6203d7275851dacad34eaa8df205c8898";
    sha256 = "0dk71y30pn85dr30q4dnhkbx2rsf28rinmhvv5qa7wrvcdvvbpv2";
  };

  nativeBuildInputs = [
    cmake
  ] ++ lib.optionals withUndeclaredInputs [
    # Python 3
    python3
    # usages: path /// dependency type; usage
    # + ./clang/CMakeLists.txt /// hard
    #   - controlled by LLVM_INCLUDE_TESTS, enabled by default
    #     (unless LLVM_LIT is false, which it isn't, as
    #     ./llvm/utils/lit/lit.py exists in our vendoring)
    # + ./clang/bindings/python/tests/CMakeLists.txt /// soft
    #   - used for target "check-clang-python"
    #   - controlled by RUN_PYTHON_TESTS, enabled by default
    #     (unless LLVM_ENABLE_PIC is false, due to a libclang.so dep)
    # + ./clang/utils/perf-training/CMakeLists.txt /// hard?
    #   - used for targets "clear-profraw", "generate-profdata",
    #     "clear-dtrace-logs", & "generate-order-file"
    #   - these all just unceremoniously use PYTHON_EXECUTABLE
    #   - target "generate-order-file" creates CLANG_ORDER_FILE,
    #     which ./clang/CMakeLists.txt hard depends on existing when APPLE
    # + ./lld/CMakeLists.txt /// hard
    #   - controlled by LLVM_INCLUDE_TESTS, enabled by default
    #     (unless LLVM_LIT is false, as above)
    # + ./llvm/CMakeLists.txt /// hard
    #   - requires 2.7 or newer
    #   - used for LLVMBuild (llvm-build)
    #   - mocking out an executable to only pass FindPythonInterp's tests
    #     results in LLVMBUILDCMAKEFRAG not being generated & its include
    #     failing a screen of code later
    # + ./llvm/cmake/config-ix.cmake /// hard?
    #   - PYTHON_EXECUTABLE used in the "find_python_module" function,
    #     which is used later in the same file to start finding modules
    # + ./llvm/cmake/modules/AddLLVM.cmake /// hard
    #   - various utility usages? could probably be replaced here reasonably
    #     by invocations of other languages.
    # + ./llvm/runtimes/CMakeLists.txt /// none
    #   - this file includes FindPythonInterp and then does nothing with it?
    # + ./llvm/tools/llvm-shlib/CMakeLists.txt /// hard
    #   - controlled by MSVC
    #   - PYTHON_EXECUTABLE used to generate a C API export file for Windows
  ];

  dontUseCmakeConfigure = true;

  preBuild = ''
    export HOME=$TMPDIR;
  '';

  buildPhase = let host = hostTriple; in ''
    runHook preBuild
    ./build -j1 ${host.cpu}-${host.os}-${host.abi} baseline
    runHook postBuild
  '';

  doCheck = false;

  # everything is dumped in the `out` output b/c this takes forever to build
  installPhase = ''
    runHook preInstall
    cp -R ./out/ "$out"/
    runHook postInstall
  '';

  meta = {
    description =
      "General-purpose programming language and toolchain for maintaining robust, optimal, and reusable software";
    homepage = "https://ziglang.org/";
    license = with lib.licenses; [ l.mit ];
    platforms = lib.platforms.unix;
    maintainers = with lib.maintainers; [ andrewrk bb010g ];
  };
}

) {
  inherit withUndeclaredInputs;
}

Run nix-build --show-trace -k ./zig-bootstrap-29-0-repro.nix to reproduce. To build successfully, pass an argument --arg withUndeclaredInputs true. To use your local Nixpkgs channel instead of downloading & using the provided pinned version, pass an argument --arg pkgsPath '<nixpkgs>'.

(An actual issue after bringing this up to Andrew on Twitter yesterday)

[bb010g]

LLVMBuild is probably the biggest hurdle to removing Python 3.

LLVMBuild consists of two primary components:

• The LLVMBuild configuration file format for components

  The LLVMBuild files themselves are just a declarative way to describe the project structure. The actual building of the LLVM project is handled by another build system (See: CMake).

  The build system implementation will load the relevant contents of the LLVMBuild files and use that to drive the actual project build. Typically, the build system will only need to load this information at “configure” time, and use it to generate native information. Build systems will also handle automatically reconfiguring their information when the contents of the LLVMBuild.txt files change.

• The llvm-build utility tool to help interfacing with the build system

  llvm-build is a tool for working with LLVM projects that use the LLVMBuild system for describing their components.

  At heart, llvm-build is responsible for loading, verifying, and manipulating the project’s component data. The tool is primarily designed for use in implementing build systems and tools which need access to the project structure information.

[matu3ba]

can this be closed?