azurelinux icon indicating copy to clipboard operation
azurelinux copied to clipboard

Early WIP of tooling improvements

Open dmcilvaney opened this issue 3 years ago • 0 comments

This is a very early WIP Of some tooling improvements.

I am planning to continue developing these and will split them into separate PRs when they are more ready for review.

High level changes:

  • Refactor Makefiles so we can selectively disable $(shell ...) commands when we are running in --dry-run mode (aka when computing targets for tab-completion in the shell, etc.) by adding the shell_real_build_only function.
  • Move all toolchain RPMs to live excusively in ./build/toolchain_rpms to fix the need for copy-toolchain-rpms extra command to resolve cycles, and fix the constant undesirable rebuilding of the worker chroot.
    • Update specreader to know about the toolchain manifest and accurately point package RPM paths to this folder.
    • Add a new local repo which targets this location (in addition to .build/rpm_cache and ./out/RPMS for use with the rpmcloner package.
    • Prioritize pulling from this directory over other sources
    • When REBUILD_TOOLS=y copy these packages into ./out/RPMS as well to maintain current functionality.
  • Fix the fetchers so they actually check if they have already pulled a package into the cache. Currently this flow only works for the Docker based reusable chroots (which look to cacheRepoID = "upstream-cache-repo") while the normal build flow only enables the existing cache (fetcherRepoID = "fetcher-cloned-repo") when we enable "*". tdnf tends to prefer contancting PMC instead of looking to the local repo (unclear why). This massively improves performance of the fetches when the cache is already hydrated from a previous build.

dmcilvaney avatar Oct 21 '22 21:10 dmcilvaney