Pluggable abstraction of base image interactions and support scripts

[nitzmahone]

Pluggable abstraction of base image interactions and support scripts

It's becoming apparent that Builder may need a pluggable distro/environment support abstraction. Package manager interactions and dependency resolution are by far the most common need, but there are many others.

Stream-of-conciousness dump follows; mainly just wanted to capture these thoughts in one place publicly instead of dribbling them out repeatedly in various conversations...

A primary theme for Builder 3.0 was making things more generic, e.g. support for arbitrary base images, additional user-definable hook points during key build events. It makes the tool more flexible and EE builds more performant by eliminating hard-coded steps that might be redundant or unnecessary, at the cost of requiring users to be more specific about the environment they're building upon. Since then, we've fielded a lot of requests that seem, at first blush, diametrically opposed. Some want to make builder even more generic, e.g. supporting non-RHEL base images. Others want us to add more "magical" handling of environment/distro specific situations, e.g. Python interpreter discovery, package manager detection/arg generation, OS package repo config, custom CA cert management. On the surface, they're all reasonable requests- especially for common situations, it shouldn't be necessary for every user to reinvent the wheel with the same custom build steps, but implementing everything as a first-class feature in builder itself is likely prohibitive for both implementation and ongoing maintenance/testing costs.

I'm also loathe to watch Builder follow the same path Ansible Core itself did: over time, the codebase became littered with untestable special cases that were accepted with little more assurance than "worked on my machine". Even the reasons for their existence are often lost to history, and they become a major impediment to platform improvements for fear of unknown breakage. The collections abstraction has allowed us to minimize new occurrences of these problems in Ansible Core, since responsibility for bespoke behavior is offloaded to collection authors. If we can come up with a similar pluggable abstraction (perhaps even packaged in collections) that allows reusable customizations for these common cases, we can wrap the existing upstream functionality in it to provide the same experience, while allowing users to augment or replace them when the need arises to do something more or different. The builder team's responsibility is limited to carefully defining and maintaining the abstraction itself, along with whatever builtin implementations we choose to support upstream.

For some of these items, there may be a lot of overlap with Ansible Core. Package manager abstraction? Sounds like the package action. Python dep installs? Sounds like the pip module. Finding a Python interpreter? Sounds like Core interpreter discovery. Seems like we should probably allow leveraging that where reasonable. Need to also be careful not to go so far as just reimplementing Ansible Container :wink:.