Allow modifiable _sources (output) dir for builds

[jjwatt]

SUMMARY

This dir, _sources is an output/target directory used by make rules and playbooks for storing generated files and for mounting some things into containers. Before this patchset, _sources is sometimes hard-coded, and it is at other times in a sources_dest var. This patchset makes it possible to set/change this sources dir. Some simplifications along the way.

Use

$ make SOURCES=out docker-compose

Why?

This patch cleans things up a little, and it may fix some hidden bugs, but the main force behind it is a weird incompatibility with podman: Docker-compose will automatically name a network after the name of the directory it's started from, but podman, using docker-compose yml through the socket API, but podman rejects the network name because it starts with an _. Strangely, I couldn't find where that is specified in OCI or anything, but I did find the code in podman that tests against a regexp (iirc, it uses a hostname or dns name regexp), so this may be a spec miss/bug in podman. But, being able to change the name makes it work as a workaround, so I set things up to change the name so that I can continue use podman for testing on RHEL 8/9.

There are still more patches needed to make podman work. I have it working in a branch, and I'm pulling those on top of HEAD one at a time and fixing them up for PRs.

ISSUE TYPE

• Bug, Docs Fix or other nominal change

COMPONENT NAME

• Other

AWX VERSION

awx: 0.1.dev32930+g3088eba.d20230314

ADDITIONAL INFORMATION

• Add override for '_sources' dir in Makefile SOURCES var
• Change docker-compose rules to use $(SOURCES)
• Add $SOURCES sh var to docker-compose container's sh scripts
• Add a SOURCES make variable that defaults to _sources to replace hard-coded _sources
• Add new SOURCES env var to docker-compose template
• Change plumb* ymls to use sources_dest or _sources
• Change pre_translation.sh to enable custom SOURCES
  • Use $SOURCES if it's in the env, otherwise original behavior (use '_sources' string for the dir)
• Change migrate-from-local-docker to use var sources_dest
• Change playbook to copy over all config files used by docker-compose
  • Add copy grafana dir rule
  • Copy supervisor.conf and redis.conf to roles/sources/templates dir
  • Add superviosr.conf and redis.conf to what's copied from the playbook
• Add awx_root to sources/defaults/main.yml
  • Discover awx_root by discovering the git root
  • Default to ../../../../ like the old behavior
• Change sources/tasks/main.yml to use awx_root in playbook
  • Introduce awx_root var because docker-compose yml kept trying to guess directory names using a scary anchored relative path, and it would fail if _sources moved
• Change docker-compose template
  • Remove use of relative path with anchor (../docker-compose/../)
• Change plumb playbooks to use sources_dest/default
• Change $DIR/sources_dest references to be just sources_dest

[jjwatt]

I took care of the minikube standout by not trying to fix it :). Basically, since it's the only thing that needs to use _sources from inside the container, I leave the path hard-coded for it in the bootstrap_development.sh script, and then I make sure that the minikube playbook copies the generated bootstrap_minikube.py from our newly configurable $SOURCES/sources_dest to the old hard-coded location so that it'll be where it expects.

So this already starts to pay off: Before, the minikube playbooks were downloading huge minikube and k8s go binaries to the _sources directory inside the project directory, so they were always copying all of those over when you ran the container. Now that you can specify $SOURCES outside of the project dir (e.g., /tmp/sources), it automatically won't re-copy those files in every time you start a container! hooray

[jjwatt]

Letting this sit for a bit. Hao and I are pairing on some other Makefile changes, and Hao has an idea to simplify the docker-compose file/build even more. We're going to land those other changes and see if we still need this.

[AlanCoding]

@jjwatt what you mentioned in your last comment sounds like https://github.com/ansible/awx/pull/14188

And I believe this was a well-considered patch that would have been helpful. I'm not sure if it's revivable with the current conflicts, but I would support that still.