Long startup delay when running `podman-compose up -d`

[bmbouter]

To reproduce:

1. git clone [email protected]:pulp/pulp-oci-images.git
2. cd pulp-oci-images/images/compose
3. podman-compose up -d

I then see the images created:

CONTAINER ID  IMAGE                           COMMAND               CREATED        STATUS            PORTS                     NAMES
1d37c99db6de  docker.io/library/postgres:13   postgres              7 minutes ago  Up 7 minutes ago  0.0.0.0:5432->5432/tcp    compose_postgres_1
a975a4c0ec14  docker.io/library/redis:latest  redis-server          7 minutes ago  Up 7 minutes ago                            compose_redis_1
c7aa6d2b9425  quay.io/pulp/pulp:latest        pulp-api              6 minutes ago  Up 6 minutes ago  0.0.0.0:24817->24817/tcp  compose_pulp_api_1
2af94d93c9f5  quay.io/pulp/pulp:latest        pulp-content          6 minutes ago  Up 6 minutes ago  0.0.0.0:24816->24816/tcp  compose_pulp_content_1
b930c30b2d67  quay.io/pulp/pulp:latest        pulp-worker           6 minutes ago  Up 6 minutes ago                            compose_pulp_worker_1
caa284bc08e1  quay.io/pulp/pulp:latest        pulp-worker           6 minutes ago  Up 6 minutes ago                            compose_pulp_worker_2
98d32664aa0c  quay.io/pulp/pulp-web:latest    /bin/sh -c nginx ...  6 minutes ago  Up 6 minutes ago  0.0.0.0:8080->8080/tcp    compose_pulp_web_1

The compose_pulp_api_1 is working great right away, but the compose_pulp_content_1 and compose_pulp_worker_1 and 2 hang for several minutes and after ~ 8 minutes become available.

When I look at the container logs of compose_pulp_content_1 just after starting I see it shows only: /usr/bin/wait_on_postgres.py

Then after 8 minutes it prints out a wall of a zillion entries of:

Checking postgres host postgres
Checking postgres port 5432

Then at the end it shows:

Checking postgres host postgres
Checking postgres port 5432
Checking postgres host postgres
Checking postgres port 5432
Unable to reach postgres on port 5432
+ /usr/bin/wait_on_database_migrations.sh
Checking for database migrations

Database migrated!
+ PULP_GUNICORN_TIMEOUT=90
+ PULP_CONTENT_WORKERS=2
+ exec gunicorn pulpcore.content:server --name pulp-content --bind '[::]:24816' --worker-class aiohttp.GunicornWebWorker --timeout 90 --workers 2 --access-logfile -
[2022-10-18 18:38:07 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2022-10-18 18:38:07 +0000] [1] [INFO] Listening at: http://[::]:24816 (1)
[2022-10-18 18:38:07 +0000] [1] [INFO] Using worker: aiohttp.GunicornWebWorker
[2022-10-18 18:38:07 +0000] [8] [INFO] Booting worker with pid: 8
[2022-10-18 18:38:07 +0000] [9] [INFO] Booting worker with pid: 9

My theory is that the /usr/bin/wait_on_postgres.py is timing out and looping to print out the Checking postgres host postgres and Checking postgres port 5432 messages, which times out yielding Unable to reach postgres on port 5432. It then continues (I expected it to stop then) to check if the migrations are migrated and since it does it finishes it's startup.