Container dependencies using `podman play kube`

[joshmalbrecht]

Feature request description

We are in the process of migrating our CI from using docker-compose to podman play kube and we used the podman generate kube command to generate a Kubernetes YAML file based on our existing docker-compose.yml. However, it appears that there is no container dependency functionality that is equivalent to the docker-compose depends_on functionality.

We would like to create a dependency between our application container and a database container so that the application container does not attempt to start until the database container is created and has a healthy healthcheck. This would guarantee that the application container can connect to the database container. Currently both containers are started in parallel using the podman play kube command and it is possible that the database container is not available by the time that the application container attempts to connect and causes issues.

Suggest potential solution

No response

Have you considered any alternatives?

An alternative would be adding database connection retry logic to the application to give the database container more time to get to a healthy state and then make a connection. This will work as alternative and I have a PR out for these changes (https://github.com/candlepin/candlepin/pull/4656), but I was wondering if this would be helpful functionality for others.

Additional context

Example docker-compose.yml using depends_on:

version: '3.8'

services:
  postgres:
    image: postgres:latest
    container_name: postgres
    environment:
      POSTGRES_USER: candlepin
      POSTGRES_PASSWORD: candlepin
      POSTGRES_DB: candlepin
      POSTGRES_HOST_AUTH_METHOD: trust
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U candlepin -d candlepin"]
      interval: 5s
      timeout: 5s
      retries: 10

  candlepin:
    image: quay.io/candlepin/candlepin:dev-latest
    container_name: candlepin
    environment:
      JPA_CONFIG_HIBERNATE_CONNECTION_URL: jdbc:postgresql://postgres/candlepin
      CANDLEPIN_AUTH_CLOUD_ENABLE: "false"
      CANDLEPIN_AUTH_TRUSTED_ENABLE: "true"
      CANDLEPIN_STANDALONE: "false"
      MODULE_CONFIG_HOSTED_CONFIGURATION_MODULE: org.candlepin.testext.hostedtest.HostedTestModule

    ports:
      - "8443:8443"
    healthcheck:
      test: curl --fail -k https://localhost:8443/candlepin/status
      timeout: 5s
      retries: 10
      start_period: 30s
    depends_on:
      postgres:
        condition: service_healthy

[rhatdan]

@ygalblum @umohnani8 @haircommander PTAL

[rhatdan]

Is this the function of an init container? I think there is a special one that continues to run after init is complete.

[haircommander]

yeah initContainers are started and run to completion before the main containers. There's also the notion of sidecar containers which are also initContainers, but have restartPolicy Always, so they're started before the main containers but don't complete before the others are started

[joshmalbrecht]

yeah initContainers are started and run to completion before the main containers. There's also the notion of sidecar containers which are also initContainers, but have restartPolicy Always, so they're started before the main containers but don't complete before the others are started

If we added a sidecar container under initContainers that was able to check the health of the database container in the postStart lifecycle using a script, would that be able to halt only the application container from attempting to start while the database container is starting? One thing to note, is that the k8s yaml that is generated from the podman generate kube puts the database and application containers in the same pod.

Our generated Kubernetes yaml file for reference: https://github.com/candlepin/candlepin/blob/main/dev-container/candlepin-deployment.yaml

[ygalblum]

Hi, There's a pull request addressing sidecar containers. While it is not yet merged, @joshmalbrecht is this what you are looking for?

[joshmalbrecht]

@ygalblum are you suggesting to run the application image as a sidecar container? I think that would work with the pull request you linked and the added restart functionality. I believe the sidecar container would first startup and fail and then continue retrying to restart until the database container in the same pod is healthy and the application can make a database connection.

I'm not very familiar with Kubernetes, so please let me know if this isn't correct, but it seems like the database container and the application container should be in two separate pods. To get similar functionality to the docker-compose depends_on, you would need an init container in the application pod that waits until the database pod is healthy before completing and allowing the application container to start. From doing some reading it seems like this is the Kubernetes equivalent of the docker compose depends_on functionality, so maybe this is the pattern we should follow?

Both of these options should work for this use case!

[ygalblum]

@joshmalbrecht first, my suggestion was the other way around (that is having the DB as a sidecar to the application). Having said that, I agree that ideally they should be in different pods. However, as you've mentioned podman does not handle dependencies between pods.

The second option you suggested does look promising. Podman will not start the main container until the init container has returned successfully (regular init containers, not the ones supported by the PR I linked). So, having an init container wait until the DB connection is up should do the trick.

Another option is to use Quadlet which allows you to define dependencies between containers (or pods) using Systemd. I actually wrote a blog post about it: https://www.redhat.com/sysadmin/multi-container-application-podman-quadlet which also has some code accompanying it: https://github.com/ygalblum/quadlet-demo along

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.