Intermittent python: can't open file '/app/manage.py': [Errno 2] No such file or directory

[scheung38]

Expected behavior

k8s manifest files to be generated

Actual behavior

Intermittently getting:

deployment/django: container arthur-production-django terminated with exit code 2 pod/django-697ddc88b9-hks6r: container arthur-production-django terminated with exit code 2 [django-697ddc88b9-hks6r arthur-production-django] python: can't open file '/app/manage.py': [Errno 2] No such file or directory deployment/django failed. Error: container arthur-production-django terminated with exit code 2.

• Skaffold version: v1.38.0
• Operating system: Mac OSX 12.4
• Installed via: brew install skaffold
• Contents of skaffold.yaml:

skaffold.yaml:

apiVersion: skaffold/v2beta28
kind: Config
metadata:
  name: skaffold
deploy:
  kubectl:
    manifests:
    - django-claim0-persistentvolumeclaim.yaml
    - django-claim1-persistentvolumeclaim.yaml
    - django-deployment.yaml
    - django-service.yaml
    - envs--production--django-configmap.yaml
    - envs--production--mongodb-configmap.yaml
    - mongo-deployment.yaml
    - mongo-service.yaml
    - production-mongodb-data-persistentvolumeclaim.yaml
    - production-traefik-persistentvolumeclaim.yaml
    - traefik-deployment.yaml
    - traefik-service.yaml

docker-compose-prod.yaml:

version: '3'

volumes:
  production_mongodb_data:
  production_traefik: {}

services:
  django:
    build:
      context: .
      dockerfile: ./compose/production/django/Dockerfile
    image: arthuracr.azurecr.io/arthur_production_django:v1.0.0
    
    container_name: arthur_production_django
    # platform: linux/x86_64
    depends_on:
      # - postgres
      # - redis
      - mongo
    volumes:
      - .:/app:z
      - ../RTL_data/_PAF_Data_samples/PAF_MAIN_FILE:/uploaddata

    env_file:
      - ./.envs/.production/.django
      # - ./.envs/.production/.postgres
      - ./.envs/.production/.mongodb
    ports:
      - "8000:8000"
      - "3000:3000"
    command: /start
    stdin_open: true
    tty: true

  mongo:
    image: arthuracr.azurecr.io/mongo:5.0.6
    container_name: "mongo"
    restart: always
    env_file:
      - ./.envs/.production/.mongodb
    environment:
      - MONGO_INITDB_ROOT_USERNAME=XXXX
      - MONGO_INITDB_ROOT_PASSWORD=XXXX
      - MONGO_INITDB_DATABASE= XXXX 
      - MONGO_INITDB_USERNAME=XXXX 
      - MONGO_INITDB_PASSWORD=XXXX 
    volumes:
      - production_mongodb_data:/data/db
      # - ${PWD}/_data/mongo:/data/db
      # - ${PWD}/docker/_mongo/fixtures:/import
      # - ${PWD}/docker/_mongo/scripts/init.sh:/docker-entrypoint-initdb.d/setup.sh
    ports:
      - 27017:27017

  traefik:
    build:
      context: .
      dockerfile: ./compose/production/traefik/Dockerfile
    image: arthuracr.azurecr.io/arthur_production_traefik:v2.2.11
    container_name: arthur_production_traefik
    depends_on:
      - django
    volumes:
      - production_traefik:/etc/traefik/acme:z
    ports:
      - "0.0.0.0:80:80"
      - "0.0.0.0:443:443"
    # command: --api.insecure=true --providers.docker

Steps to reproduce the behavior

1. a clonable repository with the sample skaffold project
2. skaffold dev --default-repo $MY_ACR
3. ...

Listing files to watch...
Generating tags...
Checking cache...
Tags used in deployment:
Starting deploy...
 - persistentvolumeclaim/django-claim0 created
 - persistentvolumeclaim/django-claim1 created
 - deployment.apps/django created
 - service/django created
 - configmap/envs--production--django created
 - configmap/envs--production--mongodb created
 - deployment.apps/mongo created
 - service/mongo created
 - persistentvolumeclaim/production-mongodb-data created
 - persistentvolumeclaim/production-traefik created
 - deployment.apps/traefik created
 - service/traefik created
Waiting for deployments to stabilize...
 - deployment/django: creating container arthur-production-django
    - pod/django-697ddc88b9-hks6r: creating container arthur-production-django
 - deployment/mongo: creating container mongo
    - pod/mongo-7775588866-n6zqn: creating container mongo
 - deployment/traefik: creating container arthur-production-traefik
    - pod/traefik-9668b69fc-s9cpb: creating container arthur-production-traefik
 - deployment/traefik is ready. [2/3 deployment(s) still pending]
 - deployment/django: container arthur-production-django terminated with exit code 2
    - pod/django-697ddc88b9-hks6r: container arthur-production-django terminated with exit code 2
      > [django-697ddc88b9-hks6r arthur-production-django] python: can't open file '/app/manage.py': **[Errno 2] No such file or directory**
 - deployment/django failed. Error: container arthur-production-django terminated with exit code 2.
Cleaning up...
 - persistentvolumeclaim "django-claim0" deleted
 - persistentvolumeclaim "django-claim1" deleted
 - deployment.apps "django" deleted
 - service "django" deleted
 - configmap "envs--production--django" deleted
 - configmap "envs--production--mongodb" deleted
 - deployment.apps "mongo" deleted
 - service "mongo" deleted
 - persistentvolumeclaim "production-mongodb-data" deleted
 - persistentvolumeclaim "production-traefik" deleted
 - deployment.apps "traefik" deleted
 - service "traefik" deleted
1/3 deployment(s) failed

Is this related to periodSeconds or timeoutSeconds to specified? That django is waiting for MongoDB when it was not ready etc? livelinessProbe or readinessProbe is the solution ?