cookiecutter-django

cookiecutter-django copied to clipboard

What happened?

When project gets created with Celery and run with Docker, it fails with multiple errors due not having the POSTGRES and REDIS environment variables set correctly in the env file.

The errors are initially "column django_celery_beat_periodictask.clocked_id does not exist". This gets fixed by adding the POSTGRES variables.

The next error is "/entrypoint: line 10: REDIS_URL: unbound variable" which is fixed if the REDIS_URL is added manually.

I had to add the "env_file" section to the local.yml to make it work:

  celerybeat:
    <<: *django
    image: seldon_klaw_local_celerybeat
    depends_on:
      - redis
      - postgres
      
    ports: []
    command: /start-celerybeat
    env_file:
      - ./.envs/.local/.django
      - ./.envs/.local/.postgres

What should've happened instead?

The env variables and docker config should be configured such that there are no errors on startup

Steps to reproduce

Install with following steps:

version [0.1.0]: 0.0.1
Select open_source_license:
1 - MIT
2 - BSD
3 - GPLv3
4 - Apache Software License 2.0
5 - Not open source
Choose from 1, 2, 3, 4, 5 [1]: 4
timezone [UTC]:
windows [n]:
use_pycharm [n]:
use_docker [n]: y
Select postgresql_version:
1 - 11.3
2 - 10.8
3 - 9.6
4 - 9.5
5 - 9.4
Choose from 1, 2, 3, 4, 5 [1]:
Select js_task_runner:
1 - None
2 - Gulp
Choose from 1, 2 [1]:
Select cloud_provider:
1 - AWS
2 - GCP
3 - None
Choose from 1, 2, 3 [1]: 3
custom_bootstrap_compilation [n]:
use_compressor [n]:
use_celery [n]: y
use_mailhog [n]:
use_sentry [n]:
use_whitenoise [n]: y
use_heroku [n]:
Select ci_tool:
1 - None
2 - Travis
3 - Gitlab
Choose from 1, 2, 3 [1]:
keep_local_envs_in_vcs [y]:
debug [n]:
 [WARNING]: You chose not to use a cloud provider, media files won't be served in production.
 [SUCCESS]: Project initialized, keep up the good work!

Then run docker-compose local.yml.

I checked today and I'm unable to reproduce this issue. The env file should already be included in the django anchor at the top of the local.yml file and we shouldn't need to repeat them:

https://github.com/pydanny/cookiecutter-django/blob/8fd0d42de0e7c3c8fea92bb4d1d2df3e4af6aa31/%7B%7Bcookiecutter.project_slug%7D%7D/local.yml#L8-L22

Someone else had a similar issue recently #2417, maybe it's a issue with Docker/docker-compose on a specific environment. Can you please share a bit more details about your setup? OS and various docker (compose/machine) versions that you have installed.

Sure.

> docker-compose version
docker-compose version 1.25.0, build b42d419
docker-py version: 4.1.0
CPython version: 3.7.4
OpenSSL version: OpenSSL 1.1.1d  10 Sep 2019

> lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.3 LTS
Release:        18.04
Codename:       bionic

> docker version
Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea838
 Built:             Wed Nov 13 07:29:52 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:29:19 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Let me know what other details work

I'm running macOS, and my version of docker-compose is different:

> docker-compose version
docker-compose version 1.25.2, build 698e2846
docker-py version: 4.1.0
CPython version: 3.7.5
OpenSSL version: OpenSSL 1.1.1d  10 Sep 2019

I can see in the compose release notes that they've updated the PyYAML version in 1.25.2.

Could you try to upgrade to 1.25.2 or higher please?

I've not had this issue in awhile, but how did you run your docker starts? Typically, when I just want to build through everything, I do:

docker-compose -f local.yml build && docker-compose -f local.yml run --rm django python manage.py migrate && docker-compose -f local.yml up

Did you do (could justify below reason #2):

docker-compose -f local.yml up --build

There are two possibilities:

1. Docker just messed up
2. This might be because celerybeat is tied to Django (the &django anchor in the compose file), so when Django starts after the entrypoint, the celery beat models haven't migrated. Or at least not fully.

Why do we actually need this line in django /entrypoint?

# N.B. If only .env files supported variable expansion...
export CELERY_BROKER_URL="${REDIS_URL}"

This can be set in compose file (?). Got hit because I prefer rabbitmq :)

@pySilver I made a PR at #2576 since I do, as well, for celery. The line is there simply because in production, to be "DRY", the REDIS_URL env needs to be set for cache and the CELERY_BROKER_URL. I think.

Related #3422 (and possible solution)