Cronjob documentation

[kousu]

It wasn't obvious to me in installing Sous-Chef that there were cronjobs needed. INSTALL.md doesn't say anything about them. And though eventually cronjob.adoc was pointed out to me, I just found out today that it is missing processscheduledstatuschange.

Apparently the correct cronjob setup is

@daily /usr/bin/docker exec -it souschef_web_1 python src/manage.py generateorders $(date +%Y-%m-%d) --days 90
@daily /usr/bin/docker exec -it souschef_web_1 python src/manage.py processscheduledstatuschange

I would like to request cronjob.adoc be deleted and merged into INSTALL.md.

Also it's probably better to write those jobs as

@daily (cd path/to/sous-chef; /usr/bin/docker-compose exec web python src/manage.py generateorders $(date +%Y-%m-%d) --days 90)
@daily (cd path/to/sous-chef; /usr/bin/docker-compose exec web python src/manage.py processscheduledstatuschange)

to handle the case of multiple parallel Sous-Chefs on a single host (which is, afterall, why they're containerized isn't it?)

The install process would be more reliable if there wasn't anything manual at all like this. Both http://www.celeryproject.org/ and https://github.com/Tivix/django-cron can be used to do cron-like things but driven by Django itself; maybe they are useful?

[kousu]

Relatedly, the first cronjob needs quoting. It should be

@daily /usr/bin/docker exec -it souschef_web_1 python src/manage.py generateorders $(date +\%Y-\%m-\%d) --days 30

[kousu]

Also, you need to make sure the user that's run as has permissions to docker, either via sudo or by being in the docker group; a note to this effect in the docs would help.

[lingxiaoyang]

@kousu Thanks for the heads-up! I never tested this part during my dev. I'll check it out according to your comment and update the docs.

Celery and django-cron are overkill for our purpose. We expect it being executed only once a day and we don't expect to re-program the job later.

[kousu]

There was another bug with these: they say to run docker exec -it but when run by cron that generates the poorly explained "the input device is not a TTY". The answer there says that any of -i, -t or should work, but not -it. I tried that and now the cronjobs are working on DEV, hooray. I think the root of the problem here is Docker and its poor docs.

The manpage describes

       -i, --interactive[=false]
           Keep STDIN open even if not attached

       -t, --tty[=false]
           Allocate a pseudo-TTY

which makes it sound like -it should behave like running the command inside of screen: it will think its connected to a terminal (via isatty()) and have an open stdin even when its not either of those things.

We can simulate the effect of crond with </dev/null:

sfl@sous-chef-test:~/sous-chef$ docker exec -it souschef_web_1 ls </dev/null
the input device is not a TTY

As the SO answer says, using any combination except for -it together works:

sfl@sous-chef-test:~/sous-chef$ docker exec -i souschef_web_1 ls </dev/null
CONTRIBUTING.md
Dockerfile
INSTALL.md
LICENSE
README.adoc
deploy_latest.sh
docker-compose.override.yml
docker-compose.prod.yml
docker-compose.travis.yml
docker-compose.yml
docs
mock_addresses.csv
mock_clients.csv
mock_meals.csv
mock_orders.csv
mock_relationships.csv
requirements.txt
src
tools

sfl@sous-chef-test:~/sous-chef$ docker exec -t souschef_web_1 ls </dev/null
CONTRIBUTING.md		     docker-compose.prod.yml	mock_orders.csv
Dockerfile		     docker-compose.travis.yml	mock_relationships.csv
INSTALL.md		     docker-compose.yml		requirements.txt
LICENSE			     docs			src
README.adoc		     mock_addresses.csv		tools
deploy_latest.sh	     mock_clients.csv
docker-compose.override.yml  mock_meals.csv

sfl@sous-chef-test:~/sous-chef$ docker exec souschef_web_1 ls </dev/null
CONTRIBUTING.md
Dockerfile
INSTALL.md
LICENSE
README.adoc
deploy_latest.sh
docker-compose.override.yml
docker-compose.prod.yml
docker-compose.travis.yml
docker-compose.yml
docs
mock_addresses.csv
mock_clients.csv
mock_meals.csv
mock_orders.csv
mock_relationships.csv
requirements.txt
src
tools

I don't understand what docker is doing here. What you wrote should work, to me. It smells like a docker bug, but in the meantime can you update the cronjob.adoc to remove the -it --- just using works for me, but maybe -i is a little , and leave a comment for yourselves somewhere that docker + cron glitch out when combined like this?

Thank you for all your diligent work.

[johnsaigle]

^ Ran into the above issue again today with sous-chef. An update would be greatly appreciated!

[kousu]

Notice that #822 means we need a third cronjob now:

python3 manage.py setordersdelivered $(date --date="yesterday"  +"%Y-%m-%d")

and remember to double-quote those percent-signs when you put them into the crontab! because cron adds even more obscure quoting rules on top of shell!

[kousu]

For reference, what I actually have installed right now in the crontab for the user docker is run as:

# the 2>&1 | tees are for debugging: you can always inspect the last run with them
# (aka |& tee ?)
@daily /usr/bin/docker exec souschef_web_1 python src/manage.py generateorders $(date +\%Y-\%m-\%d) --days 100 2>&1 | tee /tmp/generateorders
@daily /usr/bin/docker exec souschef_web_1 python src/manage.py processscheduledstatuschange 2>&1 | tee /tmp/processscheduledstatuschange
@daily /usr/bin/docker exec souschef_web_1 python src/manage.py setordersdelivered $(date --date="yesterday" +"\%Y-\%m-\%d") 2>&1 | tee /tmp/setordersdelivered

[kousu]

Another issue: the example cronjobs suggest

docker exec -it souschef_db_1 mysqldump -uroot -p123456 feast > souschef_data.sql

this is insecure, especially when used with cron, because cron in its default config emails the command line if there's any output to the user running the job (aka their ~/.forward file, usually, if they're a daemon user and you've set that up properly).

I think this bug needs an edit to the docker config; db needs to gain a ~/.my.cnf file, or maybe docker-compose.yml needs to set MYSQL_PWD.