symfony-docs
symfony-docs copied to clipboard
symfony
[Messenger] Added section about "Consuming Messages in Docker"
Like announced on the syfomy-devs slack, some hints and tipps about running messenger:consume in a container environment.
My fork was heavy out of sync, so I needed three approaches for this PR :wink:
The approach I use is to run container for each queue.
FROM MyPHPContainer
USER www-data
ENTRYPOINT ["php", "bin/console", "messenger:consume", "--limit=10"]
CMD ["main"]
The approach I use is to run container for each queue.
You have to re-build a container for each queue on each deployment. Having a parameterized entrypoint in the same image makes much more sense. You only want to build one image for all environments per release (following twelve-factor-app concept).
The approach I use is to run container for each queue.
You have to re-build a container for each queue on each deployment. Having a parameterized entrypoint in the same image makes much more sense. You only want to build one image for all environments per release (following twelve-factor-app concept).
Yes, you build one image for all environments - one image for messenger and one for app. As per https://docs.docker.com/config/containers/multi-service_container/
It is generally recommended that you separate areas of concern by using one service per container. That service may fork into multiple processes (for example, Apache web server starts multiple worker processes). It’s ok to have multiple processes, but to get the most benefit out of Docker, avoid one container being responsible for multiple aspects of your overall application.
Image to container is like class to instance.
As long as the consumer is no seperate application but baked into your symfony website application, there should be no separate image, only another runtime configuration (entrypoint, env vars, etc).
@Warxcell the articel you've linked, explicitly describes multi-service containers, running multiple prozesses in parallel. That's exactly, what I try to avoid with this tipps.
In Symfony Slack there was a very active thread about this Pull Request. See https://symfony-devs.slack.com/archives/C8WHX21K7/p1621414331044800
Hopefully we can agree on some changes and propose them here? Thanks!
To pick this up from the Symfony Slack:
no need at all to define custom entrypoint, simply custom command is enough
If the inherited entrypoint of the base image doesn't support arguments, changing the command does nothing but handing over "wrong" parameters to a binary/script. Fortunately most official image entrypoints support overwriting the whole thing, but they don't do neccessarily.
ENTRYPOINT is the font-facing binary. CMD are the arguments to that binary.
Source: https://docs.docker.com/engine/reference/builder/#understand-how-cmd-and-entrypoint-interact
Mostly ENTRYPOINT is a shell like /bin/bash or a custom one that overwrites the whole "logic" of the entrypoint if non supported CMD is given (e.g. https://github.com/docker-library/php/blob/81ceb39449dc0d42fcecf455b6f372b0b5f3f426/8.0/buster/cli/docker-php-entrypoint)
As we cannot be sure whether the base image each and everyone uses supports entrypoint overwrite with simple CMD change, I'd like to promote having a separate (and very specific) entrypoint for this purpose.
It also follows single responsibility principle, by providing a separate and specific bootstrapping for the consumer process.
As long as the consumer is no seperate application but baked into your symfony website application, there should be no separate image, only another runtime configuration (entrypoint, env vars, etc).
If you build a separate consumer application, feel free to use whatever (single) entrypoint you like.
Basically, as you wrote - there is completely no need of using custom entrypoint for commands using standard approach, so I see no point for writing docs that says something different. KISS. Instead of this two paragraphs you can simply cover base usecase with CMD overwrite.
Same with limits tbh. It is something completely user-specific and there is no point in requiring from someone to restart their workers every N messages/every M minutes because it is only waste of resources. Only soft memory limit should be mentioned as obligatory in production environment, because not having it properly set up will result in app crash. But having it same as php native memory_limit won't help and won't prevent errors - you need to handle memory peeks during handling events, so this value must be php memory_limit decreased by peek memory usage during message handling.
Basically, as you wrote - there is completely no need of using custom entrypoint for commands using standard approach, so I see no point for writing docs that says something different. KISS. Instead of this two paragraphs you can simply cover base usecase with CMD overwrite.
Fair enough. Maybe I'm going to add a hint about ENTRYPOINT vs CMD.
Same with limits tbh. It is something completely user-specific and there is no point in requiring from someone to restart their workers every N messages/every M minutes because it is only waste of resources
I don't understand why restarting is a waste of resources (it frees them), but anyways.
The existing part about Deploying to Production tells axaclty that: "Don’t Let Workers Run Forever", so I've picked it up for Docker usage, as well. And as seamless restarting should be in the lifecycle of every containerized application, I shouldn't bother.
But having it same as php native memory_limit won't help and won't prevent errors
But having a too low memory_limit and wondering why the consumer crashes with OOM is also not fun to debug.
New insights from my side: if your use single image with FPM and Messenger, you cannot gracefully restart messenger, since FPM image overrides STOPSIGNAL SIGQUIT, so when you run same image with php bin/console messenger:consume $queue, you cannot stop it gracefully. You would need to use hacks to achieve that, or manually add a listener for SIGQUIT.
Filled also a issue for it, in case symfony decides to handle that case aswell.
New insights from my side: if your use single image with FPM and Messenger, you cannot gracefully restart messenger
It would happen if you use single container and run two processes inside - but no one is proposing it. You can use single fpm image with multiple containers.
That's exactly what I do. I have SINGLE image, and I run MULTIPLE containers from it. Did I ever said that I'm running 2 processes within single container? Did you even read what I wrote? I will quote myself
so when you run same image with php bin/console messenger:consume
which basically mean "run another container from same image, but different command which is php bin/console messenger:consume"
Symfony 4.4 is not supported anymore, you may want to rebase on top of 5.4 branch, otherwise please close the PR.
Thank you
