Worker Concurrency configuration on .ini does not work.

[debonzi]

Versions used: * Python: 3.9.4 * Celery: 5.0.5 * Pyramid: 2.0 * Pyramid-Celery: 4.0.0

Before openinig this issue a digged a lot and I think I kind of understand why and how it happens but I am not sure how to fix it and even if it is possible to fix without some ajustments in the celery project itself.

I will put my findings here and hopefuly we may find a way to address it.

How to reproduce:

Files

The files can also be found here: https://github.com/debonzi/pyramid-celery-issue-sample

setup.py

from setuptools import setup, find_packages

requires = [
    "celery==5.0.5",
    "pyramid==2.0",
    "pyramid-celery==4.0.0",
]
setup(
    name="demo",
    version="0.0.1",
    install_requires=requires,
    packages=find_packages(exclude=['tests']),
    entry_points={
        "paste.app_factory": [
            "main = demo:main",
        ],
    },
)

demo.py

from pyramid.config import Configurator


def main(global_config, **settings):
    """This function returns a Pyramid WSGI application."""
    with Configurator(settings=settings) as config:
        config.include("pyramid_celery")
        config.configure_celery(global_config["__file__"])

    return config.make_wsgi_app()

conf.ini

[app:main]
use = egg:demo

[server:main]
use = egg:waitress#main
listen = localhost:6543

[celery]
broker_url = redis://localhost:6379/1
worker_concurrency = 4

Install and Run

$ pip install -e .
$ celery -A pyramid_celery.celery_app worker --ini config.ini

Output:

 -------------- celery@Doomhammer v5.0.5 (singularity)
--- ***** -----
-- ******* ---- Linux-5.4.72-microsoft-standard-WSL2-x86_64-with-glibc2.31 2021-05-02 16:15:11
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app:         __main__:0x7f3e25b9e670
- ** ---------- .> transport:   redis://localhost:6379/1
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 8 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery

Concurrency is 8 (Number of CPU Cores on my machine) and not 4 as expected.

As fas as I could understand, celery 5 uses click and celery-pyramid uses the signal user_preload_options to configure celery using config_from_object method.

At this point it seems that celery has already loaded the click parameters into internal variables and since we didn't use --concurrency it behaves as if we have used --concurrency=0 (which will later translate to Number of CPU cores). Since command line parameters has precedency over config file parameters (config_from_object) celery will ignore the value from .ini` file and use the default value.

If you guys have a idea on how to fix (or work around it) or need any more information I might have I will be glad to provide it.

Cheers