`celery_worker` fixture name colides with `celery_worker` fixture from `celery.contrib.pytest`

[vesnikos]

pytest-celery contains a container based celery_worker fixture. celery_worker fixture also used for celery in celery.contrib.pytest.

When a test is called like this:

def test_a(celery_app, celery_worker):
    celery_app
    celery_worker
    assert True

celery_app is from celery.contrib.pytest and celery_worker is from pytest-celery (docker based)

[auvipy]

thanks for the report

[dgaus]

This is currently preventing us from adopting pytest-celery > 1 as the clash breaks our existing tests

[Nusnus]

@dgaus

This is currently preventing us from adopting pytest-celery > 1 as the clash breaks our existing tests

I have a list of pytest-celery items to attend to right after the upcoming Celery release. I apologize for the long delay. I have to prioritize release efforts above all at the moment.

TL;DR for this issue - the pytest-celery codebase and celery.contrib.pytest are different plugins and their fixtures should not be mixed. If it does automatically, check the installed version/imports.

This is currently preventing us from adopting pytest-celery > 1 as the clash breaks our existing tests

Is this the only challenge?

P.S The unit/integration tests use the infrastructure from celery.contrib.pytest. The smoke tests use the new Docker-based infra. You may also look there for use-case examples.

[dgaus]

@Nusnus No problem, thanks for the response. We are currently using celery.contrib.pytest for unit tests, and we use the celery_worker fixture often, ie

def test_webhook_task(self, mocker, celery_worker):
    ...

The problem is that if we install celery-pytest then the code tries to use the docker-based fixture, which obviously fails, so it breaks all our existing tests. I know you can specify a name attribute in pytest.fixture when defining one, but afaik there's no way of disambiguating the two. I cannot see any usages of celery_worker in the unit/integration tests so you are probably not having this issue.

[Nusnus]

@dgaus

@Nusnus No problem, thanks for the response. We are currently using celery.contrib.pytest for unit tests, and we use the celery_worker fixture often, ie

def test_webhook_task(self, mocker, celery_worker):
    ...

The problem is that if we install celery-pytest then the code tries to use the docker-based fixture, which obviously fails, so it breaks all our existing tests. I know you can specify a name attribute in pytest.fixture when defining one, but afaik there's no way of disambiguating the two. I cannot see any usages of celery_worker in the unit/integration tests so you are probably not having this issue.

If I had to guess, I'd say the bug is here: https://github.com/celery/pytest-celery/blob/main/src/pytest_celery/plugin.py

The second import overrides the fixtures with the same name probably. Renaming the docker based fixtures will solve it specifically but unless this is fixed in plugin.py, the issue remains at its core.

Personally I prefer to solve it correctly and avoid renaming unless it's urgent. LMK what you think.

[dgaus]

Not urgent at all

[piotrszyma]

We had a similar issue caused by this that happened when bumping from celery==5.3.6 to celery==5.5.3

1. Initial state, tests passing: celery==5.3.6 & celery[pytest] as dev dependency to enable fixtures defined in celery.contrib.pytest as described in celery docs
2. Bump to celery==5.5.3 caused celery[pytest] to update and install its dependency - pytest-celery
3. Now all tests using the celery_worker fixture started to fail because pytest-celery installed a conflicting fixture that silently overrode the one from celery.contrib.pytest

The fix? Changed the way we enable fixtures from celery.contrib.pytest:

1. added pytest_plugins = ("celery.contrib.pytest", ) to root conftest.py
2. removed celery[pytest] from dev dependencies