v2 beta: Django connector seems broken

[ewjoachim]

trafficstars

This is about the v2 beta testing.

From discord, from @paulzakin:

error connecting in 'pool-1': connection failed: could not receive data from server: Connection refused
error connecting in 'pool-1': connection failed: could not receive data from server: Connection refused
error connecting in 'pool-1': connection failed: could not receive data from server: Connection refused
error connecting in 'pool-1': connection failed: could not receive data from server: Connection refused
Traceback (most recent call last):
  File "/Users/code/server/worker.py", line 39, in <module>
    main()
  File "/Users/code/server/worker.py", line 35, in main
    x.run_worker()  # pyright: ignore [reportUnknownMemberType]
    ^^^^^^^^^^^^^^
  File "/Users/code/.venv/lib/python3.11/site-packages/procrastinate/app.py", line 270, in run_worker
    asyncio.run(f())
  File "/Users/foo/.local/share/mise/installs/python/3.11.7/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/Users/foo/.local/share/mise/installs/python/3.11.7/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/Users/foo/.local/share/mise/installs/python/3.11.7/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/code/.venv/lib/python3.11/site-packages/procrastinate/app.py", line 267, in f
    async with self.open_async():
  File "/Users/code/.venv/lib/python3.11/site-packages/procrastinate/utils.py", line 194, in __aenter__
    await self._open_coro()
  File "/Users/code/.venv/lib/python3.11/site-packages/procrastinate/psycopg_connector.py", line 170, in open_async
    await self._async_pool.open(wait=True)  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/code/.venv/lib/python3.11/site-packages/psycopg_pool/pool_async.py", line 382, in open
    await self.wait(timeout=timeout)
  File "/Users/code/.venv/lib/python3.11/site-packages/psycopg_pool/pool_async.py", line 169, in wait
    raise PoolTimeout(f"pool initialization incomplete after {timeout} sec")
psycopg_pool.PoolTimeout: pool initialization incomplete after 30.0 sec

I invoke the entry with poetry run python server/worker.py And the entry looks like this

import django

django.setup()

import procrastinate
from procrastinate.contrib.django import app

from server.env import EnvService
from server.services.db import ProcrastinateService

ENV = EnvService.get_env()
env = EnvService.get_env_variables(ENV)

connector = procrastinate.PsycopgConnector(
    kwargs={
        "host": env["POSTGRES_HOST"],
        "user": env["POSTGRES_USER"],
        "password": env["POSTGRES_PASSWORD"],
        "port": env["POSTGRES_PORT"],
        "dbname": env["POSTGRES_DB"],
        **EnvService.get_ssl(),
    }
)


def main() -> None:
    x = app.with_connector(connector)
    blueprints = ProcrastinateService.get_blueprints(show_warnings=True)
    for blueprint, namespace in blueprints:
        x.add_tasks_from(blueprint, namespace=namespace)
    x.run_worker()  # pyright: ignore [reportUnknownMemberType]


if __name__ == "__main__":
    main()

And the way I used to run it was poetry run procrastinate --app=server.worker.app worker And I used to define app = procrastinate.App(connector=connector) in the settings.py basically and then import that into worker.py Even more interesting: when all I do is change the version to v2 and leave all the code as it is (no django stuff changed), app defined in settings.py and then invoked in worker.py, I get that same error pool initialization incomplete after 30.0 sec.

So (my guess) when I invoke poetry run procrastinate --app=server.worker.app worker --concurrency=1 with the imported app on the new version, something changed.

[ewjoachim]

Suggestion: could it be that you're hitting this ? What version of psycopg are you using ?

[ewjoachim]

I'm trying the following script and it works as expected:

from __future__ import annotations

import django

django.setup()

from os import environ

import procrastinate
from procrastinate.contrib.django import app

connector = procrastinate.PsycopgConnector(
    kwargs={
        "host": environ["POSTGRES_HOST"],
        "user": environ["POSTGRES_USER"],
        "password": environ["POSTGRES_PASSWORD"],
        "port": environ["POSTGRES_PORT"],
        "dbname": environ["POSTGRES_DB"],
    }
)


def main() -> None:
    x = app.with_connector(connector)
    x.run_worker()  # pyright: ignore [reportUnknownMemberType]


if __name__ == "__main__":
    main()

And it works as expected with the following envvars:

export POSTGRES_HOST=$PGHOST POSTGRES_USER=$PGUSER POSTGRES_PASSWORD=$PGPASSWORD POSTGRES_PORT=5432 POSTGRES_DB=$PGDATABASE DJANGO_SETTINGS_MODULE=procrastinate_demos.demo_django.project.settings

(from the dev env, so PG variables are set to match the docker connection defined here.

[paulzakin]

OK this is great, I can look at this on Monday! I am using 3.1.18. What are you using? This will be a good way for us to sanity check.

[ewjoachim]

I'll triple check bit I'm using the dev env so all my versions are following the poetry lock file.

[ewjoachim]

Yep, 3.1.18

[paulzakin]

Ok, very odd. Now when I install 2.0.0b1 and test it out, I get this error

Traceback (most recent call last):
  File "/foo/erms/manage.py", line 23, in <module>
    main()
  File "/foo/erms/manage.py", line 19, in main
    execute_from_command_line(sys.argv)
  File "/foo/erms/.venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/foo/erms/.venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/foo/erms/.venv/lib/python3.11/site-packages/django/core/management/base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/foo/erms/.venv/lib/python3.11/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/foo/erms/.venv/lib/python3.11/site-packages/django/core/management/base.py", line 107, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/foo/erms/.venv/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 117, in handle
    executor = MigrationExecutor(connection, self.migration_progress_callback)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/foo/erms/.venv/lib/python3.11/site-packages/django/db/migrations/executor.py", line 18, in __init__
    self.loader = MigrationLoader(self.connection)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/foo/erms/.venv/lib/python3.11/site-packages/django/db/migrations/loader.py", line 58, in __init__
    self.build_graph()
  File "/foo/erms/.venv/lib/python3.11/site-packages/django/db/migrations/loader.py", line 276, in build_graph
    self.graph.validate_consistency()
  File "/foo/erms/.venv/lib/python3.11/site-packages/django/db/migrations/graph.py", line 198, in validate_consistency
    [n.raise_error() for n in self.node_map.values() if isinstance(n, DummyNode)]
  File "/foo/erms/.venv/lib/python3.11/site-packages/django/db/migrations/graph.py", line 198, in <listcomp>
    [n.raise_error() for n in self.node_map.values() if isinstance(n, DummyNode)]
     ^^^^^^^^^^^^^^^
  File "/foo/erms/.venv/lib/python3.11/site-packages/django/db/migrations/graph.py", line 60, in raise_error
    raise NodeNotFoundError(self.error_message, self.key, origin=self.origin)
django.db.migrations.exceptions.NodeNotFoundError: Migration procrastinate.0025_initial dependencies reference nonexistent parent node ('procrastinate', '0024_job_id_bigint')

It creates a <procrastinate migrations virtual path> folder in the root with migration 0025, which does not seem right.

Is this a weird thing on my machine?

[paulzakin]

And looking at https://github.com/procrastinate-org/procrastinate/releases/tag/2.0.0b1 - it looks like it is behind on main but a bunch of commits - is it force cutting a new beta release maybe with those commits?

[paulzakin]

And as a sanity check, I re-cloned out repo and resinstalled everything with v1.1.2 and that worked fine.

[ewjoachim]

I’m not on my computer to flesh out a complete answer but I’m pretty sure it’s a __pycache__ issue. We should open a different issue for that

[paulzakin]

Ya, no rush obviously - 1.1.2 is working great currently!

[ewjoachim]

Ok, the migration issue is solved in https://github.com/procrastinate-org/procrastinate/pull/941.

Now back to the connection issue. Anything you can share to help investigation is highly appreciated :)

[ewjoachim]

And looking at 2.0.0b1 (release) - it looks like it is behind on main but a bunch of commits - is it force cutting a new beta release maybe with those commits?

Sure ! I thought you had mentionned that it was the same to you testing on main and testing on a tag. Either way work for me.

https://github.com/procrastinate-org/procrastinate/releases/tag/2.0.0b2

[ewjoachim]

BTW, instead of:

connector = procrastinate.PsycopgConnector(
    kwargs={
        "host": environ["POSTGRES_HOST"],
        "user": environ["POSTGRES_USER"],
        "password": environ["POSTGRES_PASSWORD"],
        "port": environ["POSTGRES_PORT"],
        "dbname": environ["POSTGRES_DB"],
    }
)
x = app.with_connector(connector)

it might be worth trying:

x = app.with_connector(app.connector.get_worker_connector())

[paulzakin]

OK, excited to debug this!

Dumb question - I don't see a release on pypi for the next version?

Also, what is the difference between pypi and "main" of procrastinate. If it is the same I can just have poetry install from a git tag, rather than pypi, but I'd like to be sure :)

[ewjoachim]

Dumb question - I don't see a release on pypi for the next version?

Dammit, the workflow had failed. I https://github.com/procrastinate-org/procrastinate/actions/runs/7942921413/attempts/1 Second attempt worked: https://github.com/procrastinate-org/procrastinate/actions/runs/7942921413/job/21737931035

Also, what is the difference between pypi and "main" of procrastinate. If it is the same I can just have poetry install from a git tag, rather than pypi, but I'd like to be sure :)

Installing from main is poetry add git+https://github.com/procrastinate-org/procrastinate.git#main (you could remove main, it's the default branch) Installing from pypi is simply the normal install, provided I've done a release. Both are the same code I think the only difference is that installing from main, procrastinate's version will be detected through different means. When installing from main, pip freeze will report procrastinate @ git+https://github.com/procrastinate-org/procrastinate.git@7d15ef345ccf209b814776c201568d45f7979be9, and print(procrastinate.__version__) as well as importlib.metadata.metadata("procrastinate")["version"] will report 2.0.0b2.post4.dev0+7d15ef3,

[paulzakin]

Awesome, I'll check this out tomorrow now

[paulzakin]

Traceback (most recent call last):
  File "/Users/foo/manage.py", line 23, in <module>
    main()
  File "/Users/foo/manage.py", line 19, in main
    execute_from_command_line(sys.argv)
  File "/Users/foo/.venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/Users/foo/.venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 416, in execute
    django.setup()
  File "/Users/foo/.venv/lib/python3.11/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/Users/foo/.venv/lib/python3.11/site-packages/django/apps/registry.py", line 91, in populate
    app_config = AppConfig.create(entry)
                 ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/foo/.venv/lib/python3.11/site-packages/django/apps/config.py", line 123, in create
    mod = import_module(mod_path)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/me/.local/share/mise/installs/python/3.11.7/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/foo/.venv/lib/python3.11/site-packages/procrastinate/contrib/django/apps.py", line 12, in <module>
    from . import django_connector
  File "/Users/foo/.venv/lib/python3.11/site-packages/procrastinate/contrib/django/django_connector.py", line 13, in <module>
    from procrastinate.contrib.aiopg import aiopg_connector
  File "/Users/foo/.venv/lib/python3.11/site-packages/procrastinate/contrib/aiopg/__init__.py", line 3, in <module>
    from .aiopg_connector import AiopgConnector
  File "/Users/foo/.venv/lib/python3.11/site-packages/procrastinate/contrib/aiopg/aiopg_connector.py", line 8, in <module>
    import aiopg
ModuleNotFoundError: No module named 'aiopg'

[paulzakin]

Are you expected to install aiopg to get this to work? I thought we were switching to psycopg3 because it has a good async interface? I'm happy to install aiopg, just wanted to check.

[hyusetiawan]

want to point out 2 things:

1. from procrastinate.contrib.django import app throws an error, no variable named "app" is exported, there is "apps" though
2. from procrastinate.contrib.aiopg import AiopgConnector will throw a confusing error that aiopg module is not found, which made me think that it's procrastinate aiopg but in reality installing aiopg separately made it work

[ewjoachim]

@paulzakin

Are you expected to install aiopg to get this to work?

Ah indeed, I could have made it clearer in the docs, and you're right that it might not be the best choice:

It's all in https://github.com/procrastinate-org/procrastinate/blob/main/procrastinate/contrib/django/django_connector.py#L223

I've added the psycopg3 compat but I haven't removed Aiopg, however flawed may it be. When determining which one to use, I look at the is_psycopg3 bool from Django that tells me whether Django is configured to use psycopg2 or 3. If psycopg2, I use the psycopg2-based Aiopg, if psycopg3, I use psycopg3. If your procrastinate is aiming for aiopg, it means you're using a pre-psycopg3 django version.

But then, given that people might have psycopg2 installed but not aiopg, and that I'm going to make a different connection altogether, maybe I should rather:

• try to see which one of psycopg3 and aiopg is installed and use it
• provide a parameter to choose explicitly

I'll fix this and make it clearer in the docs.

[ewjoachim]

@hyusetiawan your comment makes me think that you're probably using the last stable release v1.1.2, while this issue is (sorry, unclearly) about the v2 beta.

As far as I can tell, the docs are configured to still point to 1.1.2, so maybe you're looking at the docs on the main branch ?

That being said, you're more than welcome to try and integrate v2.0.0b2 (it's worth re-reading the Django docs completely) and report any problem here. The v2 had the django integration completely re-done, and in theory it should be much smoother. That said, @paulzakin can attest that so far we're still experiencing road bumps.

[hyusetiawan]

ah okay, apologies for the confusion! it's stated in the title even that it's for v2 beta :)

[ewjoachim]

Well... It was not stated on the title at the time of your comment :)

[paulzakin]

Ok great @ewjoachim - let me know when you want to me to retest :)

[ewjoachim]

v2.0.0b3 is out, you can re-test.

[paulzakin]

Just tested - no dice :(

If you point me in the right direction in the library I can maybe see what logic is not being tripped or not?

Traceback (most recent call last):
  File "/foo/manage.py", line 23, in <module>
    main()
  File "/foo/manage.py", line 19, in main
    execute_from_command_line(sys.argv)
  File "/foo/.venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/foo/.venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 416, in execute
    django.setup()
  File "/foo/.venv/lib/python3.11/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/foo/.venv/lib/python3.11/site-packages/django/apps/registry.py", line 91, in populate
    app_config = AppConfig.create(entry)
                 ^^^^^^^^^^^^^^^^^^^^^^^
  File "/foo/.venv/lib/python3.11/site-packages/django/apps/config.py", line 123, in create
    mod = import_module(mod_path)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/foo/.local/share/mise/installs/python/3.11.7/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/foo/.venv/lib/python3.11/site-packages/procrastinate/contrib/django/apps.py", line 12, in <module>
    from . import django_connector
  File "/foo/.venv/lib/python3.11/site-packages/procrastinate/contrib/django/django_connector.py", line 13, in <module>
    from procrastinate.contrib.aiopg import aiopg_connector
  File "/foo/.venv/lib/python3.11/site-packages/procrastinate/contrib/aiopg/__init__.py", line 3, in <module>
    from .aiopg_connector import AiopgConnector
  File "/foo/.venv/lib/python3.11/site-packages/procrastinate/contrib/aiopg/aiopg_connector.py", line 8, in <module>
    import aiopg
ModuleNotFoundError: No module named 'aiopg'

[ewjoachim]

Haha. I forgot to plug the good ol' brain. If you look at the diff, it's facepalm material.

[ewjoachim]

https://github.com/procrastinate-org/procrastinate/releases/tag/2.0.0b4 https://pypi.org/project/procrastinate/2.0.0b4/

[paulzakin]

No worries - I'll look at tomorrow :)

[paulzakin]

Ok, good news! We have passed the aiopg problem. Bad news, other errors!

Running code like this leads

from procrastinate.contrib.django import app
app.configure_task(name="foo", queueing_lock="bar").defer(report_id="baz")

Leads to this error: 'Blueprint' object has no attribute 'configure_task'

And I used the PROCRASTINATE_ON_APP_READY method to load the blueprints. I know it is being run, because FOO is being printed in the terminal but no dice after that.

def on_app_ready(app: procrastinate.App) -> None:
    print("FOO")
    blueprints = ProcrastinateService.get_blueprints(show_warnings=True)
    for blueprint, namespace in blueprints:  
        app.add_tasks_from(blueprint, namespace=namespace)

Also, in the documentation: https://procrastinate.readthedocs.io/en/main/howto/django.html. I get an error with the load_tasks_from method that it does not exist. Are you sure it exists?

import procrastinate
def on_app_ready(app: procrastinate.App):
    app.load_tasks_from(some_blueprint)

Either way, forward progress!