ara with postgresql: The version of psycopg2 (2.9.1) must be <2.9 due to an incompatibility with Django 2.2

[dmsimard]

What is the issue ?

Noticed an exception from a CI job running testing the postgresql database backend: https://dashboard.zuul.ansible.com/t/ansible/build/36783777b9d342e7ad134def9bb67254

This seems to be a recently documented issue between django and psycopg2: https://github.com/psycopg/psycopg2/issues/1293

2021-08-03 21:24:15,368 DEBUG ara.plugins.callback.default: v2_playbook_on_start
2021-08-03 21:24:15,369 DEBUG ara.plugins.callback.default: Ignoring argument: extra_vars
2021-08-03 21:24:15,371 DEBUG urllib3.connectionpool: Starting new HTTP connection (1): localhost:34093
2021-08-03 21:24:15,432 DEBUG urllib3.connectionpool: http://localhost:34093 "POST /api/v1/playbooks HTTP/1.1" 201 1143
2021-08-03 21:24:15,433 DEBUG ara.clients.http: HTTP 201: post on /api/v1/playbooks
2021-08-03 21:24:15,434 DEBUG ara.plugins.callback.default: File not in cache, getting or creating: /home/zuul/.ara-tests/git/ara/tests/integration/smoke.yaml
Internal Server Error: /api/v1/files
Traceback (most recent call last):
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/rest_framework/mixins.py", line 18, in create
    serializer.is_valid(raise_exception=True)
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/rest_framework/serializers.py", line 220, in is_valid
    self._validated_data = self.run_validation(self.initial_data)
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/rest_framework/serializers.py", line 419, in run_validation
    value = self.to_internal_value(data)
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/rest_framework/serializers.py", line 476, in to_internal_value
    validated_value = field.run_validation(primitive_value)
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/rest_framework/fields.py", line 799, in run_validation
    return super().run_validation(data)
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/rest_framework/fields.py", line 568, in run_validation
    value = self.to_internal_value(data)
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/ara/api/fields.py", line 74, in to_internal_value
    sha1=sha1, defaults={"sha1": sha1, "contents": zlib.compress(contents)}
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
    return self.get(**kwargs), False
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/django/db/models/query.py", line 402, in get
    num = len(clone)
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
    self._fetch_all()
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
    return list(result)
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1554, in cursor_iter
    for rows in iter((lambda: cursor.fetchmany(itersize)), sentinel):
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1554, in <lambda>
    for rows in iter((lambda: cursor.fetchmany(itersize)), sentinel):
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/django/db/utils.py", line 96, in inner
    return func(*args, **kwargs)
  File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/django/db/backends/postgresql/utils.py", line 6, in utc_tzinfo_factory
    raise AssertionError("database connection isn't set to UTC")
AssertionError: database connection isn't set to UTC

[dmsimard]

There was a fix proposed to django 2.2 LTS and the actual issue is best summarized here: https://github.com/django/django/pull/14530#issuecomment-862961533

The issue doesn't exist in newer versions of django but ara hasn't moved to 3.2 LTS yet (released in april 2021).

The fix will not be merged and Django maintainers adjusted the documentation to suggest that users should pin psycopg2 < 2.9 when using 2.2.x.

[dmsimard]

Sent a PR to pin psycopg2 in the setup.cfg extras: https://github.com/ansible-community/ara/pull/321

It's not a definitive fix because this will only help users who run pip install ara[server,postgresql] which, I suspect, is a minority... but hey, it's better than nothing.

pyscopg2 could end up being installed unpinned from PyPi or once psycopg2 2.9 lands in linux distribution repositories where the version of django is 2.2.

Other options to explore:

• Upgrade to django 3.2 (even so, users could continue running into the issue for past versions of ara) <-- not in the near future
• We need to improve docs around how to run with mysql and postgresql, this special requirement could be added as a warning.
• Determine if the version of psycopg2 is 2.9 when running django 2.2 and return a friendlier exception like here: https://github.com/ansible-community/ara/blob/03f6c100e790a309472bfa1447d9ef6af49d7b2c/ara/server/main.py#L46-L56
• Catch the AssertionError from where it was triggered: File "/home/zuul/.ara-tests/virtualenv/lib64/python3.6/site-packages/ara/api/fields.py", line 74, in to_internal_value and return a friendlier message with a warning about the version of psycopg2.

[Nold360]

This is also an issue in the lastest docker image:

[ara] Using settings file: /opt/ara/settings.yaml
Traceback (most recent call last):
  File "/usr/local/bin/ara-manage", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/ara/server/__main__.py", line 56, in main
    raise Psycopg2VersionException(version=psycopg_version)
ara.setup.exceptions.Psycopg2VersionException: The version of psycopg2 (2.9.1) must be <2.9 due to an incompatibility with Django 2.2.

[dmsimard]

Hi @Nold360, it is a known issue in Fedora because the version of psycopg is too new for the version of django.

I will address it by recovering it from PyPI instead but in the meantime, the centos8-stream-pypi-latest tag should work.

[jbpratt]

Hi @dmsimard ,

Running into this trying to get a deployment setup on K8s as a POC for my team. I'm using postgres:11 and ara-api:latest. Is centos8-stream-pypi-latest a tag for the ara-api container image? I'm unable to find it if so.

[ara] Using settings file: /opt/ara/settings.yaml
Traceback (most recent call last):
  File "/usr/local/bin/ara-manage", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/ara/server/__main__.py", line 56, in main
    raise Psycopg2VersionException(version=psycopg_version)
ara.setup.exceptions.Psycopg2VersionException: The version of psycopg2 (2.9.1) must be <2.9 due to an incompatibility with Django 2.2.

[dmsimard]

@jbpratt centos8-stream-pypi-latest is safe to use -- it's still the ara API server (and built-in reporting interface) but it's built on top of centos-stream8 instead of fedora35. There's otherwise a list of available tags in About tags in the image repo: https://quay.io/repository/recordsansible/ara-api

The version of psycopg2 in centos-stream8 is older and so doesn't run into the issue.

I'm leaving this issue opened so we can eventually fix the fedora image which probably implies installing psycopg2 pinned from PyPI instead of from distribution packages.

[bendem]

I see that master has been on django 3 for a few months, is there any plans to release it soon or should I go ahead and apply the provided workarounds?