Auto-instrumentation breaks scheduling of periodic tasks in Django

[avodaqstephan]

Hi,

we are using the auto instrumentation for Django with the following dependencies.

Celery broker image: redis:6.0.4-alpine3.11
Scheduler: Django

redis==3.5.3
Django==3.2.2
psycopg2-binary==2.8.6
autodynatrace==1.0.81
celery==5.1.2
django-celery-beat==2.2.0

We do use tasks in Django. We don't have any issues running tasks manually using the django admin. But defining a periodic task using the django scheduler on the admin page leads to the problem that celery never receives the task.

If we disable dynatrace by un-setting AUTOWRAPT_BOOTSTRAP: "autodynatrace", everything works fine.

Looking into the celery logs, there's the following error message:

[2021-07-15 06:29:16,358: INFO/Beat] beat: Starting...
[2021-07-15 06:29:16,377: ERROR/Beat] Process Beat
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/dist-packages/autodynatrace/wrappers/psycopg2/wrapper.py", line 24, in execute
    return super(DynatraceCursor, self).execute(query, vars)
psycopg2.OperationalError: SSL error: tlsv1 alert protocol version

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/billiard/process.py", line 327, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.8/dist-packages/celery/beat.py", line 721, in run
    self.service.start(embedded_process=True)
  File "/usr/local/lib/python3.8/dist-packages/celery/beat.py", line 636, in start
    humanize_seconds(self.scheduler.max_interval))
  File "/usr/local/lib/python3.8/dist-packages/kombu/utils/objects.py", line 29, in __get__
    return super().__get__(instance, owner)
  File "/usr/lib/python3.8/functools.py", line 967, in __get__
    val = self.func(instance)
  File "/usr/local/lib/python3.8/dist-packages/celery/beat.py", line 680, in scheduler
    return self.get_scheduler()
  File "/usr/local/lib/python3.8/dist-packages/celery/beat.py", line 671, in get_scheduler
    return symbol_by_name(self.scheduler_cls, aliases=aliases)(

Additional env variables we set on start-up are:

AUTOWRAPT_BOOTSTRAP: "autodynatrace"
AUTODYNATRACE_FORKABLE: True
AUTODYNATRACE_CAPTURE_HEADERS: True

Kind regards, Stephan

[avodaqstephan]

Can I help with additional information to get this solved? We are currently running blind and wanted to move to production soon.

[dlopes7]

Hello,

I am sorry this one is a bit hard to understand

The error psycopg2.OperationalError: SSL error: tlsv1 alert protocol version

Doesn't seem to be related to celery instrumentation, as it is thrown when the database query is trying to run

If you disable the psycopg2 instrumentation does the error go away?

Ie, set the environment variable:

AUTODYNATRACE_INSTRUMENT_PSYCOPG2=False

[avodaqstephan]

Hi @dlopes7,

we were able to solve it by separating our startup command from

celery -A website worker -E --beat --scheduler django --loglevel=info

to

celery -A website beat -l info -S django &
celery -A website worker -l info &

So executing beats and a celery worker within one command solved the problem for us. The psycopg2 error also disappeared. So beats together with workerseems to be the cause of that issue.

I'd close the issue or should I leave it open because its fixed by a "work around"?