Postgres SSL error: decryption failed or bad record mac

[mfitzp]

There is an error running Wooey on a Postgres database backend (and using the same backend for celery):

[2015-08-12 13:19:33,616: ERROR/MainProcess] Unrecoverable error: OperationalError('SSL error: decryption failed or bad record mac\n',)
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
psycopg2.OperationalError: SSL error: decryption failed or bad record mac

This SO post mentions the error with another library and the solution suggests opening the DB connection within the job.The Heroku Postgres documentation makes it a bit clearer:

The Postgres connection is shared between more than one process or thread. Typical offenders are Resque workers or Unicorn. Be sure to correctly establish the PG connection after the fork or thread has initialized to resolve this issue.

[mfitzp]

This issue against celery mentions a similar problem: https://github.com/celery/celery/issues/1564

To close the engine after fork you can use:

from multiprocessing.util import register_after_fork

engine = create_engine(...)
register_after_fork(engine, engine.dispose)

[mfitzp]

Have played with this some more. According to the referred issue you can work around this with CELERYD_FORCE_EXECV = True but doing that requires billiard with C extensions which the only version to work on Python 3 is incompatible with current celery.

This is something that will probably go away as the setting above will become default. But is there anywhere we can apply the engine.dispose fix? Have looked around and it appears the bug is originating outside our code.

[strongant]

DatabaseError: (psycopg2.DatabaseError) SSL error: decryption failed or bad record mac,how to chose this question?

[strongant]

this is my comelete error info: [2016-01-20 16:25:47,933: ERROR/MainProcess] Task tasks.agent.checkAllLz[46bf3312-72cd-4074-a59a-6346413ee273] raised unexpected: SQLAlchemyError('(psycopg2.DatabaseError) SSL error: decryption failed or bad record mac\n',) Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 240, in trace_task R = retval = fun(_args, *_kwargs) File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 438, in protected_call return self.run(_args, *_kwargs) File "/home/devbwh/PycharmProjects/gslz/tasks/agent.py", line 617, in checkAllLz executeSingleTaskInfo(taskResultId) File "/home/devbwh/PycharmProjects/gslz/tasks/agent.py", line 533, in executeSingleTaskInfo scheduler.start() File "/usr/local/lib/python2.7/dist-packages/APScheduler-3.0.4-py2.7.egg/apscheduler/schedulers/background.py", line 30, in start BaseScheduler.start(self) File "/usr/local/lib/python2.7/dist-packages/APScheduler-3.0.4-py2.7.egg/apscheduler/schedulers/base.py", line 128, in start self._real_add_job(job, jobstore_alias, replace_existing, False) File "/usr/local/lib/python2.7/dist-packages/APScheduler-3.0.4-py2.7.egg/apscheduler/schedulers/base.py", line 729, in _real_add_job store.add_job(job) File "/usr/local/lib/python2.7/dist-packages/APScheduler-3.0.4-py2.7.egg/apscheduler/jobstores/sqlalchemy.py", line 84, in add_job self.engine.execute(insert) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1991, in execute return connection.execute(statement, _multiparams, *_params) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute return meth(self, multiparams, params) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement compiled_sql, distilled_params File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context context) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception exc_info File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause reraise(type(exception), exception, tb=exc_tb) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context context) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute cursor.execute(statement, parameters) DatabaseError: (psycopg2.DatabaseError) SSL error: decryption failed or bad record mac

[Chris7]

The error is coming from SQLAlchemy reusing connections from its parent process after the celery process has been forked. I would say upgrade everything since this is probably fixed by now.

[riccardocagnasso]

@Chris7 at celery 3.1.20, it is not

[kamal8]

I am having the same problem in celery 3.1.20 about to try the fixes mentioned above

[Chris7]

@riccardocagnasso what do you mean it is not? To me it's clear from the traceback that the fundamental issue is outside of wooey. If you have insight otherwise, I'd like to know it.

[mfitzp]

@Chris7 I'm not 100% sure this isn't something we are doing wrong. "The Postgres connection is shared between more than one process or thread." sounds like a bad thing. I never did figure out where that engine.dispose was supposed to go, so I'll take a further look.

[revmischa]

I'm experiencing this too, inside of Flask

[pietropino]

I had the same problem with Flask + uSWGI + psycopg2. I solved it by inserting the line:

lazy-apps = true

in the file uswgi.ini.

See also: here, here and here.