Abort in GC on exit

[sam-s]

I have a large long running (many hours) process which is started daily. When it terminates, I see this on the screen on which it was started (even though I start it with nohup):

*** Error in `/home/sds/.virtualenvs/algorisk/bin/python3': free(): invalid pointer: 0x00007f6be56c9760 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81499)[0x7f6be45d0499]
/usr/lib64/libpython3.6m.so.1.0(+0xd26ff)[0x7f6be53136ff]
/usr/lib64/libpython3.6m.so.1.0(+0xd1e45)[0x7f6be5312e45]
/usr/lib64/libpython3.6m.so.1.0(+0xd1f4f)[0x7f6be5312f4f]
/usr/lib64/libpython3.6m.so.1.0(+0xf0c73)[0x7f6be5331c73]
/usr/lib64/libpython3.6m.so.1.0(+0xee312)[0x7f6be532f312]
/usr/lib64/libpython3.6m.so.1.0(+0xbe3d2)[0x7f6be52ff3d2]
/usr/lib64/libpython3.6m.so.1.0(+0xd1f94)[0x7f6be5312f94]
/usr/lib64/libpython3.6m.so.1.0(+0xf0c73)[0x7f6be5331c73]
/usr/lib64/libpython3.6m.so.1.0(+0xd1f94)[0x7f6be5312f94]
/usr/lib64/libpython3.6m.so.1.0(+0xf25cd)[0x7f6be53335cd]
/usr/lib64/libpython3.6m.so.1.0(+0x17f97a)[0x7f6be53c097a]
/usr/lib64/libpython3.6m.so.1.0(+0x18005a)[0x7f6be53c105a]
/usr/lib64/libpython3.6m.so.1.0(PyGC_Collect+0x20)[0x7f6be541a5a0]
/usr/lib64/libpython3.6m.so.1.0(Py_FinalizeEx+0x62)[0x7f6be5411382]
/usr/lib64/libpython3.6m.so.1.0(Py_Main+0x52e)[0x7f6be541999e]
/home/sds/.virtualenvs/algorisk/bin/python3(main+0x119)[0x400b19]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f6be4571445]
/home/sds/.virtualenvs/algorisk/bin/python3[0x400ca1]
======= Memory map: ========
00400000-00401000 r-xp 00000000 ca:01 526292                             /home/sds/.virtualenvs/algorisk/bin/python3
00601000-00602000 rw-p 00001000 ca:01 526292                             /home/sds/.virtualenvs/algorisk/bin/python3
01760000-5e236000 rw-p 00000000 00:00 0                                  [heap]
7f69d0000000-7f69d0204000 rw-p 00000000 00:00 0 
7f69d0204000-7f69d4000000 ---p 00000000 00:00 0 
7f69d4000000-7f69d42c2000 rw-p 00000000 00:00 0 
7f69d42c2000-7f69d8000000 ---p 00000000 00:00 0 
7f69d8000000-7f69d8204000 rw-p 00000000 00:00 0 
7f69d8204000-7f69dc000000 ---p 00000000 00:00 0 
7f69df122000-7f69df1a2000 rw-p 00000000 00:00 0 
7f69df222000-7f69df262000 rw-p 00000000 00:00 0 
7f69df322000-7f69df362000 rw-p 00000000 00:00 0 
7f69df462000-7f69df4a2000 rw-p 00000000 00:00 0 
7f69df562000-7f69df5a2000 rw-p 00000000 00:00 0 
7f69df5e2000-7f69df722000 rw-p 00000000 00:00 0 
7f69df762000-7f69df7a2000 rw-p 00000000 00:00 0 
7f69df822000-7f69df922000 rw-p 00000000 00:00 0 
7f69df9a2000-7f69df9e2000 rw-p 00000000 00:00 0 
7f69dfa22000-7f69dfa62000 rw-p 00000000 00:00 0 
7f69dfaa2000-7f69dfb62000 rw-p 00000000 00:00 0 
7f69dfc22000-7f69dfc62000 rw-p 00000000 00:00 0 
7f69dfca2000-7f69dfd22000 rw-p 00000000 00:00 0 
7f69dfda2000-7f69dfde2000 rw-p 00000000 00:00 0 
7f69dfe22000-7f69dfe62000 rw-p 00000000 00:00 0 
7f69dff22000-7f69dff62000 rw-p 00000000 00:00 0 
7f69dffa2000-7f69e0062000 rw-p 00000000 00:00 0 
7f69e00e2000-7f69e0162000 rw-p 00000000 00:00 0 
7f69e01a2000-7f69e0262000 rw-p 00000000 00:00 0 
7f69e02a2000-7f69e0322000 rw-p 00000000 00:00 0 
7f69e03a2000-7f69e0422000 rw-p 00000000 00:00 0 
7f69e0462000-7f69e04a2000 rw-p 00000000 00:00 0 
7f69e04e2000-7f69e0522000 rw-p 00000000 00:00 0 
7f69e0562000-7f69e05e2000 rw-p 00000000 00:00 0 
7f69e06e2000-7f69e0722000 rw-p 00000000 00:00 0 
7f69e07e2000-7f69e0822000 rw-p 00000000 00:00 0 
7f69e08e2000-7f69e0962000 rw-p 00000000 00:00 0 
7f69e09a2000-7f69e09e2000 rw-p 00000000 00:00 0 
7f69e0b62000-7f69e0be2000 rw-p 00000000 00:00 0 
7f69e0c22000-7f69e0c62000 rw-p 00000000 00:00 0 
7f69e1595000-7f69e1715000 rw-p 00000000 00:00 0 
7f69e1755000-7f69e1895000 rw-p 00000000 00:00 0 
7f69e1915000-7f69e19d5000 rw-p 00000000 00:00 0 
7f69e1a15000-7f69e1a55000 rw-p 00000000 00:00 0 
7f69e1a95000-7f69e1b15000 rw-p 00000000 00:00 0 
7f69e1b55000-7f69e1bd5000 rw-p 00000000 00:00 0 
7f69e1c55000-7f69e1cd5000 rw-p 00000000 00:00 0 
7f69e1d15000-7f69e1d95000 rw-p 00000000 00:00 0 
7f69e1e15000-7f69e1e55000 rw-p 00000000 00:00 0 
7f69e1e95000-7f69e1ed5000 rw-p 00000000 00:00 0 
7f69e1f15000-7f69e1f55000 rw-p 00000000 00:00 0 
7f69e2015000-7f69e2095000 rw-p 00000000 00:00 0 
7f69e20d5000-7f69e2115000 rw-p 00000000 00:00 0 
7f69e2155000-7f69e21d5000 rw-p 00000000 00:00 0 
7f69e2215000-7f69e22d5000 rw-p 00000000 00:00 0 
7f69e2355000-7f69e2455000 rw-p 00000000 00:00 0 
7f69e2495000-7f69e24d5000 rw-p 00000000 00:00 0 
7f69e2515000-7f69e2555000 rw-p 00000000 00:00 0 
7f69e2715000-7f69e2755000 rw-p 00000000 00:00 0 
7f69e2795000-7f69e2815000 rw-p 00000000 00:00 0 
7f69e2895000-7f69e2955000 rw-p 00000000 00:00 0 
7f69e2995000-7f69e2a55000 rw-p 00000000 00:00 0 
7f69e2a95000-7f69e2ad5000 rw-p 00000000 00:00 0 
7f69e2b55000-7f69e2b95000 rw-p 00000000 00:00 0 
7f69e2bd5000-7f69e2c15000 rw-p 00000000 00:00 0 
7f69e2c55000-7f69e2cd5000 rw-p 00000000 00:00 0 
7f69e2d15000-7f69e2dd5000 rw-p 00000000 00:00 0 
7f69e2e15000-7f69e2e55000 rw-p 00000000 00:00 0 
7f69e2e95000-7f69e2f15000 rw-p 00000000 00:00 0
7f69e2f55000-7f69e2f95000 rw-p 00000000 00:00 0 
7f69e2fd5000-7f69e3015000 rw-p 00000000 00:00 0 
7f69e3055000-7f69e3095000 rw-p 00000000 00:00 0 
7f69e30d5000-7f69e3195000 rw-p 00000000 00:00 0 
7f69e31d5000-7f69e3295000 rw-p 00000000 00:00 0 
7f69e3355000-7f69e3395000 rw-p 00000000 00:00 0 
7f69e3415000-7f69e3455000 rw-p 00000000 00:00 0 
7f69e3495000-7f69e3555000 rw-p 00000000 00:00 0 
7f69e3595000-7f69e35d5000 rw-p 00000000 00:00 0 
7f69e3655000-7f69e3695000 rw-p 00000000 00:00 0 
7f69e3795000-7f69e3855000 rw-p 00000000 00:00 0 
7f69e3895000-7f69e38d5000 rw-p 00000000 00:00 0 
7f69e3915000-7f69e3995000 rw-p 00000000 00:00 0 
7f69e39d5000-7f69e3a55000 rw-p 00000000 00:00 0 
7f69e3b15000-7f69e3b95000 rw-p 00000000 00:00 0 
7f69e3d15000-7f69e3d55000 rw-p 00000000 00:00 0 
7f69e3d95000-7f69e3f55000 rw-p 00000000 00:00 0 
7f69e3f95000-7f69e3fd5000 rw-p 00000000 00:00 0 
7f69e4015000-7f69e4055000 rw-p 00000000 00:00 0 
7f69e40d5000-7f69e4155000 rw-p 00000000 00:00 0 
7f69e41d5000-7f69e4295000 rw-p 00000000 00:00 0 
7f69e4355000-7f69e43d5000 rw-p 00000000 00:00 0 
7f69e4455000-7f69e4495000 rw-p 00000000 00:00 0 
7f69e4515000-7f69e4555000 rw-p 00000000 00:00 0 
7f69e4595000-7f69e4715000 rw-p 00000000 00:00 0 
7f69e4795000-7f69e47d5000 rw-p 00000000 00:00 0 
7f69e4815000-7f69e4855000 rw-p 00000000 00:00 0 
7f69e4895000-7f69e4915000 rw-p 00000000 00:00 0 
7f69e4955000-7f69e4a55000 rw-p 00000000 00:00 0 
7f69e4b15000-7f69e4b55000 rw-p 00000000 00:00 0 
7f69e4c15000-7f69e4c55000 rw-p 00000000 00:00 0 
7f69e4cd5000-7f69e4d55000 rw-p 00000000 00:00 0 
7f69e4e95000-7f69e4f15000 rw-p 00000000 00:00 0 
7f69e4f55000-7f69e4fd5000 rw-p 00000000 00:00 0 
7f69e5015000-7f69e5095000 rw-p 00000000 00:00 0 
7f69e5115000-7f69e5155000 rw-p 00000000 00:00 0 
7f69e5195000-7f69e5255000 rw-p 00000000 00:00 0 
7f69e5295000-7f69e5355000 rw-p 00000000 00:00 0 
7f69e5395000-7f69e5415000 rw-p 00000000 00:00 0 
7f69e5495000-7f69e5515000 rw-p 00000000 00:00 0 
7f69e8000000-7f69e8204000 rw-p 00000000 00:00 0 
7f69e8204000-7f69ec000000 ---p 00000000 00:00 0 
7f69ec000000-7f69ec204000 rw-p 00000000 00:00 0 
7f69ec204000-7f69f0000000 ---p 00000000 00:00 0 
7f69f0000000-7f69f2cca000 rw-p 00000000 00:00 0 
7f69f2cca000-7f69f4000000 ---p 00000000 00:00 0 
7f69f4000000-7f69f42c2000 rw-p 00000000 00:00 0 
7f69f42c2000-7f69f8000000 ---p 00000000 00:00 0 
7f69f8000000-7f69fbfff000 rw-p 00000000 00:00 0 
7f69fbfff000-7f69fc000000 ---p 00000000 00:00 0 
7f69fc000000-7f69ffffd000 rw-p 00000000 00:00 0 
7f69ffffd000-7f6a00000000 ---p 00000000 00:00 0 
7f6a00000000-7f6a00562000 rw-p 00000000 00:00 0 
7f6a00562000-7f6a04000000 ---p 00000000 00:00 0 
7f6a0552f000-7f6a0556f000 rw-p 00000000 00:00 0 
7f6a055af000-7f6a0566f000 rw-p 00000000 00:00 0 
7f6a056af000-7f6a0572f000 rw-p 00000000 00:00 0 
7f6a0576f000-7f6a057af000 rw-p 00000000 00:00 0 
7f6a057ef000-7f6a0582f000 rw-p 00000000 00:00 0 
7f6a0586f000-7f6a058ef000 rw-p 00000000 00:00 0 
7f6a059af000-7f6a059ef000 rw-p 00000000 00:00 0 
7f6a05aaf000-7f6a05aef000 rw-p 00000000 00:00 0 
7f6a05b2f000-7f6a05b6f000 rw-p 00000000 00:00 0 
7f6a05baf000-7f6a05bef000 rw-p 00000000 00:00 0 
7f6a05c6f000-7f6a05caf000 rw-p 00000000 00:00 0 
7f6a05cef000-7f6a05d6f000 rw-p 00000000 00:00 0 

(and many more pages like that). This started to happen when switched the data backend from dbm.gnu to psycopg2. Note that this might have the same root cause as https://github.com/psycopg/psycopg2/issues/790

[dvarrazzo]

I'm running the final tests to release Psycopg 2.7.6, where #790 is fixed. If that fixes you problem great, otherwise probably we'll need more investigation on your part to understand at least what is the object hit by the double free.

[sam-s]

what would you like me to do?

[dvarrazzo]

@sam-s can you at least try with psycopg 2.7.6?

[sam-s]

I am using 2.7.6.1 (dt dec pq3 ext lo64).

[dvarrazzo]

And is the bug still present? (to check if that's related to #790, which was fixed after you opened this issue, or it's unrelated)

If so, you should either provide a script to reproduce the problem or run the system under gdb and then identify what is the object receiving the double free.

[sam-s]

Alas, yes; the bug is still there. The program is huge, runs for 12-20 hours. Running under gdb requires recompiling Python with -g, right? No go, sorry.

The crash happens on exit, and an hour before that the program reports success, then sleeps for an hour, then execs the next process. During this hour, I can attach gdb to the running process and do whatever you tell me.

Sorry about not being too helpful.

[dvarrazzo]

No, it doesn't need recompiling with -g. With some practice I think you can go from the object to its type to its name, which are all Python structures members, so available in non-debug mode.

I wouldn't know exactly what else to suggest. If you are able to provide further instruction I can try solve the problem, otherwise it will stay there until we stumble in it or someone else does.

[sam-s]

Okay, so I will be regularly in a position to poke the process (which will be in a sleep(3600) call) from gdb (I can attach to it using gdb --pid). Do you have a list of gdb commands you would like me to execute?

[dvarrazzo]

Any further news? I have no other reports of such error and I can't reproduce it myself.