LibPQ.jl icon indicating copy to clipboard operation
LibPQ.jl copied to clipboard

Published 20 hours ago • verified publisher icon iamed2

Error from libuv when closing watcher

Open rofinn opened this issue 6 years ago • 5 comments
trafficstars 

SSL connection has been closed unexpectedly
julia: /tmp/srccache/libuv-ed3700c849289ed01fe04273a7bf865340b2bd7e/src/unix/core.c:886: uv__io_stop: Assertion `loop->watchers[w->fd] == w' failed.
signal (6): Aborted
in expression starting at no file:0
gsignal at /lib64/libc.so.6 (unknown line)
abort at /lib64/libc.so.6 (unknown line)
__assert_fail_base at /lib64/libc.so.6 (unknown line)
__assert_fail at /lib64/libc.so.6 (unknown line)
uv__io_stop at /tmp/srccache/libuv-ed3700c849289ed01fe04273a7bf865340b2bd7e/src/unix/core.c:886
uv__poll_stop at /tmp/srccache/libuv-ed3700c849289ed01fe04273a7bf865340b2bd7e/src/unix/poll.c:81
uv__poll_close at /tmp/srccache/libuv-ed3700c849289ed01fe04273a7bf865340b2bd7e/src/unix/poll.c:123
uv_close at /tmp/srccache/libuv-ed3700c849289ed01fe04273a7bf865340b2bd7e/src/unix/core.c:160
jl_close_uv at /buildworker/worker/package_linux64/build/src/jl_uv.c:265
uvfinalize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/FileWatching/src/FileWatching.jl:192
close at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/FileWatching/src/FileWatching.jl:263 [inlined]
close at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/FileWatching/src/FileWatching.jl:270 [inlined]
_consume at /root/.julia/packages/LibPQ/gEnWD/src/asyncresults.jl:121
#handle_result#86 at /root/.julia/packages/LibPQ/gEnWD/src/asyncresults.jl:56
#handle_result at /root/.julia/packages/LibPQ/gEnWD/src/asyncresults.jl:0 [inlined]
#97 at /root/.julia/packages/LibPQ/gEnWD/src/asyncresults.jl:234
lock at /root/.julia/packages/LibPQ/gEnWD/src/connections.jl:310
#96 at ./task.jl:259
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2162
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1537 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:268
unknown function (ip: 0xffffffffffffffff)
Allocations: 963001541 (Pool: 962692671; Big: 308870); GC: 1563
signal (11): Segmentation fault
in expression starting at no file:0
abort at /lib64/libc.so.6 (unknown line)

My understanding from some issues on the libuv issue is that this may be caused by having multiple watchers on the same descriptor https://github.com/libuv/libuv/issues/1172

rofinn avatar Oct 24 '19 18:10 rofinn

Update: If SSL is turned off I get something like:

server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
julia: /tmp/srccache/libuv-ed3700c849289ed01fe04273a7bf865340b2bd7e/src/unix/core.c:886: uv__io_stop: Assertion `loop->watchers[w->fd] == w' failed.
signal (6): Aborted
in expression starting at no file:0
gsignal at /lib64/libc.so.6 (unknown line)
abort at /lib64/libc.so.6 (unknown line)
__assert_fail_base at /lib64/libc.so.6 (unknown line)
__assert_fail at /lib64/libc.so.6 (unknown line)
uv__io_stop at /tmp/srccache/libuv-ed3700c849289ed01fe04273a7bf865340b2bd7e/src/unix/core.c:886
uv__poll_stop at /tmp/srccache/libuv-ed3700c849289ed01fe04273a7bf865340b2bd7e/src/unix/poll.c:81
uv__poll_close at /tmp/srccache/libuv-ed3700c849289ed01fe04273a7bf865340b2bd7e/src/unix/poll.c:123
uv_close at /tmp/srccache/libuv-ed3700c849289ed01fe04273a7bf865340b2bd7e/src/unix/core.c:160
jl_close_uv at /buildworker/worker/package_linux64/build/src/jl_uv.c:265
uvfinalize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/FileWatching/src/FileWatching.jl:192
close at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/FileWatching/src/FileWatching.jl:263 [inlined]
close at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/FileWatching/src/FileWatching.jl:270 [inlined]
_consume at /root/.julia/packages/LibPQ/WrMcC/src/asyncresults.jl:121
#handle_result#88 at /root/.julia/packages/LibPQ/WrMcC/src/asyncresults.jl:56
#handle_result at /root/.julia/packages/LibPQ/WrMcC/src/asyncresults.jl:0 [inlined]
#99 at /root/.julia/packages/LibPQ/WrMcC/src/asyncresults.jl:234
lock at /root/.julia/packages/LibPQ/WrMcC/src/connections.jl:326
#98 at ./task.jl:259
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2162
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1537 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:268
unknown function (ip: 0xffffffffffffffff)
Allocations: 908438081 (Pool: 908214332; Big: 223749); GC: 1459
signal (11): Segmentation fault
in expression starting at no file:0
abort at /lib64/libc.so.6 (unknown line)
__assert_fail_base at /lib64/libc.so.6 (unknown line)
__assert_fail at /lib64/libc.so.6 (unknown line)
uv__io_stop at /tmp/srccache/libuv-ed3700c849289ed01fe04273a7bf865340b2bd7e/src/unix/core.c:886
uv__poll_stop at /tmp/srccache/libuv-ed3700c849289ed01fe04273a7bf865340b2bd7e/src/unix/poll.c:81
uv__poll_close at /tmp/srccache/libuv-ed3700c849289ed01fe04273a7bf865340b2bd7e/src/unix/poll.c:123
uv_close at /tmp/srccache/libuv-ed3700c849289ed01fe04273a7bf865340b2bd7e/src/unix/core.c:160
jl_close_uv at /buildworker/worker/package_linux64/build/src/jl_uv.c:265
uvfinalize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/FileWatching/src/FileWatching.jl:192
close at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/FileWatching/src/FileWatching.jl:263 [inlined]
close at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/FileWatching/src/FileWatching.jl:270 [inlined]
_consume at /root/.julia/packages/LibPQ/WrMcC/src/asyncresults.jl:121
#handle_result#88 at /root/.julia/packages/LibPQ/WrMcC/src/asyncresults.jl:56
#handle_result at /root/.julia/packages/LibPQ/WrMcC/src/asyncresults.jl:0 [inlined]
#99 at /root/.julia/packages/LibPQ/WrMcC/src/asyncresults.jl:234
lock at /root/.julia/packages/LibPQ/WrMcC/src/connections.jl:326
#98 at ./task.jl:259
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2162
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1537 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:268
unknown function (ip: 0xffffffffffffffff)
Allocations: 908438081 (Pool: 908214332; Big: 223749); GC: 1459

rofinn avatar Oct 27 '19 19:10 rofinn

Hmmmmmmmmmmm

On Monday I'll implement some way to track this and check to see if the multiple-watchers thing is the culprit.

iamed2 avatar Oct 27 '19 20:10 iamed2

Iiiiiiiit's back!

[error | LibPQ]: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
julia: src/unix/core.c:889: uv__io_stop: Assertion `loop->watchers[w->fd] == w' failed.
signal (6): Aborted
in expression starting at none:1
gsignal at /lib64/libc.so.6 (unknown line)
abort at /lib64/libc.so.6 (unknown line)
__assert_fail_base at /lib64/libc.so.6 (unknown line)
__assert_fail at /lib64/libc.so.6 (unknown line)
uv__io_stop at /workspace/srcdir/libuv/src/unix/core.c:889
uv__poll_stop at /workspace/srcdir/libuv/src/unix/poll.c:97
uv__poll_close at /workspace/srcdir/libuv/src/unix/poll.c:143
uv_close at /workspace/srcdir/libuv/src/unix/core.c:155
jl_close_uv at /buildworker/worker/package_linux64/build/src/jl_uv.c:272
uvfinalize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/FileWatching/src/FileWatching.jl:205
_consume at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/FileWatching/src/FileWatching.jl:283
#handle_result#88 at /root/.julia/packages/LibPQ/isUfk/src/asyncresults.jl:56
#handle_result at ./none:0 [inlined]
#99 at /root/.julia/packages/LibPQ/isUfk/src/asyncresults.jl:249
lock at /root/.julia/packages/LibPQ/isUfk/src/connections.jl:337
#98 at ./task.jl:333
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1631 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:659
unknown function (ip: 0xffffffffffffffff)
Allocations: 2417044783 (Pool: 2416945289; Big: 99494); GC: 364
signal (11): Segmentation fault
in expression starting at none:1
abort at /lib64/libc.so.6 (unknown line)
__assert_fail_base at /lib64/libc.so.6 (unknown line)
__assert_fail at /lib64/libc.so.6 (unknown line)
uv__io_stop at /workspace/srcdir/libuv/src/unix/core.c:889
uv__poll_stop at /workspace/srcdir/libuv/src/unix/poll.c:97
uv__poll_close at /workspace/srcdir/libuv/src/unix/poll.c:143
uv_close at /workspace/srcdir/libuv/src/unix/core.c:155
jl_close_uv at /buildworker/worker/package_linux64/build/src/jl_uv.c:272
uvfinalize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/FileWatching/src/FileWatching.jl:205
_consume at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/FileWatching/src/FileWatching.jl:283
#handle_result#88 at /root/.julia/packages/LibPQ/isUfk/src/asyncresults.jl:56
#handle_result at ./none:0 [inlined]
#99 at /root/.julia/packages/LibPQ/isUfk/src/asyncresults.jl:249
lock at /root/.julia/packages/LibPQ/isUfk/src/connections.jl:337
#98 at ./task.jl:333
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1631 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:659
unknown function (ip: 0xffffffffffffffff)
Allocations: 2417044783 (Pool: 2416945289; Big: 99494); GC: 364

iamed2 avatar Mar 17 '20 22:03 iamed2

Update: this happened again

Both times, 1 minute after on the server I saw FATAL: connection to client lost, but I have also seen this error when the crash does not happen.

iamed2 avatar Mar 18 '20 19:03 iamed2

https://github.com/JuliaLang/libuv/commit/c9406ba0e3d67907c1973a71968b89a6bd83c63c suggests that this should not be a problem on Julia 1.7.0+

iamed2 avatar Dec 03 '21 20:12 iamed2