haproxy
haproxy copied to clipboard
haproxy
3.2.3 - Crash under high load: ha_panic
Detailed Description of the Problem
Hello,
We're using haproxy with cpu-policy group-by-cluster, 128 threads (2 thread groups).
And under very high loads, over 140Gbps, we've been getting crashes. We got a core out of it, though...however couldn't get much more info because the worker crashed and the stats socket was closed.
Expected Behavior
.
Steps to Reproduce the Behavior
Not sure, but it just happens under very high loads.
Do you have any idea what may have caused this?
Not really
Do you have an idea how to solve the issue?
I wish :-)
What is your configuration?
This is a more complex setup, it uses a lua service for some services, but mostly just an enormous amount of typical http processing:
global
hard-stop-after 7d
maxconn 900000
ulimit-n 3150919
set-dumpable
user haproxy
group haproxy
daemon
master-worker no-exit-on-failure
cpu-policy group-by-cluster
#ssl-engine rdrand
#ssl-mode-async
unix-bind user haproxy mode 777
stats socket /var/run/haproxy_conexoes.sock mode 666 level admin expose-fd listeners
stats timeout 10m
tune.lua.bool-sample-conversion normal
lua-load-per-thread /etc/haproxy/response_lua.lua
h1-case-adjust content-length Content-Length
h1-case-adjust content-type Content-Type
h1-case-adjust content-range Content-Range
peers peer_local
bind 127.0.0.1:1000 thread 1/all
server peer_reload
defaults
mode http
retries 1
maxconn 900000
timeout connect 1s
timeout server 100s
timeout client 100s
timeout http-request 12s
timeout http-keep-alive 40s
timeout tunnel 300s
timeout client-fin 4s
timeout server-fin 4s
option splice-response
...
Output of haproxy -vv
HAProxy version 3.2.3-1844da7 2025/07/09 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2030.
Known bugs: http://www.haproxy.org/bugs/bugs-3.2.3.html
Running on: Linux 6.12.41 #1 SMP PREEMPT_DYNAMIC Fri Aug 8 20:22:46 UTC 2025 x86_64
Build options :
TARGET = linux-glibc
CC = cc
CFLAGS = -O2 -g -fwrapv
OPTIONS = USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_NS=1 USE_PCRE=1 USE_PCRE_JIT=1
DEBUG =
Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT +PCRE -PCRE2 -PCRE2_JIT +PCRE_JIT +POLL +PRCTL -PROCCTL -PROMEX -PTHREAD_EMULATION -QUIC -QUIC_OPENSSL_COMPAT +RT -SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL +ZLIB
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=32, MAX_THREADS=1024, default=128).
Built with SSL library version : OpenSSL 1.1.1zb 20 Jan 2025
Running on SSL library version : OpenSSL 1.1.1zb 20 Jan 2025
SSL library supports TLS extensions : yes
SSL library supports SNI : yes
SSL library FIPS mode : no
SSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.4.6
Built with network namespace support.
Built with zlib version : 1.2.13
Running on zlib version : 1.2.13
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE version : 8.45 2021-06-15
Running on PCRE version : 8.45 2021-06-15
PCRE library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 11.2.0
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
<default> : mode=SPOP side=BE mux=SPOP flags=HOL_RISK|NO_UPG
spop : mode=SPOP side=BE mux=SPOP flags=HOL_RISK|NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
Available services : none
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
Last Outputs and Backtraces
(gdb) bt full
#0 0x00007f701fd14868 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f701fcfb546 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x00000000005db0ef in ha_panic () at src/debug.c:826
buf = <optimized out>
thr = <optimized out>
__v = <optimized out>
#3 0x00000000005db195 in ha_panic () at src/debug.c:758
buf = <optimized out>
thr = <optimized out>
__v = <optimized out>
__v = <optimized out>
__v = <optimized out>
__v = <optimized out>
__v = <optimized out>
__v = <optimized out>
#4 0x00000000006c5f8d in wdt_handler (sig=<optimized out>, si=<optimized out>, arg=<optimized out>) at src/wdt.c:190
n = <optimized out>
p = <optimized out>
thr_bit = <optimized out>
thr = 40
tgrp = <optimized out>
#5 <signal handler called>
No symbol table info available.
#6 0x00007f701fe4b092 in __memmove_avx_unaligned_erms () from /lib64/libc.so.6
No symbol table info available.
#7 0x00000000005c90c5 in sc_conn_send (sc=sc@entry=0x7f6f750105a0) at include/haproxy/buf.h:80
send_flag = <optimized out>
conn = <optimized out>
sco = <optimized out>
s = <optimized out>
oc = <optimized out>
ret = <optimized out>
did_send = <optimized out>
__func__ = <optimized out>
__dbg_cnt_1622 = {file = 0x71af40 "src/stconn.c", func = 0x71ac98 <__func__.44> "sc_conn_send", desc = 0x71a790 "\"sc_ep_test(sc, SE_FL_EOS|SE_FL_ERROR|SE_FL_ERR_PENDING) == (SE_FL_EOS|SE_FL_ERR_PENDING)\"", line = 1622,
type = 0 '\000', count = 0}
msg = <optimized out>
msg = <optimized out>
__v = <optimized out>
__dbg_cnt_1756 = {file = 0x71af40 "src/stconn.c", func = 0x71ac98 <__func__.44> "sc_conn_send", desc = 0x71a790 "\"sc_ep_test(sc, SE_FL_EOS|SE_FL_ERROR|SE_FL_ERR_PENDING) == (SE_FL_EOS|SE_FL_ERR_PENDING)\"", line = 1756,
type = 0 '\000', count = 0}
msg = <optimized out>
msg = <optimized out>
__v = <optimized out>
#8 0x00000000005cd546 in sc_conn_sync_send (sc=sc@entry=0x7f6f750105a0) at src/stconn.c:1808
oc = <optimized out>
#9 0x00000000005263ac in sc_sync_send (sc=<optimized out>) at include/haproxy/sc_strm.h:350
__func__ = "sc_sync_send"
_ = {func = 0x6fbd28 <__func__.2> "sc_sync_send", file = 0x6fb6f6 "include/haproxy/sc_strm.h", line = 357, what = 1 '\001', arg8 = 0 '\000', arg32 = 0}
#10 process_stream (t=<optimized out>, context=0x7f6f7500bbf0, state=<optimized out>) at src/stream.c:2396
srv = <optimized out>
s = 0x7f6f7500bbf0
sess = <optimized out>
scf_flags = 1120
scb_flags = 1129
rqf_last = <optimized out>
rpf_last = 2147483649
rq_prod_last = <optimized out>
--Type <RET> for more, q to quit, c to continue without paging--
rq_cons_last = <optimized out>
rp_cons_last = 8
rp_prod_last = 8
req_ana_back = <optimized out>
res_ana_back = <optimized out>
req = 0x7f6f7500bc18
res = 0x7f6f7500bc60
scf = <optimized out>
scb = <optimized out>
rate = <optimized out>
resync_stconns = <optimized out>
__func__ = <optimized out>
resync_request = <optimized out>
resync_response = <optimized out>
#11 0x0000000000676fb5 in run_tasks_from_lists (budgets=budgets@entry=0x7f6fbdf40f70) at src/task.c:655
process = <optimized out>
tl_queues = 0x1028b50 <ha_thread_ctx+20688>
t = 0x7f6f7500ba50
budget_mask = <optimized out>
profile_entry = 0x0
done = <optimized out>
queue = <optimized out>
state = <optimized out>
ctx = <optimized out>
__func__ = <optimized out>
#12 0x0000000000677526 in process_runnable_tasks () at src/task.c:889
tt = 0x1028a80 <ha_thread_ctx+20480>
lrq = <optimized out>
grq = <optimized out>
t = <optimized out>
max = {0, 14, 0, 0}
max_total = <optimized out>
tmp_list = <optimized out>
queue = <optimized out>
max_processed = <optimized out>
lpicked = <optimized out>
gpicked = <optimized out>
heavy_queued = 1
budget = <optimized out>
not_done_yet = <optimized out>
#13 0x00000000005eccc8 in run_poll_loop () at src/haproxy.c:2864
next = <optimized out>
wake = <optimized out>
__func__ = <optimized out>
#14 0x00000000005ed337 in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3080
ptaf = <optimized out>
ptif = 0x7c1aa0 <per_thread_init_list>
ptdf = <optimized out>
ptff = <optimized out>
init_left = 0
init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}
init_cond = {__data = {{__wseq = 3148, __wseq32 = {__low = 3148, __high = 0}}, {__g1_start = 3000, __g1_start32 = {__low = 3000, __high = 0}}, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 296, __wrefs = 0, __g_signals = {
0, 0}}, __size = "L\f\000\000\000\000\000\000\270\v", '\000' <repeats 22 times>, "(\001", '\000' <repeats 13 times>, __align = 3148}
#15 0x00007f70202fde45 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#16 0x00007f701fde44af in clone () from /lib64/libc.so.6
No symbol table info available.
(gdb)
(gdb)
Additional Information
I'll attach the full backtrace, and can provide the core off-github, if you want. It's 6,6GB, though.
not sure it's related, but for the sake of completion, here's the cpu layout:
[keep] cpu= 0 pk=00 no=00 cl=000(000) l3=000 l2=00 l1=00 ts=000 capa=5250 smt=0/2
[keep] cpu= 1 pk=00 no=00 cl=000(000) l3=000 l2=01 l1=01 ts=001 capa=5250 smt=0/2
[keep] cpu= 2 pk=00 no=00 cl=000(000) l3=000 l2=02 l1=02 ts=002 capa=5250 smt=0/2
[keep] cpu= 3 pk=00 no=00 cl=000(000) l3=000 l2=03 l1=03 ts=003 capa=5250 smt=0/2
[keep] cpu= 4 pk=00 no=00 cl=000(000) l3=000 l2=04 l1=04 ts=004 capa=5250 smt=0/2
[keep] cpu= 5 pk=00 no=00 cl=000(000) l3=000 l2=05 l1=05 ts=005 capa=5250 smt=0/2
[keep] cpu= 6 pk=00 no=00 cl=000(000) l3=000 l2=06 l1=06 ts=006 capa=5250 smt=0/2
[keep] cpu= 7 pk=00 no=00 cl=000(000) l3=000 l2=07 l1=07 ts=007 capa=5250 smt=0/2
[keep] cpu= 8 pk=00 no=00 cl=000(000) l3=000 l2=08 l1=08 ts=008 capa=5250 smt=0/2
[keep] cpu= 9 pk=00 no=00 cl=000(000) l3=000 l2=09 l1=09 ts=009 capa=5250 smt=0/2
[keep] cpu= 10 pk=00 no=00 cl=000(000) l3=000 l2=10 l1=10 ts=010 capa=5250 smt=0/2
[keep] cpu= 11 pk=00 no=00 cl=000(000) l3=000 l2=11 l1=11 ts=011 capa=5250 smt=0/2
[keep] cpu= 12 pk=00 no=00 cl=000(000) l3=000 l2=12 l1=12 ts=012 capa=5250 smt=0/2
[keep] cpu= 13 pk=00 no=00 cl=000(000) l3=000 l2=13 l1=13 ts=013 capa=5250 smt=0/2
[keep] cpu= 14 pk=00 no=00 cl=000(000) l3=000 l2=14 l1=14 ts=014 capa=5250 smt=0/2
[keep] cpu= 15 pk=00 no=00 cl=000(000) l3=000 l2=15 l1=15 ts=015 capa=5250 smt=0/2
[keep] cpu= 16 pk=00 no=00 cl=000(000) l3=000 l2=16 l1=16 ts=016 capa=5250 smt=0/2
[keep] cpu= 17 pk=00 no=00 cl=000(000) l3=000 l2=17 l1=17 ts=017 capa=5250 smt=0/2
[keep] cpu= 18 pk=00 no=00 cl=000(000) l3=000 l2=18 l1=18 ts=018 capa=5250 smt=0/2
[keep] cpu= 19 pk=00 no=00 cl=000(000) l3=000 l2=19 l1=19 ts=019 capa=5250 smt=0/2
[keep] cpu= 20 pk=00 no=00 cl=000(000) l3=000 l2=20 l1=20 ts=020 capa=5250 smt=0/2
[keep] cpu= 21 pk=00 no=00 cl=000(000) l3=000 l2=21 l1=21 ts=021 capa=5250 smt=0/2
[keep] cpu= 22 pk=00 no=00 cl=000(000) l3=000 l2=22 l1=22 ts=022 capa=5250 smt=0/2
[keep] cpu= 23 pk=00 no=00 cl=000(000) l3=000 l2=23 l1=23 ts=023 capa=5250 smt=0/2
[keep] cpu= 24 pk=00 no=00 cl=000(000) l3=000 l2=24 l1=24 ts=024 capa=5250 smt=0/2
[keep] cpu= 25 pk=00 no=00 cl=000(000) l3=000 l2=25 l1=25 ts=025 capa=5250 smt=0/2
[keep] cpu= 26 pk=00 no=00 cl=000(000) l3=000 l2=26 l1=26 ts=026 capa=5250 smt=0/2
[keep] cpu= 27 pk=00 no=00 cl=000(000) l3=000 l2=27 l1=27 ts=027 capa=5250 smt=0/2
[keep] cpu= 28 pk=00 no=00 cl=000(000) l3=000 l2=28 l1=28 ts=028 capa=5250 smt=0/2
[keep] cpu= 29 pk=00 no=00 cl=000(000) l3=000 l2=29 l1=29 ts=029 capa=5250 smt=0/2
[keep] cpu= 30 pk=00 no=00 cl=000(000) l3=000 l2=30 l1=30 ts=030 capa=5250 smt=0/2
[keep] cpu= 31 pk=00 no=00 cl=000(000) l3=000 l2=31 l1=31 ts=031 capa=5250 smt=0/2
[keep] cpu= 32 pk=01 no=01 cl=001(000) l3=001 l2=32 l1=32 ts=000 capa=5250 smt=0/2
[keep] cpu= 33 pk=01 no=01 cl=001(000) l3=001 l2=33 l1=33 ts=001 capa=5250 smt=0/2
[keep] cpu= 34 pk=01 no=01 cl=001(000) l3=001 l2=34 l1=34 ts=002 capa=5250 smt=0/2
[keep] cpu= 35 pk=01 no=01 cl=001(000) l3=001 l2=35 l1=35 ts=003 capa=5250 smt=0/2
[keep] cpu= 36 pk=01 no=01 cl=001(000) l3=001 l2=36 l1=36 ts=004 capa=5250 smt=0/2
[keep] cpu= 37 pk=01 no=01 cl=001(000) l3=001 l2=37 l1=37 ts=005 capa=5250 smt=0/2
[keep] cpu= 38 pk=01 no=01 cl=001(000) l3=001 l2=38 l1=38 ts=006 capa=5250 smt=0/2
[keep] cpu= 39 pk=01 no=01 cl=001(000) l3=001 l2=39 l1=39 ts=007 capa=5250 smt=0/2
[keep] cpu= 40 pk=01 no=01 cl=001(000) l3=001 l2=40 l1=40 ts=008 capa=5250 smt=0/2
[keep] cpu= 41 pk=01 no=01 cl=001(000) l3=001 l2=41 l1=41 ts=009 capa=5250 smt=0/2
[keep] cpu= 42 pk=01 no=01 cl=001(000) l3=001 l2=42 l1=42 ts=010 capa=5250 smt=0/2
[keep] cpu= 43 pk=01 no=01 cl=001(000) l3=001 l2=43 l1=43 ts=011 capa=5250 smt=0/2
[keep] cpu= 44 pk=01 no=01 cl=001(000) l3=001 l2=44 l1=44 ts=012 capa=5250 smt=0/2
[keep] cpu= 45 pk=01 no=01 cl=001(000) l3=001 l2=45 l1=45 ts=013 capa=5250 smt=0/2
[keep] cpu= 46 pk=01 no=01 cl=001(000) l3=001 l2=46 l1=46 ts=014 capa=5250 smt=0/2
[keep] cpu= 47 pk=01 no=01 cl=001(000) l3=001 l2=47 l1=47 ts=015 capa=5250 smt=0/2
[keep] cpu= 48 pk=01 no=01 cl=001(000) l3=001 l2=48 l1=48 ts=016 capa=5250 smt=0/2
[keep] cpu= 49 pk=01 no=01 cl=001(000) l3=001 l2=49 l1=49 ts=017 capa=5250 smt=0/2
[keep] cpu= 50 pk=01 no=01 cl=001(000) l3=001 l2=50 l1=50 ts=018 capa=5250 smt=0/2
[keep] cpu= 51 pk=01 no=01 cl=001(000) l3=001 l2=51 l1=51 ts=019 capa=5250 smt=0/2
[keep] cpu= 52 pk=01 no=01 cl=001(000) l3=001 l2=52 l1=52 ts=020 capa=5250 smt=0/2
[keep] cpu= 53 pk=01 no=01 cl=001(000) l3=001 l2=53 l1=53 ts=021 capa=5250 smt=0/2
[keep] cpu= 54 pk=01 no=01 cl=001(000) l3=001 l2=54 l1=54 ts=022 capa=5250 smt=0/2
[keep] cpu= 55 pk=01 no=01 cl=001(000) l3=001 l2=55 l1=55 ts=023 capa=5250 smt=0/2
[keep] cpu= 56 pk=01 no=01 cl=001(000) l3=001 l2=56 l1=56 ts=024 capa=5250 smt=0/2
[keep] cpu= 57 pk=01 no=01 cl=001(000) l3=001 l2=57 l1=57 ts=025 capa=5250 smt=0/2
[keep] cpu= 58 pk=01 no=01 cl=001(000) l3=001 l2=58 l1=58 ts=026 capa=5250 smt=0/2
[keep] cpu= 59 pk=01 no=01 cl=001(000) l3=001 l2=59 l1=59 ts=027 capa=5250 smt=0/2
[keep] cpu= 60 pk=01 no=01 cl=001(000) l3=001 l2=60 l1=60 ts=028 capa=5250 smt=0/2
[keep] cpu= 61 pk=01 no=01 cl=001(000) l3=001 l2=61 l1=61 ts=029 capa=5250 smt=0/2
[keep] cpu= 62 pk=01 no=01 cl=001(000) l3=001 l2=62 l1=62 ts=030 capa=5250 smt=0/2
[keep] cpu= 63 pk=01 no=01 cl=001(000) l3=001 l2=63 l1=63 ts=031 capa=5250 smt=0/2
[keep] cpu= 64 pk=00 no=00 cl=000(000) l3=000 l2=00 l1=00 ts=000 capa=5250 smt=1/2
[keep] cpu= 65 pk=00 no=00 cl=000(000) l3=000 l2=01 l1=01 ts=001 capa=5250 smt=1/2
[keep] cpu= 66 pk=00 no=00 cl=000(000) l3=000 l2=02 l1=02 ts=002 capa=5250 smt=1/2
[keep] cpu= 67 pk=00 no=00 cl=000(000) l3=000 l2=03 l1=03 ts=003 capa=5250 smt=1/2
[keep] cpu= 68 pk=00 no=00 cl=000(000) l3=000 l2=04 l1=04 ts=004 capa=5250 smt=1/2
[keep] cpu= 69 pk=00 no=00 cl=000(000) l3=000 l2=05 l1=05 ts=005 capa=5250 smt=1/2
[keep] cpu= 70 pk=00 no=00 cl=000(000) l3=000 l2=06 l1=06 ts=006 capa=5250 smt=1/2
[keep] cpu= 71 pk=00 no=00 cl=000(000) l3=000 l2=07 l1=07 ts=007 capa=5250 smt=1/2
[keep] cpu= 72 pk=00 no=00 cl=000(000) l3=000 l2=08 l1=08 ts=008 capa=5250 smt=1/2
[keep] cpu= 73 pk=00 no=00 cl=000(000) l3=000 l2=09 l1=09 ts=009 capa=5250 smt=1/2
[keep] cpu= 74 pk=00 no=00 cl=000(000) l3=000 l2=10 l1=10 ts=010 capa=5250 smt=1/2
[keep] cpu= 75 pk=00 no=00 cl=000(000) l3=000 l2=11 l1=11 ts=011 capa=5250 smt=1/2
[keep] cpu= 76 pk=00 no=00 cl=000(000) l3=000 l2=12 l1=12 ts=012 capa=5250 smt=1/2
[keep] cpu= 77 pk=00 no=00 cl=000(000) l3=000 l2=13 l1=13 ts=013 capa=5250 smt=1/2
[keep] cpu= 78 pk=00 no=00 cl=000(000) l3=000 l2=14 l1=14 ts=014 capa=5250 smt=1/2
[keep] cpu= 79 pk=00 no=00 cl=000(000) l3=000 l2=15 l1=15 ts=015 capa=5250 smt=1/2
[keep] cpu= 80 pk=00 no=00 cl=000(000) l3=000 l2=16 l1=16 ts=016 capa=5250 smt=1/2
[keep] cpu= 81 pk=00 no=00 cl=000(000) l3=000 l2=17 l1=17 ts=017 capa=5250 smt=1/2
[keep] cpu= 82 pk=00 no=00 cl=000(000) l3=000 l2=18 l1=18 ts=018 capa=5250 smt=1/2
[keep] cpu= 83 pk=00 no=00 cl=000(000) l3=000 l2=19 l1=19 ts=019 capa=5250 smt=1/2
[keep] cpu= 84 pk=00 no=00 cl=000(000) l3=000 l2=20 l1=20 ts=020 capa=5250 smt=1/2
[keep] cpu= 85 pk=00 no=00 cl=000(000) l3=000 l2=21 l1=21 ts=021 capa=5250 smt=1/2
[keep] cpu= 86 pk=00 no=00 cl=000(000) l3=000 l2=22 l1=22 ts=022 capa=5250 smt=1/2
[keep] cpu= 87 pk=00 no=00 cl=000(000) l3=000 l2=23 l1=23 ts=023 capa=5250 smt=1/2
[keep] cpu= 88 pk=00 no=00 cl=000(000) l3=000 l2=24 l1=24 ts=024 capa=5250 smt=1/2
[keep] cpu= 89 pk=00 no=00 cl=000(000) l3=000 l2=25 l1=25 ts=025 capa=5250 smt=1/2
[keep] cpu= 90 pk=00 no=00 cl=000(000) l3=000 l2=26 l1=26 ts=026 capa=5250 smt=1/2
[keep] cpu= 91 pk=00 no=00 cl=000(000) l3=000 l2=27 l1=27 ts=027 capa=5250 smt=1/2
[keep] cpu= 92 pk=00 no=00 cl=000(000) l3=000 l2=28 l1=28 ts=028 capa=5250 smt=1/2
[keep] cpu= 93 pk=00 no=00 cl=000(000) l3=000 l2=29 l1=29 ts=029 capa=5250 smt=1/2
[keep] cpu= 94 pk=00 no=00 cl=000(000) l3=000 l2=30 l1=30 ts=030 capa=5250 smt=1/2
[keep] cpu= 95 pk=00 no=00 cl=000(000) l3=000 l2=31 l1=31 ts=031 capa=5250 smt=1/2
[keep] cpu= 96 pk=01 no=01 cl=001(000) l3=001 l2=32 l1=32 ts=000 capa=5250 smt=1/2
[keep] cpu= 97 pk=01 no=01 cl=001(000) l3=001 l2=33 l1=33 ts=001 capa=5250 smt=1/2
[keep] cpu= 98 pk=01 no=01 cl=001(000) l3=001 l2=34 l1=34 ts=002 capa=5250 smt=1/2
[keep] cpu= 99 pk=01 no=01 cl=001(000) l3=001 l2=35 l1=35 ts=003 capa=5250 smt=1/2
[keep] cpu=100 pk=01 no=01 cl=001(000) l3=001 l2=36 l1=36 ts=004 capa=5250 smt=1/2
[keep] cpu=101 pk=01 no=01 cl=001(000) l3=001 l2=37 l1=37 ts=005 capa=5250 smt=1/2
[keep] cpu=102 pk=01 no=01 cl=001(000) l3=001 l2=38 l1=38 ts=006 capa=5250 smt=1/2
[keep] cpu=103 pk=01 no=01 cl=001(000) l3=001 l2=39 l1=39 ts=007 capa=5250 smt=1/2
[keep] cpu=104 pk=01 no=01 cl=001(000) l3=001 l2=40 l1=40 ts=008 capa=5250 smt=1/2
[keep] cpu=105 pk=01 no=01 cl=001(000) l3=001 l2=41 l1=41 ts=009 capa=5250 smt=1/2
[keep] cpu=106 pk=01 no=01 cl=001(000) l3=001 l2=42 l1=42 ts=010 capa=5250 smt=1/2
[keep] cpu=107 pk=01 no=01 cl=001(000) l3=001 l2=43 l1=43 ts=011 capa=5250 smt=1/2
[keep] cpu=108 pk=01 no=01 cl=001(000) l3=001 l2=44 l1=44 ts=012 capa=5250 smt=1/2
[keep] cpu=109 pk=01 no=01 cl=001(000) l3=001 l2=45 l1=45 ts=013 capa=5250 smt=1/2
[keep] cpu=110 pk=01 no=01 cl=001(000) l3=001 l2=46 l1=46 ts=014 capa=5250 smt=1/2
[keep] cpu=111 pk=01 no=01 cl=001(000) l3=001 l2=47 l1=47 ts=015 capa=5250 smt=1/2
[keep] cpu=112 pk=01 no=01 cl=001(000) l3=001 l2=48 l1=48 ts=016 capa=5250 smt=1/2
[keep] cpu=113 pk=01 no=01 cl=001(000) l3=001 l2=49 l1=49 ts=017 capa=5250 smt=1/2
[keep] cpu=114 pk=01 no=01 cl=001(000) l3=001 l2=50 l1=50 ts=018 capa=5250 smt=1/2
[keep] cpu=115 pk=01 no=01 cl=001(000) l3=001 l2=51 l1=51 ts=019 capa=5250 smt=1/2
[keep] cpu=116 pk=01 no=01 cl=001(000) l3=001 l2=52 l1=52 ts=020 capa=5250 smt=1/2
[keep] cpu=117 pk=01 no=01 cl=001(000) l3=001 l2=53 l1=53 ts=021 capa=5250 smt=1/2
[keep] cpu=118 pk=01 no=01 cl=001(000) l3=001 l2=54 l1=54 ts=022 capa=5250 smt=1/2
[keep] cpu=119 pk=01 no=01 cl=001(000) l3=001 l2=55 l1=55 ts=023 capa=5250 smt=1/2
[keep] cpu=120 pk=01 no=01 cl=001(000) l3=001 l2=56 l1=56 ts=024 capa=5250 smt=1/2
[keep] cpu=121 pk=01 no=01 cl=001(000) l3=001 l2=57 l1=57 ts=025 capa=5250 smt=1/2
[keep] cpu=122 pk=01 no=01 cl=001(000) l3=001 l2=58 l1=58 ts=026 capa=5250 smt=1/2
[keep] cpu=123 pk=01 no=01 cl=001(000) l3=001 l2=59 l1=59 ts=027 capa=5250 smt=1/2
[keep] cpu=124 pk=01 no=01 cl=001(000) l3=001 l2=60 l1=60 ts=028 capa=5250 smt=1/2
[keep] cpu=125 pk=01 no=01 cl=001(000) l3=001 l2=61 l1=61 ts=029 capa=5250 smt=1/2
[keep] cpu=126 pk=01 no=01 cl=001(000) l3=001 l2=62 l1=62 ts=030 capa=5250 smt=1/2
[keep] cpu=127 pk=01 no=01 cl=001(000) l3=001 l2=63 l1=63 ts=031 capa=5250 smt=1/2
CPU clusters:
0 cpus= 64 cores= 32 capa=224000
1 cpus= 64 cores= 32 capa=224000
Thread CPU Bindings:
Tgrp/Thr Tid CPU set
1/1-64 1-64 64: 0-31,64-95
2/1-64 65-128 64: 32-63,96-127
I've mentioned before but just to make sure:
We've tried collecting show info, threads, stats..., but stats socket was dead by the time we've detected the crash:
2025/08/11 00:23:06 socat[1163362] E connect(5, AF=1 "/var/run/haproxy_conexoes.sock", 32): Connection refused
2025/08/11 00:23:06 socat[1163364] E connect(5, AF=1 "/var/run/haproxy_conexoes.sock", 32): Connection refused
2025/08/11 00:23:06 socat[1163366] E connect(5, AF=1 "/var/run/haproxy_conexoes.sock", 32): Connection refused
2025/08/11 00:23:06 socat[1163368] E connect(5, AF=1 "/var/run/haproxy_conexoes.sock", 32): Connection refused
Hi Felipe!
I'm not seeing anything wrong that could explain anything like this. It died while transferring a buffer, nothing unusual there. I can't see any explanation for this one based one the trace. Thus the only explanation I'm seeing is that this thread got stuck for a while on some contention, was finally released and expired its time slot just there. Do you have the stderr output ? It should show several warnings after 100, 200, ... ms before the panic. These ones will indicate where the task was stuck.
Hi @wtarreau !
We don't have anything else, unfortunately. If you'd like to inspect the core, I can uploaded it somewhere and send you the link.
Since this happens under very high loads (we've been testing 200Gbps setups for over a month and this happens on one of those machines), we thought it might be related to that.
I think the core will be too late, the warnings could have helped because they would have caught the situation in progress.
It's very possible that it's related to the high load, you're totally right, but even then it's hard to figure what. If you manage next time to capture the stderr of the process showing the watchdog dump and the prior warnings, that could help immensely I think.
Hi @wtarreau, we'll do that.
We don't use syslog we rather our own software to collect the "log" parameter to get all user logs
We discard any non-log lines, though.
We'll look through the config options to see if we can define a different log for error or non-normal log lines, though. And use syslog for those.
Thanks!
No, for warnings/panics it's only emitted on stderr. If you start under systemd, you could have them in your systemd logs for the service for example.
Hi @wtarreau we were able to get a few crashes tonight under very high loads:
WARNING! thread 23 has stopped processing traffic for 226 milliseconds
with 257 streams currently blocked, prevented from making any progress.
While this may occasionally happen with inefficient configurations
involving excess of regular expressions, map_reg, or heavy Lua processing,
this must remain exceptional because the system's stability is now at risk.
Timers in logs may be reported incorrectly, spurious timeouts may happen,
some incoming connections may silently be dropped, health checks may
randomly fail, and accesses to the CLI may block the whole process. The
blocking delay before emitting this warning may be adjusted via the global
'warn-blocked-traffic-after' directive. Please check the trace below for
any clues about configuration elements that need to be corrected:
* Thread 23: id=0x7fdea2f66640 act=1 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/23 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=954960080173 now=955186331254 diff=226251081
curr_task=0x7fde8d06d9b0 (task) calls=2 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fddb0bff6a0
stream=0x7fddb0bff6a0: [1754954493.357510] id=5220517 proto=tcpv4 source=179.124.182.69:12549
flags=0x35c0a, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=(nil), pend_pos=(nil) waiting=0 epoch=0
p_stc=3 p_req=3 p_res=3 p_prp=2
frontend=ssl (id=3 mode=http), listener=? (id=854) addr=179.96.0.90:80
backend=hyper_vod_tvs (id=15 mode=http)
server=tvs_hyper (id=1) addr=unix
task=0x7fde8d06d9b0 (state=0x01 nice=0 calls=2 rate=0 exp=1m30s tid=22(1/22) age=10s)
txn=0x7fddb8e14b30 flags=0x3000 meth=1 status=502 req.st=MSG_DONE rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fddb0bdd2d0 flags=0x000004a2 ioto=1m40s state=EST endp=CONN,0x7fddb0b884f0,0x50404001 sub=1 rex=<NEVER> wex=<NEVER> rto=1m30s wto=<NEVER>
h1s=0x7fddb0b884f0 h1s.flg=0x100010 .req.state=MSG_DONE .res.state=MSG_RPBEFORE
.meth=GET status=0 .sd.flg=0x50404001 .sd.evts= .sc.flg=0x000004a2 .sc.app=0x7fddb0bff6a0 .sc.evts=
.subs=0x7fddb0bdd2e8(ev=1 tl=0x7fdd20f52530 tl.calls=0 tl.ctx=0x7fddb0bdd2d0 tl.fct=sc_conn_io_cb+0)
h1c=0x7fddb0e6f8f0 h1c.flg=0x0 .sub=0 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fddb0b685e0 .exp=<NEVER>
co0=0x7fde50b9d4f0 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=LISTENER:0x442f7310
flags=0x80000300 fd=30829 fd.state=120 updt=0 fd.tmask=0x400000
scb=0x7fddb0bc8ec0 flags=0x0006103d ioto=1m40s state=CLO endp=CONN,0x7fddb0dcc5a0,0x50418a01 sub=0 rex=<NEVER> wex=<NEVER> rto=1m40s wto=<NEVER>
h1s=0x7fddb0dcc5a0 h1s.flg=0x14010 .req.state=MSG_DONE .res.state=MSG_RPBEFORE
.meth=GET status=0 .sd.flg=0x50418a01 .sd.evts=E5E1 .sc.flg=0x0006103d .sc.app=0x7fddb0bff6a0 .sc.evts=S5 .subs=(nil)
h1c=0x7fddb0eb7be0 h1c.flg=0x80000900 .sub=0 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts=M5M1 .task=0x7fddb0dff530 .exp=4s
co1=0x7fddb0b5dcc0 ctrl=abns_stream xprt=RAW mux=H1 data=STRM target=SERVER:0x440f2c00
flags=0x000c0300 fd=48634 fd.state=1122 updt=1 fd.tmask=0x400000
req=0x7fddb0bff6c8 (f=0x1840000 an=0x8000 tofwd=0 total=406)
an_exp=<NEVER> buf=0x7fddb0bff6d0 data=(nil) o=0 p=0 i=0 size=0
htx=0x7bcec0 flags=0x0 size=0 data=0 used=0 wrap=NO extra=0
res=0x7fddb0bff710 (f=0x80008001 an=0x1400000 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fddb0bff718 data=0x7fddb0f1ab10 o=0 p=0 i=0 size=1048576
htx=0x7fddb0f1ab10 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: R:TASK_WQ U:TASK_WQ R:STK_TABLE U:STK_TABLE R:STK_SESS U:STK_SESS R:TASK_WQ U:TASK_WQ
call trace(16):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db380 <df ff ff e8 a0 f5 ff ff]: ha_stuck_warning+0xd0/0x188 > ha_thread_dump_one
| 0x6c5f7d <44 00 00 e8 33 53 f1 ff]: wdt_handler+0x1dd/0x1ff > ha_stuck_warning
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee4564092 <83 f9 3f 76 73 48 89 d1]: libc:+0x172092
| 0x53b4d0 <45 31 e4 e8 20 9b ef ff]: http_reply_to_htx+0x8f0/0x901 > main-0x18b0
| 0x53dfef <48 89 df e8 f1 cb ff ff]: http_reply_message+0x9f/0x28b > http_reply_to_htx
| 0x53e1fe <48 89 d6 e8 52 fd ff ff]: http_reply_and_close+0x1e/0x2ff > http_reply_message
| 0x542df7 <48 89 c2 e8 e9 b3 ff ff]: http_wait_for_response+0x11e7/0x14d2 > http_reply_and_close
| 0x52514d <44 00 00 e8 c3 ca 01 00]: process_stream+0xa9d/0x3ac3 > http_wait_for_response
=> Trying to gracefully recover now.
WARNING! thread 121 has stopped processing traffic for 295 milliseconds
with 295 streams currently blocked, prevented from making any progress.
While this may occasionally happen with inefficient configurations
involving excess of regular expressions, map_reg, or heavy Lua processing,
this must remain exceptional because the system's stability is now at risk.
Timers in logs may be reported incorrectly, spurious timeouts may happen,
some incoming connections may silently be dropped, health checks may
randomly fail, and accesses to the CLI may block the whole process. The
blocking delay before emitting this warning may be adjusted via the global
'warn-blocked-traffic-after' directive. Please check the trace below for
any clues about configuration elements that need to be corrected:
* Thread 121: id=0x7fdd7dffb640 act=1 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/57 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=737141939679 now=737437495341 diff=295555662
curr_task=0x7fdc50bae3c0 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdc50b728f0
stream=0x7fdc50b728f0: [1754954519.099987] id=5236622 proto=tcpv4 source=187.110.209.206:39784
flags=0xc08, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=(nil), pend_pos=(nil) waiting=0 epoch=0
p_stc=1 p_req=1 p_res=0 p_prp=0
frontend=ssl (id=3 mode=http), listener=? (id=1336) addr=177.126.14.18:80
backend=hyper_vod_tvs (id=15 mode=http)
server=<none> (id=0)
task=0x7fdc50bae3c0 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=120(2/56) age=0s)
txn=0x7fdc50b7e7a0 flags=0x3000 meth=1 status=-1 req.st=MSG_BODY rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdc50befca0 flags=0x00000422 ioto=1m40s state=EST endp=CONN,0x7fdc50e5b910,0x40404001 sub=1 rex=<NEVER> wex=<NEVER> rto=1m40s wto=<NEVER>
h1s=0x7fdc50e5b910 h1s.flg=0x100010 .req.state=MSG_DONE .res.state=MSG_RPBEFORE
.meth=GET status=0 .sd.flg=0x40404001 .sd.evts= .sc.flg=0x00000422 .sc.app=0x7fdc50b728f0 .sc.evts=
.subs=0x7fdc50befcb8(ev=1 tl=0x7fdc50d4a7f0 tl.calls=0 tl.ctx=0x7fdc50befca0 tl.fct=sc_conn_io_cb+0)
h1c=0x7fdc50d30fb0 h1c.flg=0x0 .sub=0 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdc50ef5210 .exp=<NEVER>
co0=0x7fdce4c11680 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=LISTENER:0x4434c8b0
flags=0x80000300 fd=38858 fd.state=120 updt=0 fd.tmask=0x100000000000000
scb=0x7fdc50bf2f90 flags=0x00000821 ioto=0s state=INI endp=NONE,(nil),0x00000010 sub=0 rex=<NEVER> wex=<NEVER> rto=<NEVER> wto=<NEVER>
req=0x7fdc50b72918 (f=0x808001 an=0x900 tofwd=0 total=428)
an_exp=<NEVER> buf=0x7fdc50b72920 data=0x7fdc50f1fd70 o=0 p=0 i=1048576 size=1048576
htx=0x7fdc50f1fd70 flags=0x14 size=1048528 data=601 used=14 wrap=NO extra=0
res=0x7fdc50b72960 (f=0x80000000 an=0x0 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdc50b72968 data=0x7fdc50c12f60 o=0 p=0 i=0 size=1048576
htx=0x7fdc50c12f60 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP
call trace(24):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db380 <df ff ff e8 a0 f5 ff ff]: ha_stuck_warning+0xd0/0x188 > ha_thread_dump_one
| 0x6c5f7d <44 00 00 e8 33 53 f1 ff]: wdt_handler+0x1dd/0x1ff > ha_stuck_warning
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee447d947 <48 83 c9 01 48 89 4a 08]: libc:+0x8b947
| 0x7fdee447ee07 <7c 24 30 e8 b9 e9 ff ff]: libc:+0x8ce07 > libc:+0x8b7c0
| 0x7fdee4480871 <4c 89 e7 e8 0f d8 ff ff]: libc:__libc_malloc+0x1c1/0x327 > libc:+0x8c080
| 0x64517a <89 c0 c3 e8 e6 07 df ff]: pool_get_from_os_noinc+0xca/0xf0 > main-0xf40
| 0x645232 <48 89 fb e8 7e fe ff ff]: pool_alloc_nocache+0x12/0xaa > pool_get_from_os_noinc
| 0x6460df <4c 89 f7 e8 41 f1 ff ff]: __pool_alloc+0x16f/0x279 > pool_alloc_nocache
| 0x46f555 <00 31 f6 e8 1b 6a 1d 00]: jwt_verify+0x115/0x671 > __pool_alloc
| 0x57a8d9 <00 00 00 e8 67 4b ef ff]: main+0x144039 > jwt_verify
| 0x57961f <7b 18 4c 89 ee ff 51 08]: sample_process_cnv+0x6f/0x90
| 0x57969c <4c 89 ef e8 14 ff ff ff]: sample_process+0x5c/0xa7 > sample_process_cnv
| 0x691822 <4c 89 f7 e8 1e 7e ee ff]: acl_exec_cond+0x132/0x249 > sample_process
| 0x53c98d <89 7d c8 e8 63 4d 15 00]: process_stream+0x182dd > acl_exec_cond
| 0x53f696 <89 4d 98 e8 fa d1 ff ff]: http_process_req_common+0xb6/0x1590 > process_stream+0x181e0
| 0x5259cf <4c 89 ff e8 11 9c 01 00]: process_stream+0x131f/0x3ac3 > http_process_req_common
=> Trying to gracefully recover now.
WARNING! thread 112 has stopped processing traffic for 226 milliseconds
with 332 streams currently blocked, prevented from making any progress.
While this may occasionally happen with inefficient configurations
involving excess of regular expressions, map_reg, or heavy Lua processing,
this must remain exceptional because the system's stability is now at risk.
Timers in logs may be reported incorrectly, spurious timeouts may happen,
some incoming connections may silently be dropped, health checks may
randomly fail, and accesses to the CLI may block the whole process. The
blocking delay before emitting this warning may be adjusted via the global
'warn-blocked-traffic-after' directive. Please check the trace below for
any clues about configuration elements that need to be corrected:
* Thread 112: id=0x7fdd877fe640 act=1 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/48 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=739223867359 now=739450379628 diff=226512269
curr_task=0x7fdcf4e2eb70 (task) calls=2 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdcf4b7aa00
stream=0x7fdcf4b7aa00: [1754954516.455063] id=5235082 proto=tcpv4 source=189.124.72.63:39004
flags=0x36c0a, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=(nil), pend_pos=(nil) waiting=0 epoch=0
p_stc=3 p_req=3 p_res=3 p_prp=2
frontend=ssl (id=3 mode=http), listener=? (id=1327) addr=177.126.14.18:80
backend=hyper_streaming_jwt_tvs (id=14 mode=http)
server=v01 (id=1) addr=unix
task=0x7fdcf4e2eb70 (state=0x01 nice=0 calls=2 rate=1 exp=15s tid=111(2/47) age=4s)
txn=0x7fdcf4c97170 flags=0x3000 meth=1 status=502 req.st=MSG_DONE rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdcf4d1d210 flags=0x000004a2 ioto=1m40s state=EST endp=CONN,0x7fdcf4c385c0,0x50404001 sub=1 rex=<NEVER> wex=<NEVER> rto=1m36s wto=<NEVER>
h1s=0x7fdcf4c385c0 h1s.flg=0x100010 .req.state=MSG_DONE .res.state=MSG_RPBEFORE
.meth=GET status=0 .sd.flg=0x50404001 .sd.evts= .sc.flg=0x000004a2 .sc.app=0x7fdcf4b7aa00 .sc.evts=
.subs=0x7fdcf4d1d228(ev=1 tl=0x7fdcf4d7e240 tl.calls=0 tl.ctx=0x7fdcf4d1d210 tl.fct=sc_conn_io_cb+0)
h1c=0x7fdcf4e09210 h1c.flg=0x0 .sub=0 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdcf4bbb680 .exp=<NEVER>
co0=0x7fdcf4df0a30 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=LISTENER:0x4434aed0
flags=0x80000300 fd=3891 fd.state=120 updt=0 fd.tmask=0x800000000000
scb=0x7fdcf4c25ea0 flags=0x00031435 ioto=19s state=CLO endp=CONN,0x7fdcf4ca5c50,0x50410601 sub=0 rex=<NEVER> wex=<NEVER> rto=15s wto=<NEVER>
h1s=0x7fdcf4ca5c50 h1s.flg=0x14810 .req.state=MSG_DONE .res.state=MSG_RPVER
.meth=GET status=0 .sd.flg=0x50410601 .sd.evts=E9E1 .sc.flg=0x00031435 .sc.app=0x7fdcf4b7aa00 .sc.evts=S6S1S9 .subs=(nil)
h1c=0x7fdcf4cf3720 h1c.flg=0x80000000 .sub=1 .ibuf=564@0x7fdcf4e59500+48/1048576 .obuf=0@(nil)+0/0 .evts=M1 .task=0x7fdcf4cb0810 .exp=<NEVER>
co1=0x7fdcf4e1f040 ctrl=abns_stream xprt=RAW mux=H1 data=STRM target=SERVER:0x4402fb10
flags=0x10080300 fd=16885 fd.state=121 updt=1 fd.tmask=0x800000000000
req=0x7fdcf4b7aa28 (f=0x1840100 an=0x8000 tofwd=0 total=997)
an_exp=<NEVER> buf=0x7fdcf4b7aa30 data=(nil) o=0 p=0 i=0 size=0
htx=0x7bcec0 flags=0x0 size=0 data=0 used=0 wrap=NO extra=0
res=0x7fdcf4b7aa70 (f=0x80008001 an=0x1400000 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdcf4b7aa78 data=0x7fdcf4f59510 o=0 p=0 i=0 size=1048576
htx=0x7fdcf4f59510 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: R:PATEXP U:PATEXP W:LBPRM U:LBPRM S:IDLE_CONNS U:IDLE_CONNS W:PROXY U:PROXY
call trace(16):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db380 <df ff ff e8 a0 f5 ff ff]: ha_stuck_warning+0xd0/0x188 > ha_thread_dump_one
| 0x6c5f7d <44 00 00 e8 33 53 f1 ff]: wdt_handler+0x1dd/0x1ff > ha_stuck_warning
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee4564092 <83 f9 3f 76 73 48 89 d1]: libc:+0x172092
| 0x53b4d0 <45 31 e4 e8 20 9b ef ff]: http_reply_to_htx+0x8f0/0x901 > main-0x18b0
| 0x53dfef <48 89 df e8 f1 cb ff ff]: http_reply_message+0x9f/0x28b > http_reply_to_htx
| 0x53e1fe <48 89 d6 e8 52 fd ff ff]: http_reply_and_close+0x1e/0x2ff > http_reply_message
| 0x541ebc <48 89 c2 e8 24 c3 ff ff]: http_wait_for_response+0x2ac/0x14d2 > http_reply_and_close
| 0x52514d <44 00 00 e8 c3 ca 01 00]: process_stream+0xa9d/0x3ac3 > http_wait_for_response
=> Trying to gracefully recover now.
[ALERT] (1859117) : Lua function 'redis-valida': [state-id 57] runtime error: /usr/share/lua/5.4/redis.lua:309: connection error: connection closed. from [C]: in global 'error', /usr/share/lua/5.4/redis.lua:851: in field 'error', /usr/share/lua/5.4/redis.lua:309: in field 'read', /usr/share/lua/5.4/redis.lua:315: in field 'read', /usr/share/lua/5.4/redis.lua:425: in method 'srem', /etc/haproxy/valida_conjunto.lua:71: in function line 4.
WARNING! thread 39 has stopped processing traffic for 314 milliseconds
with 257 streams currently blocked, prevented from making any progress.
While this may occasionally happen with inefficient configurations
involving excess of regular expressions, map_reg, or heavy Lua processing,
this must remain exceptional because the system's stability is now at risk.
Timers in logs may be reported incorrectly, spurious timeouts may happen,
some incoming connections may silently be dropped, health checks may
randomly fail, and accesses to the CLI may block the whole process. The
blocking delay before emitting this warning may be adjusted via the global
'warn-blocked-traffic-after' directive. Please check the trace below for
any clues about configuration elements that need to be corrected:
* Thread 39: id=0x7fde6bfff640 act=1 glob=0 wq=1 rq=0 tl=1 tlsz=1 rqsz=2
1/39 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=1129926982781 now=1130241949637 diff=314966856
curr_task=0x7fdd6cbca800 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdd6cbca3d0
stream=0x7fdd6cbca3d0: [1754954766.813818] id=5484795 proto=?
flags=0x100c4a, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=0x44207490, pend_pos=(nil) waiting=0 epoch=0
p_stc=2 p_req=2 p_res=2 p_prp=2
frontend=<HTTPCLIENT> (id=4294967295 mode=http), listener=? (id=0)
backend=<HTTPCLIENT> (id=4294967295 mode=http)
server=<HTTPCLIENT> (id=1) addr=179.96.0.91:95
task=0x7fdd6cbca800 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=38(1/38) age=0s)
txn=0x7fdd6ced4f40 flags=0x83000 meth=1 status=-1 req.st=MSG_DONE rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdd6ced4ee0 flags=0x00000460 ioto=0s state=EST endp=APPCTX,0x7fdd6cee6090,0x10000002 sub=0 rex=<NEVER> wex=<NEVER> rto=0s wto=<NEVER>
app0=0x7fdd6cee6090 st0=2 st1=0 applet=<HTTPCLIENT> tid=38 nice=0 calls=0 rate=0
scb=0x7fdd6cbca890 flags=0x00001c79 ioto=0s state=EST endp=CONN,0x7fdd6cb13d50,0x40400001 sub=0 rex=0s wex=<NEVER> rto=0s wto=<NEVER>
h1s=0x7fdd6cb13d50 h1s.flg=0x100090 .req.state=MSG_RQBEFORE .res.state=MSG_RPBEFORE
.meth=UNKNOWN status=0 .sd.flg=0x40400001 .sd.evts= .sc.flg=0x00001c79 .sc.app=0x7fdd6cbca3d0 .sc.evts= .subs=(nil)
h1c=0x7fdd20fe0530 h1c.flg=0x80000000 .sub=1 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdd6ced5030 .exp=<NEVER>
co1=0x7fdd20b41ad0 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=SERVER:0x44207490
flags=0x00000300 fd=74353 fd.state=10121 updt=1 fd.tmask=0x4000000000
req=0x7fdd6cbca3f8 (f=0x1800001 an=0x8000 tofwd=0 total=161)
an_exp=<NEVER> buf=0x7fdd6cbca400 data=0x7fdd6d313320 o=161 p=161 i=1048415 size=1048576
htx=0x7fdd6d313320 flags=0x10 size=1048528 data=161 used=5 wrap=NO extra=0
res=0x7fdd6cbca440 (f=0x80000001 an=0x1400000 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdd6cbca448 data=0x7fdd6d213310 o=0 p=0 i=0 size=1048576
htx=0x7fdd6d213310 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: S:NOTIF U:NOTIF R:TASK_WQ U:TASK_WQ S:NOTIF U:NOTIF S:IDLE_CONNS U:IDLE_CONNS
call trace(13):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db380 <df ff ff e8 a0 f5 ff ff]: ha_stuck_warning+0xd0/0x188 > ha_thread_dump_one
| 0x6c5f7d <44 00 00 e8 33 53 f1 ff]: wdt_handler+0x1dd/0x1ff > ha_stuck_warning
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee4564092 <83 f9 3f 76 73 48 89 d1]: libc:+0x172092
| 0x5c90c5 <89 4d c8 e8 2b bf e6 ff]: cli_io_handler+0x2da75 > main-0x18b0
| 0x5263ac <48 89 df e8 34 71 0a 00]: process_stream+0x1cfc/0x3ac3 > sc_conn_sync_send
=> Trying to gracefully recover now.
WARNING! thread 63 has stopped processing traffic for 244 milliseconds
with 231 streams currently blocked, prevented from making any progress.
While this may occasionally happen with inefficient configurations
involving excess of regular expressions, map_reg, or heavy Lua processing,
this must remain exceptional because the system's stability is now at risk.
Timers in logs may be reported incorrectly, spurious timeouts may happen,
some incoming connections may silently be dropped, health checks may
randomly fail, and accesses to the CLI may block the whole process. The
blocking delay before emitting this warning may be adjusted via the global
'warn-blocked-traffic-after' directive. Please check the trace below for
any clues about configuration elements that need to be corrected:
* Thread 63: id=0x7fde26f66640 act=1 glob=0 wq=1 rq=0 tl=1 tlsz=3 rqsz=4
1/63 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=1062429188846 now=1062673906934 diff=244718088
curr_task=0x7fdcf0e6e6b0 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdcf0f15300
stream=0x7fdcf0f15300: [1754954767.440684] id=5485195 proto=tcpv4 source=191.243.211.0:11589
flags=0xc08, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=(nil), pend_pos=(nil) waiting=0 epoch=0
p_stc=1 p_req=1 p_res=0 p_prp=0
frontend=ssl (id=3 mode=http), listener=? (id=894) addr=179.96.0.90:80
backend=hyper_vod_tvs (id=15 mode=http)
server=<none> (id=0)
task=0x7fdcf0e6e6b0 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=62(1/62) age=0s)
txn=0x7fdcf0e6e810 flags=0x3000 meth=1 status=-1 req.st=MSG_BODY rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdcf0e38690 flags=0x00000422 ioto=1m40s state=EST endp=CONN,0x7fdcf0e6e740,0x40404001 sub=1 rex=<NEVER> wex=<NEVER> rto=1m40s wto=<NEVER>
h1s=0x7fdcf0e6e740 h1s.flg=0x100010 .req.state=MSG_DONE .res.state=MSG_RPBEFORE
.meth=GET status=0 .sd.flg=0x40404001 .sd.evts= .sc.flg=0x00000422 .sc.app=0x7fdcf0f15300 .sc.evts=
.subs=0x7fdcf0e386a8(ev=1 tl=0x7fdcf0bc7980 tl.calls=0 tl.ctx=0x7fdcf0e38690 tl.fct=sc_conn_io_cb+0)
h1c=0x7fdcf0c452a0 h1c.flg=0x0 .sub=0 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdcf0c45370 .exp=<NEVER>
co0=0x7fdcf0e66d70 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=LISTENER:0x442fe610
flags=0x80000300 fd=22072 fd.state=20 updt=0 fd.tmask=0x4000000000000000
scb=0x7fdcf0e3e1e0 flags=0x00000821 ioto=0s state=INI endp=NONE,(nil),0x00000010 sub=0 rex=<NEVER> wex=<NEVER> rto=<NEVER> wto=<NEVER>
req=0x7fdcf0f15328 (f=0x808001 an=0x900 tofwd=0 total=649)
an_exp=<NEVER> buf=0x7fdcf0f15330 data=0x7fdcf114f400 o=0 p=0 i=1048576 size=1048576
htx=0x7fdcf114f400 flags=0x14 size=1048528 data=816 used=14 wrap=NO extra=0
res=0x7fdcf0f15370 (f=0x80000000 an=0x0 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdcf0f15378 data=0x7fdcf0d22350 o=0 p=0 i=0 size=1048576
htx=0x7fdcf0d22350 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: R:TASK_WQ U:TASK_WQ R:LISTENER U:LISTENER R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP
call trace(20):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db380 <df ff ff e8 a0 f5 ff ff]: ha_stuck_warning+0xd0/0x188 > ha_thread_dump_one
| 0x6c5f7d <44 00 00 e8 33 53 f1 ff]: wdt_handler+0x1dd/0x1ff > ha_stuck_warning
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee447d947 <48 83 c9 01 48 89 4a 08]: libc:+0x8b947
| 0x7fdee447ee07 <7c 24 30 e8 b9 e9 ff ff]: libc:+0x8ce07 > libc:+0x8b7c0
| 0x7fdee4480871 <4c 89 e7 e8 0f d8 ff ff]: libc:__libc_malloc+0x1c1/0x327 > libc:+0x8c080
| 0x64517a <89 c0 c3 e8 e6 07 df ff]: pool_get_from_os_noinc+0xca/0xf0 > main-0xf40
| 0x645232 <48 89 fb e8 7e fe ff ff]: pool_alloc_nocache+0x12/0xaa > pool_get_from_os_noinc
| 0x6460df <4c 89 f7 e8 41 f1 ff ff]: __pool_alloc+0x16f/0x279 > pool_alloc_nocache
| 0x63403c <00 31 f6 e8 34 1f 01 00]: task_run_applet+0x3e2dc > __pool_alloc
| 0x53ca1c <89 ef 4c 89 5d c8 ff d0]: process_stream+0x1836c
| 0x53f696 <89 4d 98 e8 fa d1 ff ff]: http_process_req_common+0xb6/0x1590 > process_stream+0x181e0
| 0x5259cf <4c 89 ff e8 11 9c 01 00]: process_stream+0x131f/0x3ac3 > http_process_req_common
=> Trying to gracefully recover now.
WARNING! thread 9 has stopped processing traffic for 283 milliseconds
with 235 streams currently blocked, prevented from making any progress.
While this may occasionally happen with inefficient configurations
involving excess of regular expressions, map_reg, or heavy Lua processing,
this must remain exceptional because the system's stability is now at risk.
Timers in logs may be reported incorrectly, spurious timeouts may happen,
some incoming connections may silently be dropped, health checks may
randomly fail, and accesses to the CLI may block the whole process. The
blocking delay before emitting this warning may be adjusted via the global
'warn-blocked-traffic-after' directive. Please check the trace below for
any clues about configuration elements that need to be corrected:
* Thread 9 : id=0x7fdebffff640 act=1 glob=0 wq=1 rq=0 tl=1 tlsz=2 rqsz=4
1/9 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=1147925222513 now=1148208858969 diff=283636456
curr_task=0x7fdeb4badc30 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdeb4bc73b0
stream=0x7fdeb4bc73b0: [1754954767.570942] id=5485288 proto=?
flags=0x100c4a, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=0x44207490, pend_pos=(nil) waiting=0 epoch=0
p_stc=2 p_req=2 p_res=2 p_prp=2
frontend=<HTTPCLIENT> (id=4294967295 mode=http), listener=? (id=0)
backend=<HTTPCLIENT> (id=4294967295 mode=http)
server=<HTTPCLIENT> (id=1) addr=38.211.248.3:95
task=0x7fdeb4badc30 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=8(1/8) age=0s)
txn=0x7fdeb4badd90 flags=0x83000 meth=1 status=-1 req.st=MSG_DONE rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdeb4ec77d0 flags=0x00000460 ioto=0s state=EST endp=APPCTX,0x7fdeb4b490f0,0x10000002 sub=0 rex=<NEVER> wex=<NEVER> rto=0s wto=<NEVER>
app0=0x7fdeb4b490f0 st0=2 st1=0 applet=<HTTPCLIENT> tid=8 nice=0 calls=0 rate=0
scb=0x7fdeb4baa130 flags=0x00001c79 ioto=0s state=EST endp=CONN,0x7fdeb4c51de0,0x40400001 sub=0 rex=0s wex=<NEVER> rto=0s wto=<NEVER>
h1s=0x7fdeb4c51de0 h1s.flg=0x100090 .req.state=MSG_RQBEFORE .res.state=MSG_RPBEFORE
.meth=UNKNOWN status=0 .sd.flg=0x40400001 .sd.evts= .sc.flg=0x00001c79 .sc.app=0x7fdeb4bc73b0 .sc.evts= .subs=(nil)
h1c=0x7fde991e64e0 h1c.flg=0x80000000 .sub=1 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdeb4ef9830 .exp=<NEVER>
co1=0x7fde98bb2dd0 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=SERVER:0x44207490
flags=0x00000300 fd=74476 fd.state=10121 updt=1 fd.tmask=0x100
req=0x7fdeb4bc73d8 (f=0x1800001 an=0x8000 tofwd=0 total=164)
an_exp=<NEVER> buf=0x7fdeb4bc73e0 data=0x7fdeb4d21bb0 o=164 p=164 i=1048412 size=1048576
htx=0x7fdeb4d21bb0 flags=0x10 size=1048528 data=164 used=5 wrap=NO extra=0
res=0x7fdeb4bc7420 (f=0x80000001 an=0x1400000 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdeb4bc7428 data=0x7fdeb511a1b0 o=0 p=0 i=0 size=1048576
htx=0x7fdeb511a1b0 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: S:NOTIF U:NOTIF S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ S:IDLE_CONNS U:IDLE_CONNS
call trace(13):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db380 <df ff ff e8 a0 f5 ff ff]: ha_stuck_warning+0xd0/0x188 > ha_thread_dump_one
| 0x6c5f7d <44 00 00 e8 33 53 f1 ff]: wdt_handler+0x1dd/0x1ff > ha_stuck_warning
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee4564092 <83 f9 3f 76 73 48 89 d1]: libc:+0x172092
| 0x5c90c5 <89 4d c8 e8 2b bf e6 ff]: cli_io_handler+0x2da75 > main-0x18b0
| 0x5263ac <48 89 df e8 34 71 0a 00]: process_stream+0x1cfc/0x3ac3 > sc_conn_sync_send
=> Trying to gracefully recover now.
WARNING! thread 13 has stopped processing traffic for 263 milliseconds
with 232 streams currently blocked, prevented from making any progress.
While this may occasionally happen with inefficient configurations
involving excess of regular expressions, map_reg, or heavy Lua processing,
this must remain exceptional because the system's stability is now at risk.
Timers in logs may be reported incorrectly, spurious timeouts may happen,
some incoming connections may silently be dropped, health checks may
randomly fail, and accesses to the CLI may block the whole process. The
blocking delay before emitting this warning may be adjusted via the global
'warn-blocked-traffic-after' directive. Please check the trace below for
any clues about configuration elements that need to be corrected:
* Thread 13: id=0x7fdebef66640 act=1 glob=0 wq=1 rq=0 tl=1 tlsz=2 rqsz=3
1/13 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=1120013665262 now=1120277148996 diff=263483734
curr_task=0x7fdea4b70110 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdea4b6fd20
stream=0x7fdea4b6fd20: [1754954767.896114] id=5485506 proto=?
flags=0x100c4a, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=0x44207490, pend_pos=(nil) waiting=0 epoch=0
p_stc=2 p_req=2 p_res=2 p_prp=2
frontend=<HTTPCLIENT> (id=4294967295 mode=http), listener=? (id=0)
backend=<HTTPCLIENT> (id=4294967295 mode=http)
server=<HTTPCLIENT> (id=1) addr=179.96.0.91:95
task=0x7fdea4b70110 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=12(1/12) age=0s)
txn=0x7fdea4bdd9e0 flags=0x83000 meth=1 status=-1 req.st=MSG_DONE rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdea4e0d150 flags=0x00000460 ioto=0s state=EST endp=APPCTX,0x3facd030,0x10000002 sub=0 rex=<NEVER> wex=<NEVER> rto=0s wto=<NEVER>
app0=0x3facd030 st0=2 st1=0 applet=<HTTPCLIENT> tid=12 nice=0 calls=0 rate=0
scb=0x7fdea4b701a0 flags=0x00001c79 ioto=0s state=EST endp=CONN,0x7fdea4bfc020,0x40400001 sub=0 rex=0s wex=<NEVER> rto=0s wto=<NEVER>
h1s=0x7fdea4bfc020 h1s.flg=0x100090 .req.state=MSG_RQBEFORE .res.state=MSG_RPBEFORE
.meth=UNKNOWN status=0 .sd.flg=0x40400001 .sd.evts= .sc.flg=0x00001c79 .sc.app=0x7fdea4b6fd20 .sc.evts= .subs=(nil)
h1c=0x7fde8cb42800 h1c.flg=0x80000000 .sub=1 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdea4bddad0 .exp=<NEVER>
co1=0x7fde8cbd9760 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=SERVER:0x44207490
flags=0x00000300 fd=77262 fd.state=10121 updt=1 fd.tmask=0x1000
req=0x7fdea4b6fd48 (f=0x1800001 an=0x8000 tofwd=0 total=161)
an_exp=<NEVER> buf=0x7fdea4b6fd50 data=0x7fdea5415670 o=161 p=161 i=1048415 size=1048576
htx=0x7fdea5415670 flags=0x10 size=1048528 data=161 used=5 wrap=NO extra=0
res=0x7fdea4b6fd90 (f=0x80000001 an=0x1400000 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdea4b6fd98 data=0x7fdea5315660 o=0 p=0 i=0 size=1048576
htx=0x7fdea5315660 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: S:NOTIF U:NOTIF R:TASK_WQ U:TASK_WQ S:NOTIF U:NOTIF S:IDLE_CONNS U:IDLE_CONNS
call trace(13):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db380 <df ff ff e8 a0 f5 ff ff]: ha_stuck_warning+0xd0/0x188 > ha_thread_dump_one
| 0x6c5f7d <44 00 00 e8 33 53 f1 ff]: wdt_handler+0x1dd/0x1ff > ha_stuck_warning
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee4564092 <83 f9 3f 76 73 48 89 d1]: libc:+0x172092
| 0x5c90c5 <89 4d c8 e8 2b bf e6 ff]: cli_io_handler+0x2da75 > main-0x18b0
| 0x5263ac <48 89 df e8 34 71 0a 00]: process_stream+0x1cfc/0x3ac3 > sc_conn_sync_send
=> Trying to gracefully recover now.
WARNING! thread 117 has stopped processing traffic for 396 milliseconds
with 255 streams currently blocked, prevented from making any progress.
While this may occasionally happen with inefficient configurations
involving excess of regular expressions, map_reg, or heavy Lua processing,
this must remain exceptional because the system's stability is now at risk.
Timers in logs may be reported incorrectly, spurious timeouts may happen,
some incoming connections may silently be dropped, health checks may
randomly fail, and accesses to the CLI may block the whole process. The
blocking delay before emitting this warning may be adjusted via the global
'warn-blocked-traffic-after' directive. Please check the trace below for
any clues about configuration elements that need to be corrected:
* Thread 117: id=0x7fdd7ffff640 act=1 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/53 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=831735195069 now=832131363426 diff=396168357
curr_task=0x7fdc4cb404b0 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdc4cb74040
stream=0x7fdc4cb74040: [1754954768.379854] id=5485776 proto=tcpv4 source=179.125.157.183:40966
flags=0xc08, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=(nil), pend_pos=(nil) waiting=0 epoch=0
p_stc=1 p_req=1 p_res=0 p_prp=0
frontend=ssl (id=3 mode=http), listener=? (id=1332) addr=177.126.14.18:80
backend=hyper_vod_tvs (id=15 mode=http)
server=<none> (id=0)
task=0x7fdc4cb404b0 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=116(2/52) age=0s)
txn=0x7fdc4cf70370 flags=0x3000 meth=1 status=-1 req.st=MSG_BODY rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdc4cbbc9f0 flags=0x00000422 ioto=1m40s state=EST endp=CONN,0x7fdc4cf701d0,0x40404001 sub=1 rex=<NEVER> wex=<NEVER> rto=1m40s wto=<NEVER>
h1s=0x7fdc4cf701d0 h1s.flg=0x100010 .req.state=MSG_DONE .res.state=MSG_RPBEFORE
.meth=GET status=0 .sd.flg=0x40404001 .sd.evts= .sc.flg=0x00000422 .sc.app=0x7fdc4cb74040 .sc.evts=
.subs=0x7fdc4cbbca08(ev=1 tl=0x7fdc4cf702a0 tl.calls=0 tl.ctx=0x7fdc4cbbc9f0 tl.fct=sc_conn_io_cb+0)
h1c=0x7fdc4cb403e0 h1c.flg=0x0 .sub=0 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdc4cf9e390 .exp=<NEVER>
co0=0x7fdc4cf9e2a0 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=LISTENER:0x4434bd30
flags=0x80000300 fd=43730 fd.state=120 updt=0 fd.tmask=0x10000000000000
scb=0x7fdc4cb6e1b0 flags=0x00000821 ioto=0s state=INI endp=NONE,(nil),0x00000010 sub=0 rex=<NEVER> wex=<NEVER> rto=<NEVER> wto=<NEVER>
req=0x7fdc4cb74068 (f=0x808001 an=0x900 tofwd=0 total=419)
an_exp=<NEVER> buf=0x7fdc4cb74070 data=0x7fdc4cd12f70 o=0 p=0 i=1048576 size=1048576
htx=0x7fdc4cd12f70 flags=0x14 size=1048528 data=592 used=14 wrap=NO extra=0
res=0x7fdc4cb740b0 (f=0x80000000 an=0x0 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdc4cb740b8 data=0x7fdc4cc12f60 o=0 p=0 i=0 size=1048576
htx=0x7fdc4cc12f60 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: R:STK_TABLE U:STK_TABLE R:STK_SESS U:STK_SESS R:LISTENER U:LISTENER R:PATEXP U:PATEXP
call trace(20):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db380 <df ff ff e8 a0 f5 ff ff]: ha_stuck_warning+0xd0/0x188 > ha_thread_dump_one
| 0x6c5f7d <44 00 00 e8 33 53 f1 ff]: wdt_handler+0x1dd/0x1ff > ha_stuck_warning
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee447d947 <48 83 c9 01 48 89 4a 08]: libc:+0x8b947
| 0x7fdee447ee07 <7c 24 30 e8 b9 e9 ff ff]: libc:+0x8ce07 > libc:+0x8b7c0
| 0x7fdee4480871 <4c 89 e7 e8 0f d8 ff ff]: libc:__libc_malloc+0x1c1/0x327 > libc:+0x8c080
| 0x64517a <89 c0 c3 e8 e6 07 df ff]: pool_get_from_os_noinc+0xca/0xf0 > main-0xf40
| 0x645232 <48 89 fb e8 7e fe ff ff]: pool_alloc_nocache+0x12/0xaa > pool_get_from_os_noinc
| 0x6460df <4c 89 f7 e8 41 f1 ff ff]: __pool_alloc+0x16f/0x279 > pool_alloc_nocache
| 0x63403c <00 31 f6 e8 34 1f 01 00]: task_run_applet+0x3e2dc > __pool_alloc
| 0x53ca1c <89 ef 4c 89 5d c8 ff d0]: process_stream+0x1836c
| 0x53f696 <89 4d 98 e8 fa d1 ff ff]: http_process_req_common+0xb6/0x1590 > process_stream+0x181e0
| 0x5259cf <4c 89 ff e8 11 9c 01 00]: process_stream+0x131f/0x3ac3 > http_process_req_common
=> Trying to gracefully recover now.
WARNING! thread 111 has stopped processing traffic for 365 milliseconds
with 263 streams currently blocked, prevented from making any progress.
While this may occasionally happen with inefficient configurations
involving excess of regular expressions, map_reg, or heavy Lua processing,
this must remain exceptional because the system's stability is now at risk.
Timers in logs may be reported incorrectly, spurious timeouts may happen,
some incoming connections may silently be dropped, health checks may
randomly fail, and accesses to the CLI may block the whole process. The
blocking delay before emitting this warning may be adjusted via the global
'warn-blocked-traffic-after' directive. Please check the trace below for
any clues about configuration elements that need to be corrected:
* Thread 111: id=0x7fdd87fff640 act=1 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/47 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=784419268434 now=784784730313 diff=365461879
curr_task=0x7fdd30d6f4c0 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdd30dbe0d0
stream=0x7fdd30dbe0d0: [1754954768.458553] id=5485820 proto=tcpv4 source=191.15.63.101:46076
flags=0xc08, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=(nil), pend_pos=(nil) waiting=0 epoch=0
p_stc=1 p_req=1 p_res=0 p_prp=0
frontend=ssl (id=3 mode=http), listener=? (id=1326) addr=177.126.14.18:80
backend=hyper_vod_tvs (id=15 mode=http)
server=<none> (id=0)
task=0x7fdd30d6f4c0 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=110(2/46) age=0s)
txn=0x7fdd30dbe4c0 flags=0x3000 meth=1 status=-1 req.st=MSG_BODY rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdd30b131e0 flags=0x00000422 ioto=1m40s state=EST endp=CONN,0x7fdd30dbe000,0x40404001 sub=1 rex=<NEVER> wex=<NEVER> rto=1m40s wto=<NEVER>
h1s=0x7fdd30dbe000 h1s.flg=0x100010 .req.state=MSG_DONE .res.state=MSG_RPBEFORE
.meth=GET status=0 .sd.flg=0x40404001 .sd.evts= .sc.flg=0x00000422 .sc.app=0x7fdd30dbe0d0 .sc.evts=
.subs=0x7fdd30b131f8(ev=1 tl=0x7fdd30d6f550 tl.calls=0 tl.ctx=0x7fdd30b131e0 tl.fct=sc_conn_io_cb+0)
h1c=0x7fdd30d6f310 h1c.flg=0x0 .sub=0 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdd30d6f430 .exp=<NEVER>
co0=0x7fdd30d6f150 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=LISTENER:0x4434abf0
flags=0x80000300 fd=78826 fd.state=20 updt=0 fd.tmask=0x400000000000
scb=0x7fdd30b13180 flags=0x00000821 ioto=0s state=INI endp=NONE,(nil),0x00000010 sub=0 rex=<NEVER> wex=<NEVER> rto=<NEVER> wto=<NEVER>
req=0x7fdd30dbe0f8 (f=0x808001 an=0x900 tofwd=0 total=767)
an_exp=<NEVER> buf=0x7fdd30dbe100 data=0x7fdd30f14e20 o=0 p=0 i=1048576 size=1048576
htx=0x7fdd30f14e20 flags=0x14 size=1048528 data=936 used=14 wrap=NO extra=0
res=0x7fdd30dbe140 (f=0x80000000 an=0x0 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdd30dbe148 data=0x7fdd30c145b0 o=0 p=0 i=0 size=1048576
htx=0x7fdd30c145b0 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP
call trace(20):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db380 <df ff ff e8 a0 f5 ff ff]: ha_stuck_warning+0xd0/0x188 > ha_thread_dump_one
| 0x6c5f7d <44 00 00 e8 33 53 f1 ff]: wdt_handler+0x1dd/0x1ff > ha_stuck_warning
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee447d947 <48 83 c9 01 48 89 4a 08]: libc:+0x8b947
| 0x7fdee447ee07 <7c 24 30 e8 b9 e9 ff ff]: libc:+0x8ce07 > libc:+0x8b7c0
| 0x7fdee4480871 <4c 89 e7 e8 0f d8 ff ff]: libc:__libc_malloc+0x1c1/0x327 > libc:+0x8c080
| 0x64517a <89 c0 c3 e8 e6 07 df ff]: pool_get_from_os_noinc+0xca/0xf0 > main-0xf40
| 0x645232 <48 89 fb e8 7e fe ff ff]: pool_alloc_nocache+0x12/0xaa > pool_get_from_os_noinc
| 0x6460df <4c 89 f7 e8 41 f1 ff ff]: __pool_alloc+0x16f/0x279 > pool_alloc_nocache
| 0x63403c <00 31 f6 e8 34 1f 01 00]: task_run_applet+0x3e2dc > __pool_alloc
| 0x53ca1c <89 ef 4c 89 5d c8 ff d0]: process_stream+0x1836c
| 0x53f696 <89 4d 98 e8 fa d1 ff ff]: http_process_req_common+0xb6/0x1590 > process_stream+0x181e0
| 0x5259cf <4c 89 ff e8 11 9c 01 00]: process_stream+0x131f/0x3ac3 > http_process_req_common
=> Trying to gracefully recover now.
WARNING! thread 13 has stopped processing traffic for 346 milliseconds
with 243 streams currently blocked, prevented from making any progress.
While this may occasionally happen with inefficient configurations
involving excess of regular expressions, map_reg, or heavy Lua processing,
this must remain exceptional because the system's stability is now at risk.
Timers in logs may be reported incorrectly, spurious timeouts may happen,
some incoming connections may silently be dropped, health checks may
randomly fail, and accesses to the CLI may block the whole process. The
blocking delay before emitting this warning may be adjusted via the global
'warn-blocked-traffic-after' directive. Please check the trace below for
any clues about configuration elements that need to be corrected:
* Thread 13: id=0x7fdebef66640 act=1 glob=0 wq=1 rq=0 tl=1 tlsz=1 rqsz=2
1/13 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=1122022213538 now=1122368325237 diff=346111699
curr_task=0x7fdea4d22430 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdea4d22000
stream=0x7fdea4d22000: [1754954771.961547] id=5488291 proto=?
flags=0x100c4a, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=0x44207490, pend_pos=(nil) waiting=0 epoch=0
p_stc=2 p_req=2 p_res=2 p_prp=2
frontend=<HTTPCLIENT> (id=4294967295 mode=http), listener=? (id=0)
backend=<HTTPCLIENT> (id=4294967295 mode=http)
server=<HTTPCLIENT> (id=1) addr=45.226.210.12:95
task=0x7fdea4d22430 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=12(1/12) age=0s)
txn=0x7fdea4be26f0 flags=0x83000 meth=1 status=-1 req.st=MSG_DONE rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdea4be2690 flags=0x00000460 ioto=0s state=EST endp=APPCTX,0x7fdea4d6d130,0x10000002 sub=0 rex=<NEVER> wex=<NEVER> rto=0s wto=<NEVER>
app0=0x7fdea4d6d130 st0=2 st1=0 applet=<HTTPCLIENT> tid=12 nice=0 calls=0 rate=0
scb=0x7fdea4d224c0 flags=0x00001c79 ioto=0s state=EST endp=CONN,0x7fdea4d094b0,0x40400001 sub=0 rex=0s wex=<NEVER> rto=0s wto=<NEVER>
h1s=0x7fdea4d094b0 h1s.flg=0x100090 .req.state=MSG_RQBEFORE .res.state=MSG_RPBEFORE
.meth=UNKNOWN status=0 .sd.flg=0x40400001 .sd.evts= .sc.flg=0x00001c79 .sc.app=0x7fdea4d22000 .sc.evts= .subs=(nil)
h1c=0x7fdea4b34850 h1c.flg=0x80000000 .sub=1 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdea4b34970 .exp=<NEVER>
co1=0x7fdea4e7bdc0 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=SERVER:0x44207490
flags=0x00000300 fd=17354 fd.state=10121 updt=0 fd.tmask=0x1000
req=0x7fdea4d22028 (f=0x1800001 an=0x8000 tofwd=0 total=165)
an_exp=<NEVER> buf=0x7fdea4d22030 data=0x7fdea5215650 o=165 p=165 i=1048411 size=1048576
htx=0x7fdea5215650 flags=0x10 size=1048528 data=165 used=5 wrap=NO extra=0
res=0x7fdea4d22070 (f=0x80000001 an=0x1400000 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdea4d22078 data=0x7fdea5115640 o=0 p=0 i=0 size=1048576
htx=0x7fdea5115640 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: S:NOTIF U:NOTIF R:TASK_WQ U:TASK_WQ S:NOTIF U:NOTIF S:IDLE_CONNS U:IDLE_CONNS
call trace(13):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db380 <df ff ff e8 a0 f5 ff ff]: ha_stuck_warning+0xd0/0x188 > ha_thread_dump_one
| 0x6c5f7d <44 00 00 e8 33 53 f1 ff]: wdt_handler+0x1dd/0x1ff > ha_stuck_warning
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee4564092 <83 f9 3f 76 73 48 89 d1]: libc:+0x172092
| 0x5c90c5 <89 4d c8 e8 2b bf e6 ff]: cli_io_handler+0x2da75 > main-0x18b0
| 0x5263ac <48 89 df e8 34 71 0a 00]: process_stream+0x1cfc/0x3ac3 > sc_conn_sync_send
=> Trying to gracefully recover now.
WARNING! thread 13 has stopped processing traffic for 464 milliseconds
with 243 streams currently blocked, prevented from making any progress.
While this may occasionally happen with inefficient configurations
involving excess of regular expressions, map_reg, or heavy Lua processing,
this must remain exceptional because the system's stability is now at risk.
Timers in logs may be reported incorrectly, spurious timeouts may happen,
some incoming connections may silently be dropped, health checks may
randomly fail, and accesses to the CLI may block the whole process. The
blocking delay before emitting this warning may be adjusted via the global
'warn-blocked-traffic-after' directive. Please check the trace below for
any clues about configuration elements that need to be corrected:
* Thread 13: id=0x7fdebef66640 act=1 glob=0 wq=1 rq=0 tl=1 tlsz=1 rqsz=2
1/13 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=1122022213538 now=1122486217415 diff=464003877
curr_task=0x7fdea4d22430 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdea4d22000
stream=0x7fdea4d22000: [1754954771.961547] id=5488291 proto=?
flags=0x100c4a, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=0x44207490, pend_pos=(nil) waiting=0 epoch=0
p_stc=2 p_req=2 p_res=2 p_prp=2
frontend=<HTTPCLIENT> (id=4294967295 mode=http), listener=? (id=0)
backend=<HTTPCLIENT> (id=4294967295 mode=http)
server=<HTTPCLIENT> (id=1) addr=45.226.210.12:95
task=0x7fdea4d22430 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=12(1/12) age=0s)
txn=0x7fdea4be26f0 flags=0x83000 meth=1 status=-1 req.st=MSG_DONE rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdea4be2690 flags=0x00000460 ioto=0s state=EST endp=APPCTX,0x7fdea4d6d130,0x10000002 sub=0 rex=<NEVER> wex=<NEVER> rto=0s wto=<NEVER>
app0=0x7fdea4d6d130 st0=2 st1=0 applet=<HTTPCLIENT> tid=12 nice=0 calls=0 rate=0
scb=0x7fdea4d224c0 flags=0x00001c79 ioto=0s state=EST endp=CONN,0x7fdea4d094b0,0x40400001 sub=0 rex=0s wex=<NEVER> rto=0s wto=<NEVER>
h1s=0x7fdea4d094b0 h1s.flg=0x100090 .req.state=MSG_RQBEFORE .res.state=MSG_RPBEFORE
.meth=UNKNOWN status=0 .sd.flg=0x40400001 .sd.evts= .sc.flg=0x00001c79 .sc.app=0x7fdea4d22000 .sc.evts= .subs=(nil)
h1c=0x7fdea4b34850 h1c.flg=0x80000000 .sub=1 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdea4b34970 .exp=<NEVER>
co1=0x7fdea4e7bdc0 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=SERVER:0x44207490
flags=0x00000300 fd=17354 fd.state=10121 updt=0 fd.tmask=0x1000
req=0x7fdea4d22028 (f=0x1800001 an=0x8000 tofwd=0 total=165)
an_exp=<NEVER> buf=0x7fdea4d22030 data=0x7fdea5215650 o=165 p=165 i=1048411 size=1048576
htx=0x7fdea5215650 flags=0x10 size=1048528 data=165 used=5 wrap=NO extra=0
res=0x7fdea4d22070 (f=0x80000001 an=0x1400000 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdea4d22078 data=0x7fdea5115640 o=0 p=0 i=0 size=1048576
htx=0x7fdea5115640 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: S:NOTIF U:NOTIF R:TASK_WQ U:TASK_WQ S:NOTIF U:NOTIF S:IDLE_CONNS U:IDLE_CONNS
call trace(13):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db380 <df ff ff e8 a0 f5 ff ff]: ha_stuck_warning+0xd0/0x188 > ha_thread_dump_one
| 0x6c5f7d <44 00 00 e8 33 53 f1 ff]: wdt_handler+0x1dd/0x1ff > ha_stuck_warning
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee4564092 <83 f9 3f 76 73 48 89 d1]: libc:+0x172092
| 0x5c90c5 <89 4d c8 e8 2b bf e6 ff]: cli_io_handler+0x2da75 > main-0x18b0
| 0x5263ac <48 89 df e8 34 71 0a 00]: process_stream+0x1cfc/0x3ac3 > sc_conn_sync_send
=> Trying to gracefully recover now.
WARNING! thread 42 has stopped processing traffic for 300 milliseconds
with 247 streams currently blocked, prevented from making any progress.
While this may occasionally happen with inefficient configurations
involving excess of regular expressions, map_reg, or heavy Lua processing,
this must remain exceptional because the system's stability is now at risk.
Timers in logs may be reported incorrectly, spurious timeouts may happen,
some incoming connections may silently be dropped, health checks may
randomly fail, and accesses to the CLI may block the whole process. The
blocking delay before emitting this warning may be adjusted via the global
'warn-blocked-traffic-after' directive. Please check the trace below for
any clues about configuration elements that need to be corrected:
* Thread 42: id=0x7fde69f64640 act=1 glob=0 wq=1 rq=0 tl=1 tlsz=2 rqsz=4
1/42 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=1043741907604 now=1044042674560 diff=300766956
curr_task=0x7fdd20f77ef0 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdd20b88580
stream=0x7fdd20b88580: [1754954772.593040] id=5488724 proto=?
flags=0x100c4a, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=0x44207490, pend_pos=(nil) waiting=0 epoch=0
p_stc=2 p_req=2 p_res=2 p_prp=2
frontend=<HTTPCLIENT> (id=4294967295 mode=http), listener=? (id=0)
backend=<HTTPCLIENT> (id=4294967295 mode=http)
server=<HTTPCLIENT> (id=1) addr=185.236.183.85:95
task=0x7fdd20f77ef0 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=41(1/41) age=0s)
txn=0x7fdd20fd1a00 flags=0x83000 meth=1 status=-1 req.st=MSG_DONE rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdd20fd1870 flags=0x00000460 ioto=0s state=EST endp=APPCTX,0x7fdd20f9a0a0,0x10000002 sub=0 rex=<NEVER> wex=<NEVER> rto=0s wto=<NEVER>
app0=0x7fdd20f9a0a0 st0=2 st1=0 applet=<HTTPCLIENT> tid=41 nice=0 calls=0 rate=0
scb=0x7fdd20fd19a0 flags=0x00001c79 ioto=0s state=EST endp=CONN,0x7fdd20f63000,0x40400001 sub=0 rex=0s wex=<NEVER> rto=0s wto=<NEVER>
h1s=0x7fdd20f63000 h1s.flg=0x100090 .req.state=MSG_RQBEFORE .res.state=MSG_RPBEFORE
.meth=UNKNOWN status=0 .sd.flg=0x40400001 .sd.evts= .sc.flg=0x00001c79 .sc.app=0x7fdd20b88580 .sc.evts= .subs=(nil)
h1c=0x7fdd4cbd4ad0 h1c.flg=0x80000000 .sub=1 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdd20f36910 .exp=<NEVER>
co1=0x7fdd4cbdec00 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=SERVER:0x44207490
flags=0x00000300 fd=21050 fd.state=10121 updt=1 fd.tmask=0x20000000000
req=0x7fdd20b885a8 (f=0x1800001 an=0x8000 tofwd=0 total=168)
an_exp=<NEVER> buf=0x7fdd20b885b0 data=0x7fdd21015190 o=168 p=168 i=1048408 size=1048576
htx=0x7fdd21015190 flags=0x10 size=1048528 data=168 used=5 wrap=NO extra=0
res=0x7fdd20b885f0 (f=0x80000001 an=0x1400000 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdd20b885f8 data=0x7fdd20d12e20 o=0 p=0 i=0 size=1048576
htx=0x7fdd20d12e20 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: S:NOTIF U:NOTIF S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ S:IDLE_CONNS U:IDLE_CONNS
call trace(13):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db380 <df ff ff e8 a0 f5 ff ff]: ha_stuck_warning+0xd0/0x188 > ha_thread_dump_one
| 0x6c5f7d <44 00 00 e8 33 53 f1 ff]: wdt_handler+0x1dd/0x1ff > ha_stuck_warning
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee4564092 <83 f9 3f 76 73 48 89 d1]: libc:+0x172092
| 0x5c90c5 <89 4d c8 e8 2b bf e6 ff]: cli_io_handler+0x2da75 > main-0x18b0
| 0x5263ac <48 89 df e8 34 71 0a 00]: process_stream+0x1cfc/0x3ac3 > sc_conn_sync_send
=> Trying to gracefully recover now.
WARNING! thread 42 has stopped processing traffic for 406 milliseconds
with 247 streams currently blocked, prevented from making any progress.
While this may occasionally happen with inefficient configurations
involving excess of regular expressions, map_reg, or heavy Lua processing,
this must remain exceptional because the system's stability is now at risk.
Timers in logs may be reported incorrectly, spurious timeouts may happen,
some incoming connections may silently be dropped, health checks may
randomly fail, and accesses to the CLI may block the whole process. The
blocking delay before emitting this warning may be adjusted via the global
'warn-blocked-traffic-after' directive. Please check the trace below for
any clues about configuration elements that need to be corrected:
* Thread 42: id=0x7fde69f64640 act=1 glob=0 wq=1 rq=0 tl=1 tlsz=2 rqsz=4
1/42 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=1043741907604 now=1044148775601 diff=406867997
curr_task=0x7fdd20f77ef0 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdd20b88580
stream=0x7fdd20b88580: [1754954772.593040] id=5488724 proto=?
flags=0x100c4a, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=0x44207490, pend_pos=(nil) waiting=0 epoch=0
p_stc=2 p_req=2 p_res=2 p_prp=2
frontend=<HTTPCLIENT> (id=4294967295 mode=http), listener=? (id=0)
backend=<HTTPCLIENT> (id=4294967295 mode=http)
server=<HTTPCLIENT> (id=1) addr=185.236.183.85:95
task=0x7fdd20f77ef0 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=41(1/41) age=0s)
txn=0x7fdd20fd1a00 flags=0x83000 meth=1 status=-1 req.st=MSG_DONE rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdd20fd1870 flags=0x00000460 ioto=0s state=EST endp=APPCTX,0x7fdd20f9a0a0,0x10000002 sub=0 rex=<NEVER> wex=<NEVER> rto=0s wto=<NEVER>
app0=0x7fdd20f9a0a0 st0=2 st1=0 applet=<HTTPCLIENT> tid=41 nice=0 calls=0 rate=0
scb=0x7fdd20fd19a0 flags=0x00001c79 ioto=0s state=EST endp=CONN,0x7fdd20f63000,0x40400001 sub=0 rex=0s wex=<NEVER> rto=0s wto=<NEVER>
h1s=0x7fdd20f63000 h1s.flg=0x100090 .req.state=MSG_RQBEFORE .res.state=MSG_RPBEFORE
.meth=UNKNOWN status=0 .sd.flg=0x40400001 .sd.evts= .sc.flg=0x00001c79 .sc.app=0x7fdd20b88580 .sc.evts= .subs=(nil)
h1c=0x7fdd4cbd4ad0 h1c.flg=0x80000000 .sub=1 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdd20f36910 .exp=<NEVER>
co1=0x7fdd4cbdec00 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=SERVER:0x44207490
flags=0x00000300 fd=21050 fd.state=10121 updt=1 fd.tmask=0x20000000000
req=0x7fdd20b885a8 (f=0x1800001 an=0x8000 tofwd=0 total=168)
an_exp=<NEVER> buf=0x7fdd20b885b0 data=0x7fdd21015190 o=168 p=168 i=1048408 size=1048576
htx=0x7fdd21015190 flags=0x10 size=1048528 data=168 used=5 wrap=NO extra=0
res=0x7fdd20b885f0 (f=0x80000001 an=0x1400000 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdd20b885f8 data=0x7fdd20d12e20 o=0 p=0 i=0 size=1048576
htx=0x7fdd20d12e20 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: S:NOTIF U:NOTIF S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ S:IDLE_CONNS U:IDLE_CONNS
call trace(13):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db380 <df ff ff e8 a0 f5 ff ff]: ha_stuck_warning+0xd0/0x188 > ha_thread_dump_one
| 0x6c5f7d <44 00 00 e8 33 53 f1 ff]: wdt_handler+0x1dd/0x1ff > ha_stuck_warning
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee4564092 <83 f9 3f 76 73 48 89 d1]: libc:+0x172092
| 0x5c90c5 <89 4d c8 e8 2b bf e6 ff]: cli_io_handler+0x2da75 > main-0x18b0
| 0x5263ac <48 89 df e8 34 71 0a 00]: process_stream+0x1cfc/0x3ac3 > sc_conn_sync_send
=> Trying to gracefully recover now.
WARNING! thread 42 has stopped processing traffic for 524 milliseconds
with 247 streams currently blocked, prevented from making any progress.
While this may occasionally happen with inefficient configurations
involving excess of regular expressions, map_reg, or heavy Lua processing,
this must remain exceptional because the system's stability is now at risk.
Timers in logs may be reported incorrectly, spurious timeouts may happen,
some incoming connections may silently be dropped, health checks may
randomly fail, and accesses to the CLI may block the whole process. The
blocking delay before emitting this warning may be adjusted via the global
'warn-blocked-traffic-after' directive. Please check the trace below for
any clues about configuration elements that need to be corrected:
* Thread 42: id=0x7fde69f64640 act=1 glob=0 wq=1 rq=0 tl=1 tlsz=2 rqsz=4
1/42 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=1043741907604 now=1044266003805 diff=524096201
curr_task=0x7fdd20f77ef0 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdd20b88580
stream=0x7fdd20b88580: [1754954772.593040] id=5488724 proto=?
flags=0x100c4a, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=0x44207490, pend_pos=(nil) waiting=0 epoch=0
p_stc=2 p_req=2 p_res=2 p_prp=2
frontend=<HTTPCLIENT> (id=4294967295 mode=http), listener=? (id=0)
backend=<HTTPCLIENT> (id=4294967295 mode=http)
server=<HTTPCLIENT> (id=1) addr=185.236.183.85:95
task=0x7fdd20f77ef0 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=41(1/41) age=0s)
txn=0x7fdd20fd1a00 flags=0x83000 meth=1 status=-1 req.st=MSG_DONE rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdd20fd1870 flags=0x00000460 ioto=0s state=EST endp=APPCTX,0x7fdd20f9a0a0,0x10000002 sub=0 rex=<NEVER> wex=<NEVER> rto=0s wto=<NEVER>
app0=0x7fdd20f9a0a0 st0=2 st1=0 applet=<HTTPCLIENT> tid=41 nice=0 calls=0 rate=0
scb=0x7fdd20fd19a0 flags=0x00001c79 ioto=0s state=EST endp=CONN,0x7fdd20f63000,0x40400001 sub=0 rex=0s wex=<NEVER> rto=0s wto=<NEVER>
h1s=0x7fdd20f63000 h1s.flg=0x100090 .req.state=MSG_RQBEFORE .res.state=MSG_RPBEFORE
.meth=UNKNOWN status=0 .sd.flg=0x40400001 .sd.evts= .sc.flg=0x00001c79 .sc.app=0x7fdd20b88580 .sc.evts= .subs=(nil)
h1c=0x7fdd4cbd4ad0 h1c.flg=0x80000000 .sub=1 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdd20f36910 .exp=<NEVER>
co1=0x7fdd4cbdec00 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=SERVER:0x44207490
flags=0x00000300 fd=21050 fd.state=10121 updt=1 fd.tmask=0x20000000000
req=0x7fdd20b885a8 (f=0x1800001 an=0x8000 tofwd=0 total=168)
an_exp=<NEVER> buf=0x7fdd20b885b0 data=0x7fdd21015190 o=168 p=168 i=1048408 size=1048576
htx=0x7fdd21015190 flags=0x10 size=1048528 data=168 used=5 wrap=NO extra=0
res=0x7fdd20b885f0 (f=0x80000001 an=0x1400000 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdd20b885f8 data=0x7fdd20d12e20 o=0 p=0 i=0 size=1048576
htx=0x7fdd20d12e20 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: S:NOTIF U:NOTIF S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ S:IDLE_CONNS U:IDLE_CONNS
call trace(13):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db380 <df ff ff e8 a0 f5 ff ff]: ha_stuck_warning+0xd0/0x188 > ha_thread_dump_one
| 0x6c5f7d <44 00 00 e8 33 53 f1 ff]: wdt_handler+0x1dd/0x1ff > ha_stuck_warning
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee4564092 <83 f9 3f 76 73 48 89 d1]: libc:+0x172092
| 0x5c90c5 <89 4d c8 e8 2b bf e6 ff]: cli_io_handler+0x2da75 > main-0x18b0
| 0x5263ac <48 89 df e8 34 71 0a 00]: process_stream+0x1cfc/0x3ac3 > sc_conn_sync_send
=> Trying to gracefully recover now.
WARNING! thread 42 has stopped processing traffic for 671 milliseconds
with 247 streams currently blocked, prevented from making any progress.
While this may occasionally happen with inefficient configurations
involving excess of regular expressions, map_reg, or heavy Lua processing,
this must remain exceptional because the system's stability is now at risk.
Timers in logs may be reported incorrectly, spurious timeouts may happen,
some incoming connections may silently be dropped, health checks may
randomly fail, and accesses to the CLI may block the whole process. The
blocking delay before emitting this warning may be adjusted via the global
'warn-blocked-traffic-after' directive. Please check the trace below for
any clues about configuration elements that need to be corrected:
* Thread 42: id=0x7fde69f64640 act=1 glob=0 wq=1 rq=0 tl=1 tlsz=2 rqsz=4
1/42 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=1043741907604 now=1044412944852 diff=671037248
curr_task=0x7fdd20f77ef0 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdd20b88580
stream=0x7fdd20b88580: [1754954772.593040] id=5488724 proto=?
flags=0x100c4a, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=0x44207490, pend_pos=(nil) waiting=0 epoch=0
p_stc=2 p_req=2 p_res=2 p_prp=2
frontend=<HTTPCLIENT> (id=4294967295 mode=http), listener=? (id=0)
backend=<HTTPCLIENT> (id=4294967295 mode=http)
server=<HTTPCLIENT> (id=1) addr=185.236.183.85:95
task=0x7fdd20f77ef0 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=41(1/41) age=0s)
txn=0x7fdd20fd1a00 flags=0x83000 meth=1 status=-1 req.st=MSG_DONE rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdd20fd1870 flags=0x00000460 ioto=0s state=EST endp=APPCTX,0x7fdd20f9a0a0,0x10000002 sub=0 rex=<NEVER> wex=<NEVER> rto=0s wto=<NEVER>
app0=0x7fdd20f9a0a0 st0=2 st1=0 applet=<HTTPCLIENT> tid=41 nice=0 calls=0 rate=0
scb=0x7fdd20fd19a0 flags=0x00001c79 ioto=0s state=EST endp=CONN,0x7fdd20f63000,0x40400001 sub=0 rex=0s wex=<NEVER> rto=0s wto=<NEVER>
h1s=0x7fdd20f63000 h1s.flg=0x100090 .req.state=MSG_RQBEFORE .res.state=MSG_RPBEFORE
.meth=UNKNOWN status=0 .sd.flg=0x40400001 .sd.evts= .sc.flg=0x00001c79 .sc.app=0x7fdd20b88580 .sc.evts= .subs=(nil)
h1c=0x7fdd4cbd4ad0 h1c.flg=0x80000000 .sub=1 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdd20f36910 .exp=<NEVER>
co1=0x7fdd4cbdec00 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=SERVER:0x44207490
flags=0x00000300 fd=21050 fd.state=10121 updt=1 fd.tmask=0x20000000000
req=0x7fdd20b885a8 (f=0x1800001 an=0x8000 tofwd=0 total=168)
an_exp=<NEVER> buf=0x7fdd20b885b0 data=0x7fdd21015190 o=168 p=168 i=1048408 size=1048576
htx=0x7fdd21015190 flags=0x10 size=1048528 data=168 used=5 wrap=NO extra=0
res=0x7fdd20b885f0 (f=0x80000001 an=0x1400000 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdd20b885f8 data=0x7fdd20d12e20 o=0 p=0 i=0 size=1048576
htx=0x7fdd20d12e20 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: S:NOTIF U:NOTIF S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ S:IDLE_CONNS U:IDLE_CONNS
call trace(13):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db380 <df ff ff e8 a0 f5 ff ff]: ha_stuck_warning+0xd0/0x188 > ha_thread_dump_one
| 0x6c5f7d <44 00 00 e8 33 53 f1 ff]: wdt_handler+0x1dd/0x1ff > ha_stuck_warning
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee4564092 <83 f9 3f 76 73 48 89 d1]: libc:+0x172092
| 0x5c90c5 <89 4d c8 e8 2b bf e6 ff]: cli_io_handler+0x2da75 > main-0x18b0
| 0x5263ac <48 89 df e8 34 71 0a 00]: process_stream+0x1cfc/0x3ac3 > sc_conn_sync_send
=> Trying to gracefully recover now.
WARNING! thread 39 has stopped processing traffic for 293 milliseconds
with 272 streams currently blocked, prevented from making any progress.
While this may occasionally happen with inefficient configurations
involving excess of regular expressions, map_reg, or heavy Lua processing,
this must remain exceptional because the system's stability is now at risk.
Timers in logs may be reported incorrectly, spurious timeouts may happen,
some incoming connections may silently be dropped, health checks may
randomly fail, and accesses to the CLI may block the whole process. The
blocking delay before emitting this warning may be adjusted via the global
'warn-blocked-traffic-after' directive. Please check the trace below for
any clues about configuration elements that need to be corrected:
* Thread 39: id=0x7fde6bfff640 act=1 glob=0 wq=1 rq=0 tl=1 tlsz=1 rqsz=2
1/39 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=1132899234649 now=1133192335804 diff=293101155
curr_task=0x7fdd6ce41210 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdd6ce40de0
stream=0x7fdd6ce40de0: [1754954773.169873] id=5489158 proto=?
flags=0x100c4a, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=0x44207490, pend_pos=(nil) waiting=0 epoch=0
p_stc=2 p_req=2 p_res=2 p_prp=2
frontend=<HTTPCLIENT> (id=4294967295 mode=http), listener=? (id=0)
backend=<HTTPCLIENT> (id=4294967295 mode=http)
server=<HTTPCLIENT> (id=1) addr=179.96.0.91:95
task=0x7fdd6ce41210 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=38(1/38) age=0s)
txn=0x7fdd6cb87850 flags=0x83000 meth=1 status=-1 req.st=MSG_DONE rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdd6cb877f0 flags=0x00000460 ioto=0s state=EST endp=APPCTX,0x7fdd6cc15fd0,0x10000002 sub=0 rex=<NEVER> wex=<NEVER> rto=0s wto=<NEVER>
app0=0x7fdd6cc15fd0 st0=2 st1=0 applet=<HTTPCLIENT> tid=38 nice=0 calls=0 rate=0
scb=0x7fdd6ce412a0 flags=0x00001c79 ioto=0s state=EST endp=CONN,0x7fdd6cd8a960,0x40400001 sub=0 rex=0s wex=<NEVER> rto=0s wto=<NEVER>
h1s=0x7fdd6cd8a960 h1s.flg=0x100090 .req.state=MSG_RQBEFORE .res.state=MSG_RPBEFORE
.meth=UNKNOWN status=0 .sd.flg=0x40400001 .sd.evts= .sc.flg=0x00001c79 .sc.app=0x7fdd6ce40de0 .sc.evts= .subs=(nil)
h1c=0x7fde28e9f730 h1c.flg=0x80000000 .sub=1 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdd6ce41380 .exp=<NEVER>
co1=0x7fde292dd430 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=SERVER:0x44207490
flags=0x00000300 fd=52449 fd.state=10121 updt=1 fd.tmask=0x4000000000
req=0x7fdd6ce40e08 (f=0x1800001 an=0x8000 tofwd=0 total=158)
an_exp=<NEVER> buf=0x7fdd6ce40e10 data=0x7fdd6d313320 o=158 p=158 i=1048418 size=1048576
htx=0x7fdd6d313320 flags=0x10 size=1048528 data=158 used=5 wrap=NO extra=0
res=0x7fdd6ce40e50 (f=0x80000001 an=0x1400000 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdd6ce40e58 data=0x7fdd6d213310 o=0 p=0 i=0 size=1048576
htx=0x7fdd6d213310 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: S:NOTIF U:NOTIF R:TASK_WQ U:TASK_WQ S:NOTIF U:NOTIF S:IDLE_CONNS U:IDLE_CONNS
call trace(13):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db380 <df ff ff e8 a0 f5 ff ff]: ha_stuck_warning+0xd0/0x188 > ha_thread_dump_one
| 0x6c5f7d <44 00 00 e8 33 53 f1 ff]: wdt_handler+0x1dd/0x1ff > ha_stuck_warning
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee4564092 <83 f9 3f 76 73 48 89 d1]: libc:+0x172092
| 0x5c90c5 <89 4d c8 e8 2b bf e6 ff]: cli_io_handler+0x2da75 > main-0x18b0
| 0x5263ac <48 89 df e8 34 71 0a 00]: process_stream+0x1cfc/0x3ac3 > sc_conn_sync_send
=> Trying to gracefully recover now.
WARNING! thread 42 has stopped processing traffic for 859 milliseconds
with 247 streams currently blocked, prevented from making any progress.
While this may occasionally happen with inefficient configurations
involving excess of regular expressions, map_reg, or heavy Lua processing,
this must remain exceptional because the system's stability is now at risk.
Timers in logs may be reported incorrectly, spurious timeouts may happen,
some incoming connections may silently be dropped, health checks may
randomly fail, and accesses to the CLI may block the whole process. The
blocking delay before emitting this warning may be adjusted via the global
'warn-blocked-traffic-after' directive. Please check the trace below for
any clues about configuration elements that need to be corrected:
* Thread 42: id=0x7fde69f64640 act=1 glob=0 wq=1 rq=0 tl=1 tlsz=2 rqsz=4
1/42 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=1043741907604 now=1044601002229 diff=859094625
curr_task=0x7fdd20f77ef0 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdd20b88580
stream=0x7fdd20b88580: [1754954772.593040] id=5488724 proto=?
flags=0x100c4a, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=0x44207490, pend_pos=(nil) waiting=0 epoch=0
p_stc=2 p_req=2 p_res=2 p_prp=2
frontend=<HTTPCLIENT> (id=4294967295 mode=http), listener=? (id=0)
backend=<HTTPCLIENT> (id=4294967295 mode=http)
server=<HTTPCLIENT> (id=1) addr=185.236.183.85:95
task=0x7fdd20f77ef0 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=41(1/41) age=0s)
txn=0x7fdd20fd1a00 flags=0x83000 meth=1 status=-1 req.st=MSG_DONE rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdd20fd1870 flags=0x00000460 ioto=0s state=EST endp=APPCTX,0x7fdd20f9a0a0,0x10000002 sub=0 rex=<NEVER> wex=<NEVER> rto=0s wto=<NEVER>
app0=0x7fdd20f9a0a0 st0=2 st1=0 applet=<HTTPCLIENT> tid=41 nice=0 calls=0 rate=0
scb=0x7fdd20fd19a0 flags=0x00001c79 ioto=0s state=EST endp=CONN,0x7fdd20f63000,0x40400001 sub=0 rex=0s wex=<NEVER> rto=0s wto=<NEVER>
h1s=0x7fdd20f63000 h1s.flg=0x100090 .req.state=MSG_RQBEFORE .res.state=MSG_RPBEFORE
.meth=UNKNOWN status=0 .sd.flg=0x40400001 .sd.evts= .sc.flg=0x00001c79 .sc.app=0x7fdd20b88580 .sc.evts= .subs=(nil)
h1c=0x7fdd4cbd4ad0 h1c.flg=0x80000000 .sub=1 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdd20f36910 .exp=<NEVER>
co1=0x7fdd4cbdec00 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=SERVER:0x44207490
flags=0x00000300 fd=21050 fd.state=10121 updt=1 fd.tmask=0x20000000000
req=0x7fdd20b885a8 (f=0x1800001 an=0x8000 tofwd=0 total=168)
an_exp=<NEVER> buf=0x7fdd20b885b0 data=0x7fdd21015190 o=168 p=168 i=1048408 size=1048576
htx=0x7fdd21015190 flags=0x10 size=1048528 data=168 used=5 wrap=NO extra=0
res=0x7fdd20b885f0 (f=0x80000001 an=0x1400000 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdd20b885f8 data=0x7fdd20d12e20 o=0 p=0 i=0 size=1048576
htx=0x7fdd20d12e20 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: S:NOTIF U:NOTIF S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ S:IDLE_CONNS U:IDLE_CONNS
call trace(13):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db380 <df ff ff e8 a0 f5 ff ff]: ha_stuck_warning+0xd0/0x188 > ha_thread_dump_one
| 0x6c5f7d <44 00 00 e8 33 53 f1 ff]: wdt_handler+0x1dd/0x1ff > ha_stuck_warning
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee4564092 <83 f9 3f 76 73 48 89 d1]: libc:+0x172092
| 0x5c90c5 <89 4d c8 e8 2b bf e6 ff]: cli_io_handler+0x2da75 > main-0x18b0
| 0x5263ac <48 89 df e8 34 71 0a 00]: process_stream+0x1cfc/0x3ac3 > sc_conn_sync_send
=> Trying to gracefully recover now.
WARNING! thread 48 has stopped processing traffic for 286 milliseconds
with 277 streams currently blocked, prevented from making any progress.
While this may occasionally happen with inefficient configurations
involving excess of regular expressions, map_reg, or heavy Lua processing,
this must remain exceptional because the system's stability is now at risk.
Timers in logs may be reported incorrectly, spurious timeouts may happen,
some incoming connections may silently be dropped, health checks may
randomly fail, and accesses to the CLI may block the whole process. The
blocking delay before emitting this warning may be adjusted via the global
'warn-blocked-traffic-after' directive. Please check the trace below for
any clues about configuration elements that need to be corrected:
* Thread 48: id=0x7fde4bfff640 act=1 glob=0 wq=1 rq=0 tl=1 tlsz=1 rqsz=2
1/48 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=1056036426858 now=1056322962936 diff=286536078
curr_task=0x7fddd8e7d2e0 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fddd8bc11b0
stream=0x7fddd8bc11b0: [1754954773.378485] id=5489309 proto=?
flags=0x100c4a, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=0x44207490, pend_pos=(nil) waiting=0 epoch=0
p_stc=2 p_req=2 p_res=2 p_prp=2
frontend=<HTTPCLIENT> (id=4294967295 mode=http), listener=? (id=0)
backend=<HTTPCLIENT> (id=4294967295 mode=http)
server=<HTTPCLIENT> (id=1) addr=179.96.0.91:95
task=0x7fddd8e7d2e0 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=47(1/47) age=0s)
txn=0x7fddd8b87a00 flags=0x83000 meth=1 status=-1 req.st=MSG_DONE rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fddd8e7b670 flags=0x00000460 ioto=0s state=EST endp=APPCTX,0x451becc0,0x10000002 sub=0 rex=<NEVER> wex=<NEVER> rto=0s wto=<NEVER>
app0=0x451becc0 st0=2 st1=0 applet=<HTTPCLIENT> tid=47 nice=0 calls=0 rate=0
scb=0x7fddd8e7b730 flags=0x00001c79 ioto=0s state=EST endp=CONN,0x7fddd8db0340,0x40400001 sub=0 rex=0s wex=<NEVER> rto=0s wto=<NEVER>
h1s=0x7fddd8db0340 h1s.flg=0x100090 .req.state=MSG_RQBEFORE .res.state=MSG_RPBEFORE
.meth=UNKNOWN status=0 .sd.flg=0x40400001 .sd.evts= .sc.flg=0x00001c79 .sc.app=0x7fddd8bc11b0 .sc.evts= .subs=(nil)
h1c=0x7fde1907c0d0 h1c.flg=0x80000000 .sub=1 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fddd8b87af0 .exp=<NEVER>
co1=0x7fde18d69c00 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=SERVER:0x44207490
flags=0x00000300 fd=80230 fd.state=10121 updt=1 fd.tmask=0x800000000000
req=0x7fddd8bc11d8 (f=0x1800001 an=0x8000 tofwd=0 total=160)
an_exp=<NEVER> buf=0x7fddd8bc11e0 data=0x7fddd8f3a200 o=160 p=160 i=1048416 size=1048576
htx=0x7fddd8f3a200 flags=0x10 size=1048528 data=160 used=5 wrap=NO extra=0
res=0x7fddd8bc1220 (f=0x80000001 an=0x1400000 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fddd8bc1228 data=0x7fddd8c12e10 o=0 p=0 i=0 size=1048576
htx=0x7fddd8c12e10 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: S:NOTIF U:NOTIF S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ S:IDLE_CONNS U:IDLE_CONNS
call trace(13):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db380 <df ff ff e8 a0 f5 ff ff]: ha_stuck_warning+0xd0/0x188 > ha_thread_dump_one
| 0x6c5f7d <44 00 00 e8 33 53 f1 ff]: wdt_handler+0x1dd/0x1ff > ha_stuck_warning
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee4564092 <83 f9 3f 76 73 48 89 d1]: libc:+0x172092
| 0x5c90c5 <89 4d c8 e8 2b bf e6 ff]: cli_io_handler+0x2da75 > main-0x18b0
| 0x5263ac <48 89 df e8 34 71 0a 00]: process_stream+0x1cfc/0x3ac3 > sc_conn_sync_send
=> Trying to gracefully recover now.
WARNING! thread 42 has stopped processing traffic for 973 milliseconds
with 247 streams currently blocked, prevented from making any progress.
While this may occasionally happen with inefficient configurations
involving excess of regular expressions, map_reg, or heavy Lua processing,
this must remain exceptional because the system's stability is now at risk.
Timers in logs may be reported incorrectly, spurious timeouts may happen,
some incoming connections may silently be dropped, health checks may
randomly fail, and accesses to the CLI may block the whole process. The
blocking delay before emitting this warning may be adjusted via the global
'warn-blocked-traffic-after' directive. Please check the trace below for
any clues about configuration elements that need to be corrected:
* Thread 42: id=0x7fde69f64640 act=1 glob=0 wq=1 rq=0 tl=1 tlsz=2 rqsz=4
1/42 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=1043741907604 now=1044714958008 diff=973050404
curr_task=0x7fdd20f77ef0 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdd20b88580
stream=0x7fdd20b88580: [1754954772.593040] id=5488724 proto=?
flags=0x100c4a, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=0x44207490, pend_pos=(nil) waiting=0 epoch=0
p_stc=2 p_req=2 p_res=2 p_prp=2
frontend=<HTTPCLIENT> (id=4294967295 mode=http), listener=? (id=0)
backend=<HTTPCLIENT> (id=4294967295 mode=http)
server=<HTTPCLIENT> (id=1) addr=185.236.183.85:95
task=0x7fdd20f77ef0 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=41(1/41) age=0s)
txn=0x7fdd20fd1a00 flags=0x83000 meth=1 status=-1 req.st=MSG_DONE rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdd20fd1870 flags=0x00000460 ioto=0s state=EST endp=APPCTX,0x7fdd20f9a0a0,0x10000002 sub=0 rex=<NEVER> wex=<NEVER> rto=0s wto=<NEVER>
app0=0x7fdd20f9a0a0 st0=2 st1=0 applet=<HTTPCLIENT> tid=41 nice=0 calls=0 rate=0
scb=0x7fdd20fd19a0 flags=0x00001c79 ioto=0s state=EST endp=CONN,0x7fdd20f63000,0x40400001 sub=0 rex=0s wex=<NEVER> rto=0s wto=<NEVER>
h1s=0x7fdd20f63000 h1s.flg=0x100090 .req.state=MSG_RQBEFORE .res.state=MSG_RPBEFORE
.meth=UNKNOWN status=0 .sd.flg=0x40400001 .sd.evts= .sc.flg=0x00001c79 .sc.app=0x7fdd20b88580 .sc.evts= .subs=(nil)
h1c=0x7fdd4cbd4ad0 h1c.flg=0x80000000 .sub=1 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdd20f36910 .exp=<NEVER>
co1=0x7fdd4cbdec00 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=SERVER:0x44207490
flags=0x00000300 fd=21050 fd.state=10121 updt=1 fd.tmask=0x20000000000
req=0x7fdd20b885a8 (f=0x1800001 an=0x8000 tofwd=0 total=168)
an_exp=<NEVER> buf=0x7fdd20b885b0 data=0x7fdd21015190 o=168 p=168 i=1048408 size=1048576
htx=0x7fdd21015190 flags=0x10 size=1048528 data=168 used=5 wrap=NO extra=0
res=0x7fdd20b885f0 (f=0x80000001 an=0x1400000 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdd20b885f8 data=0x7fdd20d12e20 o=0 p=0 i=0 size=1048576
htx=0x7fdd20d12e20 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: S:NOTIF U:NOTIF S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ S:IDLE_CONNS U:IDLE_CONNS
call trace(13):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db380 <df ff ff e8 a0 f5 ff ff]: ha_stuck_warning+0xd0/0x188 > ha_thread_dump_one
| 0x6c5f7d <44 00 00 e8 33 53 f1 ff]: wdt_handler+0x1dd/0x1ff > ha_stuck_warning
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee4564092 <83 f9 3f 76 73 48 89 d1]: libc:+0x172092
| 0x5c90c5 <89 4d c8 e8 2b bf e6 ff]: cli_io_handler+0x2da75 > main-0x18b0
| 0x5263ac <48 89 df e8 34 71 0a 00]: process_stream+0x1cfc/0x3ac3 > sc_conn_sync_send
=> Trying to gracefully recover now.
WARNING! thread 48 has stopped processing traffic for 483 milliseconds
with 277 streams currently blocked, prevented from making any progress.
While this may occasionally happen with inefficient configurations
involving excess of regular expressions, map_reg, or heavy Lua processing,
this must remain exceptional because the system's stability is now at risk.
Timers in logs may be reported incorrectly, spurious timeouts may happen,
some incoming connections may silently be dropped, health checks may
randomly fail, and accesses to the CLI may block the whole process. The
blocking delay before emitting this warning may be adjusted via the global
'warn-blocked-traffic-after' directive. Please check the trace below for
any clues about configuration elements that need to be corrected:
* Thread 48: id=0x7fde4bfff640 act=1 glob=0 wq=1 rq=0 tl=1 tlsz=1 rqsz=2
1/48 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=1056036426858 now=1056519828571 diff=483401713
curr_task=0x7fddd8e7d2e0 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fddd8bc11b0
stream=0x7fddd8bc11b0: [1754954773.378485] id=5489309 proto=?
flags=0x100c4a, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=0x44207490, pend_pos=(nil) waiting=0 epoch=0
p_stc=2 p_req=2 p_res=2 p_prp=2
frontend=<HTTPCLIENT> (id=4294967295 mode=http), listener=? (id=0)
backend=<HTTPCLIENT> (id=4294967295 mode=http)
server=<HTTPCLIENT> (id=1) addr=179.96.0.91:95
task=0x7fddd8e7d2e0 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=47(1/47) age=0s)
txn=0x7fddd8b87a00 flags=0x83000 meth=1 status=-1 req.st=MSG_DONE rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fddd8e7b670 flags=0x00000460 ioto=0s state=EST endp=APPCTX,0x451becc0,0x10000002 sub=0 rex=<NEVER> wex=<NEVER> rto=0s wto=<NEVER>
app0=0x451becc0 st0=2 st1=0 applet=<HTTPCLIENT> tid=47 nice=0 calls=0 rate=0
scb=0x7fddd8e7b730 flags=0x00001c79 ioto=0s state=EST endp=CONN,0x7fddd8db0340,0x40400001 sub=0 rex=0s wex=<NEVER> rto=0s wto=<NEVER>
h1s=0x7fddd8db0340 h1s.flg=0x100090 .req.state=MSG_RQBEFORE .res.state=MSG_RPBEFORE
.meth=UNKNOWN status=0 .sd.flg=0x40400001 .sd.evts= .sc.flg=0x00001c79 .sc.app=0x7fddd8bc11b0 .sc.evts= .subs=(nil)
h1c=0x7fde1907c0d0 h1c.flg=0x80000000 .sub=1 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fddd8b87af0 .exp=<NEVER>
co1=0x7fde18d69c00 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=SERVER:0x44207490
flags=0x00000300 fd=80230 fd.state=10121 updt=1 fd.tmask=0x800000000000
req=0x7fddd8bc11d8 (f=0x1800001 an=0x8000 tofwd=0 total=160)
an_exp=<NEVER> buf=0x7fddd8bc11e0 data=0x7fddd8f3a200 o=160 p=160 i=1048416 size=1048576
htx=0x7fddd8f3a200 flags=0x10 size=1048528 data=160 used=5 wrap=NO extra=0
res=0x7fddd8bc1220 (f=0x80000001 an=0x1400000 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fddd8bc1228 data=0x7fddd8c12e10 o=0 p=0 i=0 size=1048576
htx=0x7fddd8c12e10 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: S:NOTIF U:NOTIF S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ S:IDLE_CONNS U:IDLE_CONNS
call trace(13):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db380 <df ff ff e8 a0 f5 ff ff]: ha_stuck_warning+0xd0/0x188 > ha_thread_dump_one
| 0x6c5f7d <44 00 00 e8 33 53 f1 ff]: wdt_handler+0x1dd/0x1ff > ha_stuck_warning
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee4564092 <83 f9 3f 76 73 48 89 d1]: libc:+0x172092
| 0x5c90c5 <89 4d c8 e8 2b bf e6 ff]: cli_io_handler+0x2da75 > main-0x18b0
| 0x5263ac <48 89 df e8 34 71 0a 00]: process_stream+0x1cfc/0x3ac3 > sc_conn_sync_send
=> Trying to gracefully recover now.
Thread 42 is about to kill the process.
Thread 1 : id=0x7fdee432b540 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/1 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1042119907919 now=1042119984354 diff=76435
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 2 : id=0x7fdee4307640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/2 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1003304326719 now=1003304338506 diff=11787
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 3 : id=0x7fdecf1cb640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/3 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1143979846112 now=1143979907839 diff=61727
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 4 : id=0x7fdece132640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/4 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1022618572091 now=1022618647785 diff=75694
curr_task=0
lock_hist: S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ
Thread 5 : id=0x7fdecd931640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/5 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1150696859004 now=1150696875276 diff=16272
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 6 : id=0x7fdecd130640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/6 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1018485278481 now=1018485464798 diff=186317
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 7 : id=0x7fdecc92f640 act=1 glob=0 wq=1 rq=0 tl=1 tlsz=1 rqsz=2
1/7 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=1008458832758 now=1008559592840 diff=100760082
curr_task=0x7fdec0b204a0 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdec0d52450
stream=0x7fdec0d52450: [1754954773.992240] id=5489712 proto=tcpv4 source=170.254.8.15:26735
flags=0xc0a, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=0x440f2c00, pend_pos=(nil) waiting=0 epoch=0
p_stc=1 p_req=1 p_res=1 p_prp=1
frontend=ssl (id=3 mode=http), listener=? (id=838) addr=179.96.0.90:80
backend=hyper_vod_tvs (id=15 mode=http)
server=tvs_hyper (id=1) addr=unix
task=0x7fdec0b204a0 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=6(1/6) age=0s)
txn=0x7fdec0b20600 flags=0x3000 meth=1 status=-1 req.st=MSG_DONE rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdec0ba7b20 flags=0x00000422 ioto=1m40s state=EST endp=CONN,0x7fdec0b20530,0x40404001 sub=1 rex=<NEVER> wex=<NEVER> rto=1m40s wto=<NEVER>
h1s=0x7fdec0b20530 h1s.flg=0x100010 .req.state=MSG_DONE .res.state=MSG_RPBEFORE
.meth=GET status=0 .sd.flg=0x40404001 .sd.evts= .sc.flg=0x00000422 .sc.app=0x7fdec0d52450 .sc.evts=
.subs=0x7fdec0ba7b38(ev=1 tl=0x7fdec13570d0 tl.calls=0 tl.ctx=0x7fdec0ba7b20 tl.fct=sc_conn_io_cb+0)
h1c=0x7fdec0b20340 h1c.flg=0x0 .sub=0 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdec0b20410 .exp=<NEVER>
co0=0x7fdec0b20180 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=LISTENER:0x442f4510
flags=0x80000300 fd=31558 fd.state=120 updt=0 fd.tmask=0x40
scb=0x7fdec0d52910 flags=0x00001831 ioto=0s state=ASS endp=CONN,(nil),0x00000001 sub=0 rex=<NEVER> wex=<NEVER> rto=<NEVER> wto=<NEVER>
co1=0x7fdec0eb9b20 ctrl=NONE xprt=RAW mux=NONE data=STRM target=SERVER:0x440f2c00
flags=0x00000000 fd=-1 fd.state=00 updt=0 fd.tmask=0x0
req=0x7fdec0d52478 (f=0x1800001 an=0x8000 tofwd=0 total=391)
an_exp=<NEVER> buf=0x7fdec0d52480 data=0x7fdec0c175b0 o=584 p=584 i=1047992 size=1048576
htx=0x7fdec0c175b0 flags=0x14 size=1048528 data=584 used=14 wrap=NO extra=0
res=0x7fdec0d524c0 (f=0x80000000 an=0x0 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdec0d524c8 data=0x7fdec13820d0 o=0 p=0 i=0 size=1048576
htx=0x7fdec13820d0 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: R:LISTENER U:LISTENER R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM
call trace(15):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db245 <2a 31 f6 e8 db f6 ff ff]: debug_handler+0x65/0xca > ha_thread_dump_one
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee44fecd7 <00 b8 29 00 00 00 0f 05]: libc:socket+0x7/0x21
| 0x6567d8 <8b 70 10 e8 68 1c e5 ff]: sock_create_server_socket+0x88/0x464 > my_socketat
| 0x6acd32 <48 89 df e8 1e 9a fa ff]: mworker_accept_wrapper+0x2d282 > sock_create_server_socket
| 0x5a2bad <08 00 00 4c 89 f7 ff d2]: connect_server+0x39d/0x1c3e
| 0x5a5118 <4c 89 e7 e8 f8 d6 ff ff]: back_try_conn_req+0x198/0x6a2 > connect_server
| 0x525e90 <4c 89 ff e8 f0 f0 07 00]: process_stream+0x17e0/0x3ac3 > back_try_conn_req
Thread 8 : id=0x7fdec7767640 act=1 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/8 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=1088946087916 now=1089006006610 diff=59918694
curr_task=0x7fde9d062a40 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fde9d060ed0
stream=0x7fde9d060ed0: [1754954774.091083] id=5489767 proto=tcpv4 source=177.8.40.33:51309
flags=0xc08, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=(nil), pend_pos=(nil) waiting=0 epoch=0
p_stc=1 p_req=1 p_res=0 p_prp=0
frontend=ssl (id=3 mode=http), listener=? (id=839) addr=179.96.0.90:80
backend=hyper_vod_tvs (id=15 mode=http)
server=<none> (id=0)
task=0x7fde9d062a40 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=7(1/7) age=0s)
txn=0x7fde9d061310 flags=0x3000 meth=1 status=-1 req.st=MSG_BODY rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fde9cbdaeb0 flags=0x00000422 ioto=1m40s state=EST endp=CONN,0x7fde9cb93320,0x40404001 sub=1 rex=<NEVER> wex=<NEVER> rto=1m40s wto=<NEVER>
h1s=0x7fde9cb93320 h1s.flg=0x100010 .req.state=MSG_DONE .res.state=MSG_RPBEFORE
.meth=GET status=0 .sd.flg=0x40404001 .sd.evts= .sc.flg=0x00000422 .sc.app=0x7fde9d060ed0 .sc.evts=
.subs=0x7fde9cbdaec8(ev=1 tl=0x7fde9d0612c0 tl.calls=0 tl.ctx=0x7fde9cbdaeb0 tl.fct=sc_conn_io_cb+0)
h1c=0x7fde9d0628e0 h1c.flg=0x0 .sub=0 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fde9d0629b0 .exp=<NEVER>
co0=0x7fde9d062720 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=LISTENER:0x442f47f0
flags=0x80000300 fd=37425 fd.state=20 updt=0 fd.tmask=0x80
scb=0x7fde9d29f940 flags=0x00000821 ioto=0s state=INI endp=NONE,(nil),0x00000010 sub=0 rex=<NEVER> wex=<NEVER> rto=<NEVER> wto=<NEVER>
req=0x7fde9d060ef8 (f=0x808001 an=0x900 tofwd=0 total=642)
an_exp=<NEVER> buf=0x7fde9d060f00 data=0x7fde9d08d6a0 o=0 p=0 i=1048576 size=1048576
htx=0x7fde9d08d6a0 flags=0x14 size=1048528 data=805 used=14 wrap=NO extra=0
res=0x7fde9d060f40 (f=0x80000000 an=0x0 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fde9d060f48 data=0x7fde9cd2bca0 o=0 p=0 i=0 size=1048576
htx=0x7fde9cd2bca0 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: R:TASK_WQ U:TASK_WQ R:LISTENER U:LISTENER R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP
call trace(22):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db245 <2a 31 f6 e8 db f6 ff ff]: debug_handler+0x65/0xca > ha_thread_dump_one
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee447d947 <48 83 c9 01 48 89 4a 08]: libc:+0x8b947
| 0x7fdee447ee07 <7c 24 30 e8 b9 e9 ff ff]: libc:+0x8ce07 > libc:+0x8b7c0
| 0x7fdee4480871 <4c 89 e7 e8 0f d8 ff ff]: libc:__libc_malloc+0x1c1/0x327 > libc:+0x8c080
| 0x64517a <89 c0 c3 e8 e6 07 df ff]: pool_get_from_os_noinc+0xca/0xf0 > main-0xf40
| 0x645232 <48 89 fb e8 7e fe ff ff]: pool_alloc_nocache+0x12/0xaa > pool_get_from_os_noinc
| 0x6460df <4c 89 f7 e8 41 f1 ff ff]: __pool_alloc+0x16f/0x279 > pool_alloc_nocache
| 0x57a6f4 <89 04 24 e8 7c b8 0c 00]: main+0x143e54 > __pool_alloc
| 0x57961f <7b 18 4c 89 ee ff 51 08]: sample_process_cnv+0x6f/0x90
| 0x57969c <4c 89 ef e8 14 ff ff ff]: sample_process+0x5c/0xa7 > sample_process_cnv
| 0x691822 <4c 89 f7 e8 1e 7e ee ff]: acl_exec_cond+0x132/0x249 > sample_process
| 0x53c98d <89 7d c8 e8 63 4d 15 00]: process_stream+0x182dd > acl_exec_cond
| 0x53f696 <89 4d 98 e8 fa d1 ff ff]: http_process_req_common+0xb6/0x1590 > process_stream+0x181e0
| 0x5259cf <4c 89 ff e8 11 9c 01 00]: process_stream+0x131f/0x3ac3 > http_process_req_common
Thread 9 : id=0x7fdebffff640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/9 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1149599550660 now=1149599677430 diff=126770
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 10: id=0x7fdec66ce640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/10 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1031266992082 now=1031267212925 diff=220843
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 11: id=0x7fdec5ecd640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/11 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1084427224751 now=1084427273714 diff=48963
curr_task=0
lock_hist: R:PATEXP U:PATEXP R:TASK_WQ U:TASK_WQ S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ
Thread 12: id=0x7fdec4e34640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/12 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1140285885158 now=1140286106179 diff=221021
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 13: id=0x7fdebef66640 act=1 glob=0 wq=1 rq=0 tl=1 tlsz=1 rqsz=2
1/13 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=1123191784575 now=1123276314947 diff=84530372
curr_task=0x7fdea4b1a8d0 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdea4d22000
stream=0x7fdea4d22000: [1754954774.076377] id=5489759 proto=?
flags=0x100c4a, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=0x44207490, pend_pos=(nil) waiting=0 epoch=0
p_stc=2 p_req=2 p_res=2 p_prp=2
frontend=<HTTPCLIENT> (id=4294967295 mode=http), listener=? (id=0)
backend=<HTTPCLIENT> (id=4294967295 mode=http)
server=<HTTPCLIENT> (id=1) addr=179.96.0.91:95
task=0x7fdea4b1a8d0 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=12(1/12) age=0s)
txn=0x7fdea4bd4080 flags=0x83000 meth=1 status=-1 req.st=MSG_DONE rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdea4ca6dc0 flags=0x00000460 ioto=0s state=EST endp=APPCTX,0x7fdea4c7dbd0,0x10000002 sub=0 rex=<NEVER> wex=<NEVER> rto=0s wto=<NEVER>
app0=0x7fdea4c7dbd0 st0=2 st1=0 applet=<HTTPCLIENT> tid=12 nice=0 calls=0 rate=0
scb=0x7fdea4d224c0 flags=0x00001c79 ioto=0s state=EST endp=CONN,0x7fdea4d04a30,0x40400001 sub=0 rex=0s wex=<NEVER> rto=0s wto=<NEVER>
h1s=0x7fdea4d04a30 h1s.flg=0x100090 .req.state=MSG_RQBEFORE .res.state=MSG_RPBEFORE
.meth=UNKNOWN status=0 .sd.flg=0x40400001 .sd.evts= .sc.flg=0x00001c79 .sc.app=0x7fdea4d22000 .sc.evts= .subs=(nil)
h1c=0x7fde28e9f730 h1c.flg=0x80000000 .sub=1 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdea4bec3a0 .exp=<NEVER>
co1=0x7fde292dd430 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=SERVER:0x44207490
flags=0x00000300 fd=52449 fd.state=10121 updt=1 fd.tmask=0x1000
req=0x7fdea4d22028 (f=0x1800001 an=0x8000 tofwd=0 total=161)
an_exp=<NEVER> buf=0x7fdea4d22030 data=0x7fdea5215650 o=161 p=161 i=1048415 size=1048576
htx=0x7fdea5215650 flags=0x10 size=1048528 data=161 used=5 wrap=NO extra=0
res=0x7fdea4d22070 (f=0x80000001 an=0x1400000 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdea4d22078 data=0x7fdea5115640 o=0 p=0 i=0 size=1048576
htx=0x7fdea5115640 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: S:NOTIF U:NOTIF R:TASK_WQ U:TASK_WQ S:NOTIF U:NOTIF S:IDLE_CONNS U:IDLE_CONNS
call trace(12):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db245 <2a 31 f6 e8 db f6 ff ff]: debug_handler+0x65/0xca > ha_thread_dump_one
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee4564092 <83 f9 3f 76 73 48 89 d1]: libc:+0x172092
| 0x5c90c5 <89 4d c8 e8 2b bf e6 ff]: cli_io_handler+0x2da75 > main-0x18b0
| 0x5263ac <48 89 df e8 34 71 0a 00]: process_stream+0x1cfc/0x3ac3 > sc_conn_sync_send
Thread 14: id=0x7fdebe765640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/14 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1093956999334 now=1093957007509 diff=8175
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 15: id=0x7fdebd6cc640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/15 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1195900767334 now=1195900883888 diff=116554
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 16: id=0x7fdebcecb640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/16 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1028720720517 now=1028721113598 diff=393081
curr_task=0
lock_hist: R:PATEXP U:PATEXP R:TASK_WQ U:TASK_WQ S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ
Thread 17: id=0x7fdeaffff640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/17 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1121815380980 now=1121815481558 diff=100578
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ
Thread 18: id=0x7fdeaef66640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/18 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1139843295507 now=1139843306195 diff=10688
curr_task=0
lock_hist: S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ
Thread 19: id=0x7fdeae765640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/19 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1080113468794 now=1080113528725 diff=59931
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ S:TASK_RQ U:TASK_RQ S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ
Thread 20: id=0x7fdead6cc640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/20 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=881315626758 now=881315700242 diff=73484
curr_task=0
lock_hist: S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ
Thread 21: id=0x7fdeacecb640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/21 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1068943657503 now=1068943678957 diff=21454
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:STK_TABLE U:STK_TABLE R:STK_SESS U:STK_SESS R:TASK_WQ U:TASK_WQ
Thread 22: id=0x7fdea3767640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/22 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=995880785467 now=995880795884 diff=10417
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 23: id=0x7fdea2f66640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/23 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=987578160636 now=987578194558 diff=33922
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 24: id=0x7fdea1ecd640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/24 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1087556010700 now=1087556020648 diff=9948
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 25: id=0x7fdea16cc640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/25 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1140757723516 now=1140757779348 diff=55832
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ
Thread 26: id=0x7fde97fff640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/26 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1024520170233 now=1024520247652 diff=77419
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 27: id=0x7fde977fe640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/27 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1069108959703 now=1069110123482 diff=1163779
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 28: id=0x7fde96765640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/28 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1102554394826 now=1102554711830 diff=317004
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ W:STK_SESS U:STK_SESS S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ
Thread 29: id=0x7fde95f64640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/29 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1115701631115 now=1115701846772 diff=215657
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 30: id=0x7fde95763640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/30 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1032752547059 now=1032752941968 diff=394909
curr_task=0
lock_hist: R:PATEXP U:PATEXP R:TASK_WQ U:TASK_WQ S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ
Thread 31: id=0x7fde82ecf640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/31 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1024508774929 now=1024508840385 diff=65456
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 32: id=0x7fde826ce640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/32 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=992731423280 now=992731431871 diff=8591
curr_task=0
lock_hist: S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ
Thread 33: id=0x7fde81635640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/33 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1023278956961 now=1023279297449 diff=340488
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ
Thread 34: id=0x7fde80e34640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/34 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1201556251372 now=1201556261874 diff=10502
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 35: id=0x7fde73767640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/35 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1016569680550 now=1016569760355 diff=79805
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 36: id=0x7fde72f66640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/36 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=949657576772 now=949657797350 diff=220578
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 37: id=0x7fde71ecd640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/37 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1134586667339 now=1134586677017 diff=9678
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ R:LISTENER U:LISTENER R:TASK_WQ U:TASK_WQ
Thread 38: id=0x7fde716cc640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/38 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1093909727454 now=1093909739226 diff=11772
curr_task=0
lock_hist: S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ
Thread 39: id=0x7fde6bfff640 act=1 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/39 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=1133712708057 now=1133764337074 diff=51629017
curr_task=0x7fdd6ceb56f0 (tasklet) calls=2
fct=0x4e1390(h1_io_cb) ctx=0x7fdd6cfe1bd0
lock_hist: W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ R:LISTENER U:LISTENER R:TASK_WQ U:TASK_WQ
call trace(17):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db245 <2a 31 f6 e8 db f6 ff ff]: debug_handler+0x65/0xca > ha_thread_dump_one
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee447d947 <48 83 c9 01 48 89 4a 08]: libc:+0x8b947
| 0x7fdee447ee07 <7c 24 30 e8 b9 e9 ff ff]: libc:+0x8ce07 > libc:+0x8b7c0
| 0x7fdee4480871 <4c 89 e7 e8 0f d8 ff ff]: libc:__libc_malloc+0x1c1/0x327 > libc:+0x8c080
| 0x64517a <89 c0 c3 e8 e6 07 df ff]: pool_get_from_os_noinc+0xca/0xf0 > main-0xf40
| 0x645232 <48 89 fb e8 7e fe ff ff]: pool_alloc_nocache+0x12/0xaa > pool_get_from_os_noinc
| 0x6460df <4c 89 f7 e8 41 f1 ff ff]: __pool_alloc+0x16f/0x279 > pool_alloc_nocache
| 0x4dfb80 <00 00 00 e8 f0 63 16 00]: main+0xa92e0 > __pool_alloc
| 0x4e1698 <4c 89 e7 e8 18 e1 ff ff]: h1_io_cb+0x308/0xe71 > main+0xa8f10
Thread 40: id=0x7fde6b7fe640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/40 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1013481799997 now=1013481811940 diff=11943
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ R:LISTENER U:LISTENER R:TASK_WQ U:TASK_WQ
Thread 41: id=0x7fde6a765640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/41 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1090232612112 now=1090232625113 diff=13001
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ S:TASK_RQ U:TASK_RQ S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ
*>Thread 42: id=0x7fde69f64640 act=1 glob=0 wq=1 rq=0 tl=1 tlsz=2 rqsz=4
1/42 stuck=1 prof=0 harmless=0 isolated=0
cpu_ns: poll=1043741907604 now=1045151869499 diff=1409961895
curr_task=0x7fdd20f77ef0 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdd20b88580
stream=0x7fdd20b88580: [1754954772.593040] id=5488724 proto=?
flags=0x100c4a, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=0x44207490, pend_pos=(nil) waiting=0 epoch=0
p_stc=2 p_req=2 p_res=2 p_prp=2
frontend=<HTTPCLIENT> (id=4294967295 mode=http), listener=? (id=0)
backend=<HTTPCLIENT> (id=4294967295 mode=http)
server=<HTTPCLIENT> (id=1) addr=185.236.183.85:95
task=0x7fdd20f77ef0 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=41(1/41) age=0s)
txn=0x7fdd20fd1a00 flags=0x83000 meth=1 status=-1 req.st=MSG_DONE rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdd20fd1870 flags=0x00000460 ioto=0s state=EST endp=APPCTX,0x7fdd20f9a0a0,0x10000002 sub=0 rex=<NEVER> wex=<NEVER> rto=0s wto=<NEVER>
app0=0x7fdd20f9a0a0 st0=2 st1=0 applet=<HTTPCLIENT> tid=41 nice=0 calls=0 rate=0
scb=0x7fdd20fd19a0 flags=0x00001c79 ioto=0s state=EST endp=CONN,0x7fdd20f63000,0x40400001 sub=0 rex=0s wex=<NEVER> rto=0s wto=<NEVER>
h1s=0x7fdd20f63000 h1s.flg=0x100090 .req.state=MSG_RQBEFORE .res.state=MSG_RPBEFORE
.meth=UNKNOWN status=0 .sd.flg=0x40400001 .sd.evts= .sc.flg=0x00001c79 .sc.app=0x7fdd20b88580 .sc.evts= .subs=(nil)
h1c=0x7fdd4cbd4ad0 h1c.flg=0x80000000 .sub=1 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdd20f36910 .exp=<NEVER>
co1=0x7fdd4cbdec00 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=SERVER:0x44207490
flags=0x00000300 fd=21050 fd.state=10121 updt=1 fd.tmask=0x20000000000
req=0x7fdd20b885a8 (f=0x1800001 an=0x8000 tofwd=0 total=168)
an_exp=<NEVER> buf=0x7fdd20b885b0 data=0x7fdd21015190 o=168 p=168 i=1048408 size=1048576
htx=0x7fdd21015190 flags=0x10 size=1048528 data=168 used=5 wrap=NO extra=0
res=0x7fdd20b885f0 (f=0x80000001 an=0x1400000 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdd20b885f8 data=0x7fdd20d12e20 o=0 p=0 i=0 size=1048576
htx=0x7fdd20d12e20 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: S:NOTIF U:NOTIF S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ S:IDLE_CONNS U:IDLE_CONNS
call trace(15):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5dae3d <48 89 fb e8 e3 fa ff ff]: ha_thread_dump_fill+0x9d/0x9f > ha_thread_dump_one
| 0x5db047 <00 89 de e8 59 fd ff ff]: ha_dump_backtrace+0x2df7 > ha_thread_dump_fill
| 0x5db195 <48 89 e5 e8 4b fe ff ff]: ha_dump_backtrace+0x2f45 > ha_dump_backtrace+0x2d90
| 0x6c5f8d <44 00 00 e8 f3 51 f1 ff]: wdt_handler+0x1ed/0x1ff > ha_panic
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee4564092 <83 f9 3f 76 73 48 89 d1]: libc:+0x172092
| 0x5c90c5 <89 4d c8 e8 2b bf e6 ff]: cli_io_handler+0x2da75 > main-0x18b0
| 0x5263ac <48 89 df e8 34 71 0a 00]: process_stream+0x1cfc/0x3ac3 > sc_conn_sync_send
Thread 43: id=0x7fde68ecb640 act=1 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/43 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=1067301299109 now=1067351414474 diff=50115365
curr_task=0x7fdedcb45360 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdcdcbb7250
stream=0x7fdcdcbb7250: [1754954774.203786] id=5489839 proto=tcpv4 source=138.0.147.96:56416
flags=0xc08, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=(nil), pend_pos=(nil) waiting=0 epoch=0
p_stc=1 p_req=1 p_res=0 p_prp=0
frontend=ssl (id=3 mode=http), listener=? (id=874) addr=179.96.0.90:80
backend=hyper_vod_tvs (id=15 mode=http)
server=<none> (id=0)
task=0x7fdedcb45360 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=42(1/42) age=0s)
txn=0x7fdcdcb37de0 flags=0x3000 meth=1 status=-1 req.st=MSG_BODY rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdcdd063000 flags=0x00000422 ioto=1m40s state=EST endp=CONN,0x7fdcdcd75b20,0x40404001 sub=1 rex=<NEVER> wex=<NEVER> rto=1m40s wto=<NEVER>
h1s=0x7fdcdcd75b20 h1s.flg=0x100010 .req.state=MSG_DONE .res.state=MSG_RPBEFORE
.meth=GET status=0 .sd.flg=0x40404001 .sd.evts= .sc.flg=0x00000422 .sc.app=0x7fdcdcbb7250 .sc.evts=
.subs=0x7fdcdd063018(ev=1 tl=0x7fdcdcb8af30 tl.calls=0 tl.ctx=0x7fdcdd063000 tl.fct=sc_conn_io_cb+0)
h1c=0x7fdcdd0b26d0 h1c.flg=0x0 .sub=0 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdcdcb9d030 .exp=<NEVER>
co0=0x7fde58ec5530 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=LISTENER:0x442fac90
flags=0x80000300 fd=81531 fd.state=20 updt=0 fd.tmask=0x40000000000
scb=0x7fdcdcd85550 flags=0x00000821 ioto=0s state=INI endp=NONE,(nil),0x00000010 sub=0 rex=<NEVER> wex=<NEVER> rto=<NEVER> wto=<NEVER>
req=0x7fdcdcbb7278 (f=0x808001 an=0x900 tofwd=0 total=412)
an_exp=<NEVER> buf=0x7fdcdcbb7280 data=0x7fdcdce26940 o=0 p=0 i=1048576 size=1048576
htx=0x7fdcdce26940 flags=0x14 size=1048528 data=577 used=14 wrap=NO extra=0
res=0x7fdcdcbb72c0 (f=0x80000000 an=0x0 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdcdcbb72c8 data=0x7fdcdcc12e10 o=0 p=0 i=0 size=1048576
htx=0x7fdcdcc12e10 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: R:TASK_WQ U:TASK_WQ R:LISTENER U:LISTENER R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP
call trace(22):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db245 <2a 31 f6 e8 db f6 ff ff]: debug_handler+0x65/0xca > ha_thread_dump_one
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee447d947 <48 83 c9 01 48 89 4a 08]: libc:+0x8b947
| 0x7fdee447ee07 <7c 24 30 e8 b9 e9 ff ff]: libc:+0x8ce07 > libc:+0x8b7c0
| 0x7fdee4480871 <4c 89 e7 e8 0f d8 ff ff]: libc:__libc_malloc+0x1c1/0x327 > libc:+0x8c080
| 0x64517a <89 c0 c3 e8 e6 07 df ff]: pool_get_from_os_noinc+0xca/0xf0 > main-0xf40
| 0x645232 <48 89 fb e8 7e fe ff ff]: pool_alloc_nocache+0x12/0xaa > pool_get_from_os_noinc
| 0x6460df <4c 89 f7 e8 41 f1 ff ff]: __pool_alloc+0x16f/0x279 > pool_alloc_nocache
| 0x57a6f4 <89 04 24 e8 7c b8 0c 00]: main+0x143e54 > __pool_alloc
| 0x57961f <7b 18 4c 89 ee ff 51 08]: sample_process_cnv+0x6f/0x90
| 0x57969c <4c 89 ef e8 14 ff ff ff]: sample_process+0x5c/0xa7 > sample_process_cnv
| 0x691822 <4c 89 f7 e8 1e 7e ee ff]: acl_exec_cond+0x132/0x249 > sample_process
| 0x53c98d <89 7d c8 e8 63 4d 15 00]: process_stream+0x182dd > acl_exec_cond
| 0x53f696 <89 4d 98 e8 fa d1 ff ff]: http_process_req_common+0xb6/0x1590 > process_stream+0x181e0
| 0x5259cf <4c 89 ff e8 11 9c 01 00]: process_stream+0x131f/0x3ac3 > http_process_req_common
Thread 44: id=0x7fde5ffff640 act=1 glob=0 wq=1 rq=0 tl=1 tlsz=0 rqsz=1
1/44 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=1133607475117 now=1133660609200 diff=53134083
curr_task=0x7fdd94bc6f20 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdd94bf4b70
stream=0x7fdd94bf4b70: [1754954774.436784] id=5489940 proto=tcpv4 source=200.165.205.91:35964
flags=0xc08, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=(nil), pend_pos=(nil) waiting=0 epoch=0
p_stc=1 p_req=1 p_res=0 p_prp=0
frontend=ssl (id=3 mode=http), listener=? (id=875) addr=179.96.0.90:80
backend=hyper_vod_tvs (id=15 mode=http)
server=<none> (id=0)
task=0x7fdd94bc6f20 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=43(1/43) age=0s)
txn=0x7fdd94b2a830 flags=0x3000 meth=1 status=-1 req.st=MSG_BODY rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdd94ec5f30 flags=0x00000422 ioto=1m40s state=EST endp=CONN,0x7fdd94b2a760,0x40404001 sub=1 rex=<NEVER> wex=<NEVER> rto=1m40s wto=<NEVER>
h1s=0x7fdd94b2a760 h1s.flg=0x100010 .req.state=MSG_DONE .res.state=MSG_RPBEFORE
.meth=GET status=0 .sd.flg=0x40404001 .sd.evts= .sc.flg=0x00000422 .sc.app=0x7fdd94bf4b70 .sc.evts=
.subs=0x7fdd94ec5f48(ev=1 tl=0x7fdd94edaac0 tl.calls=0 tl.ctx=0x7fdd94ec5f30 tl.fct=sc_conn_io_cb+0)
h1c=0x7fdd94f51fe0 h1c.flg=0x0 .sub=0 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdd94b2ac10 .exp=<NEVER>
co0=0x7fdd94f8f490 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=LISTENER:0x442faf70
flags=0x80000300 fd=36810 fd.state=120 updt=0 fd.tmask=0x80000000000
scb=0x7fdd94dbd670 flags=0x00000821 ioto=0s state=INI endp=NONE,(nil),0x00000010 sub=0 rex=<NEVER> wex=<NEVER> rto=<NEVER> wto=<NEVER>
req=0x7fdd94bf4b98 (f=0x808001 an=0x900 tofwd=0 total=637)
an_exp=<NEVER> buf=0x7fdd94bf4ba0 data=0x7fdd9501a2d0 o=0 p=0 i=1048576 size=1048576
htx=0x7fdd9501a2d0 flags=0x14 size=1048528 data=806 used=14 wrap=NO extra=0
res=0x7fdd94bf4be0 (f=0x80000000 an=0x0 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdd94bf4be8 data=0x7fdd94c12e10 o=0 p=0 i=0 size=1048576
htx=0x7fdd94c12e10 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: R:TASK_WQ U:TASK_WQ R:LISTENER U:LISTENER R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP
call trace(19):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db245 <2a 31 f6 e8 db f6 ff ff]: debug_handler+0x65/0xca > ha_thread_dump_one
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee447d947 <48 83 c9 01 48 89 4a 08]: libc:+0x8b947
| 0x7fdee447ee07 <7c 24 30 e8 b9 e9 ff ff]: libc:+0x8ce07 > libc:+0x8b7c0
| 0x7fdee4480871 <4c 89 e7 e8 0f d8 ff ff]: libc:__libc_malloc+0x1c1/0x327 > libc:+0x8c080
| 0x64517a <89 c0 c3 e8 e6 07 df ff]: pool_get_from_os_noinc+0xca/0xf0 > main-0xf40
| 0x645232 <48 89 fb e8 7e fe ff ff]: pool_alloc_nocache+0x12/0xaa > pool_get_from_os_noinc
| 0x6460df <4c 89 f7 e8 41 f1 ff ff]: __pool_alloc+0x16f/0x279 > pool_alloc_nocache
| 0x63403c <00 31 f6 e8 34 1f 01 00]: task_run_applet+0x3e2dc > __pool_alloc
| 0x53ca1c <89 ef 4c 89 5d c8 ff d0]: process_stream+0x1836c
| 0x53f696 <89 4d 98 e8 fa d1 ff ff]: http_process_req_common+0xb6/0x1590 > process_stream+0x181e0
| 0x5259cf <4c 89 ff e8 11 9c 01 00]: process_stream+0x131f/0x3ac3 > http_process_req_common
Thread 45: id=0x7fde5ef66640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/45 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1020674253619 now=1020674265385 diff=11766
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:STK_TABLE U:STK_TABLE R:STK_SESS U:STK_SESS R:TASK_WQ U:TASK_WQ
Thread 46: id=0x7fde5e765640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/46 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=992590056959 now=992590065990 diff=9031
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 47: id=0x7fde5d6cc640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/47 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1036431148689 now=1036432143992 diff=995303
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
>Thread 48: id=0x7fde4bfff640 act=1 glob=0 wq=1 rq=0 tl=1 tlsz=1 rqsz=2
1/48 stuck=1 prof=0 harmless=0 isolated=0
cpu_ns: poll=1056036426858 now=1057154891317 diff=1118464459
curr_task=0x7fddd8e7d2e0 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fddd8bc11b0
stream=0x7fddd8bc11b0: [1754954773.378485] id=5489309 proto=?
flags=0x100c4a, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=0x44207490, pend_pos=(nil) waiting=0 epoch=0
p_stc=2 p_req=2 p_res=2 p_prp=2
frontend=<HTTPCLIENT> (id=4294967295 mode=http), listener=? (id=0)
backend=<HTTPCLIENT> (id=4294967295 mode=http)
server=<HTTPCLIENT> (id=1) addr=179.96.0.91:95
task=0x7fddd8e7d2e0 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=47(1/47) age=0s)
txn=0x7fddd8b87a00 flags=0x83000 meth=1 status=-1 req.st=MSG_DONE rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fddd8e7b670 flags=0x00000460 ioto=0s state=EST endp=APPCTX,0x451becc0,0x10000002 sub=0 rex=<NEVER> wex=<NEVER> rto=0s wto=<NEVER>
app0=0x451becc0 st0=2 st1=0 applet=<HTTPCLIENT> tid=47 nice=0 calls=0 rate=0
scb=0x7fddd8e7b730 flags=0x00001c79 ioto=0s state=EST endp=CONN,0x7fddd8db0340,0x40400001 sub=0 rex=0s wex=<NEVER> rto=0s wto=<NEVER>
h1s=0x7fddd8db0340 h1s.flg=0x100090 .req.state=MSG_RQBEFORE .res.state=MSG_RPBEFORE
.meth=UNKNOWN status=0 .sd.flg=0x40400001 .sd.evts= .sc.flg=0x00001c79 .sc.app=0x7fddd8bc11b0 .sc.evts= .subs=(nil)
h1c=0x7fde1907c0d0 h1c.flg=0x80000000 .sub=1 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fddd8b87af0 .exp=<NEVER>
co1=0x7fde18d69c00 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=SERVER:0x44207490
flags=0x00000300 fd=80230 fd.state=10121 updt=1 fd.tmask=0x800000000000
req=0x7fddd8bc11d8 (f=0x1800001 an=0x8000 tofwd=0 total=160)
an_exp=<NEVER> buf=0x7fddd8bc11e0 data=0x7fddd8f3a200 o=160 p=160 i=1048416 size=1048576
htx=0x7fddd8f3a200 flags=0x10 size=1048528 data=160 used=5 wrap=NO extra=0
res=0x7fddd8bc1220 (f=0x80000001 an=0x1400000 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fddd8bc1228 data=0x7fddd8c12e10 o=0 p=0 i=0 size=1048576
htx=0x7fddd8c12e10 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: S:NOTIF U:NOTIF S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ S:IDLE_CONNS U:IDLE_CONNS
call trace(12):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db245 <2a 31 f6 e8 db f6 ff ff]: debug_handler+0x65/0xca > ha_thread_dump_one
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee4564092 <83 f9 3f 76 73 48 89 d1]: libc:+0x172092
| 0x5c90c5 <89 4d c8 e8 2b bf e6 ff]: cli_io_handler+0x2da75 > main-0x18b0
| 0x5263ac <48 89 df e8 34 71 0a 00]: process_stream+0x1cfc/0x3ac3 > sc_conn_sync_send
Thread 49: id=0x7fde43fff640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/49 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1121512432562 now=1121512443824 diff=11262
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 50: id=0x7fde4af66640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/50 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1084067397639 now=1084067409583 diff=11944
curr_task=0
lock_hist: S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ
Thread 51: id=0x7fde49ecd640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/51 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1109437327876 now=1109437338857 diff=10981
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:STK_TABLE U:STK_TABLE R:STK_SESS U:STK_SESS R:TASK_WQ U:TASK_WQ
Thread 52: id=0x7fde496cc640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/52 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1040077748729 now=1040077768469 diff=19740
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 53: id=0x7fde48ecb640 act=1 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/53 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=1082639890129 now=1082689296539 diff=49406410
curr_task=0x7fde38b99350 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fde38b8d770
stream=0x7fde38b8d770: [1754954774.519567] id=5489976 proto=tcpv4 source=187.72.78.85:47467
flags=0xc08, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=(nil), pend_pos=(nil) waiting=0 epoch=0
p_stc=1 p_req=1 p_res=0 p_prp=0
frontend=ssl (id=3 mode=http), listener=? (id=884) addr=179.96.0.90:80
backend=hyper_vod_tvs (id=15 mode=http)
server=<none> (id=0)
task=0x7fde38b99350 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=52(1/52) age=0s)
txn=0x7fde39016260 flags=0x3000 meth=1 status=-1 req.st=MSG_BODY rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fde3916d8c0 flags=0x00000422 ioto=1m40s state=EST endp=CONN,0x7fde38dd6500,0x40404001 sub=1 rex=<NEVER> wex=<NEVER> rto=1m40s wto=<NEVER>
h1s=0x7fde38dd6500 h1s.flg=0x100010 .req.state=MSG_DONE .res.state=MSG_RPBEFORE
.meth=GET status=0 .sd.flg=0x40404001 .sd.evts= .sc.flg=0x00000422 .sc.app=0x7fde38b8d770 .sc.evts=
.subs=0x7fde3916d8d8(ev=1 tl=0x7fde38d43080 tl.calls=0 tl.ctx=0x7fde3916d8c0 tl.fct=sc_conn_io_cb+0)
h1c=0x7fde38e92610 h1c.flg=0x0 .sub=0 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fde38e120e0 .exp=<NEVER>
co0=0x7fde38d73e20 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=LISTENER:0x442fc950
flags=0x80000300 fd=20423 fd.state=120 updt=0 fd.tmask=0x10000000000000
scb=0x7fde38e41eb0 flags=0x00000821 ioto=0s state=INI endp=NONE,(nil),0x00000010 sub=0 rex=<NEVER> wex=<NEVER> rto=<NEVER> wto=<NEVER>
req=0x7fde38b8d798 (f=0x808001 an=0x900 tofwd=0 total=602)
an_exp=<NEVER> buf=0x7fde38b8d7a0 data=0x7fde38c12f60 o=0 p=0 i=1048576 size=1048576
htx=0x7fde38c12f60 flags=0x14 size=1048528 data=767 used=14 wrap=NO extra=0
res=0x7fde38b8d7e0 (f=0x80000000 an=0x0 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fde38b8d7e8 data=0x7fde38f13cc0 o=0 p=0 i=0 size=1048576
htx=0x7fde38f13cc0 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: R:TASK_WQ U:TASK_WQ R:LISTENER U:LISTENER R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP
call trace(22):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db245 <2a 31 f6 e8 db f6 ff ff]: debug_handler+0x65/0xca > ha_thread_dump_one
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee447d947 <48 83 c9 01 48 89 4a 08]: libc:+0x8b947
| 0x7fdee447ee07 <7c 24 30 e8 b9 e9 ff ff]: libc:+0x8ce07 > libc:+0x8b7c0
| 0x7fdee4480871 <4c 89 e7 e8 0f d8 ff ff]: libc:__libc_malloc+0x1c1/0x327 > libc:+0x8c080
| 0x64517a <89 c0 c3 e8 e6 07 df ff]: pool_get_from_os_noinc+0xca/0xf0 > main-0xf40
| 0x645232 <48 89 fb e8 7e fe ff ff]: pool_alloc_nocache+0x12/0xaa > pool_get_from_os_noinc
| 0x6460df <4c 89 f7 e8 41 f1 ff ff]: __pool_alloc+0x16f/0x279 > pool_alloc_nocache
| 0x57a6f4 <89 04 24 e8 7c b8 0c 00]: main+0x143e54 > __pool_alloc
| 0x57961f <7b 18 4c 89 ee ff 51 08]: sample_process_cnv+0x6f/0x90
| 0x57969c <4c 89 ef e8 14 ff ff ff]: sample_process+0x5c/0xa7 > sample_process_cnv
| 0x691822 <4c 89 f7 e8 1e 7e ee ff]: acl_exec_cond+0x132/0x249 > sample_process
| 0x53c98d <89 7d c8 e8 63 4d 15 00]: process_stream+0x182dd > acl_exec_cond
| 0x53f696 <89 4d 98 e8 fa d1 ff ff]: http_process_req_common+0xb6/0x1590 > process_stream+0x181e0
| 0x5259cf <4c 89 ff e8 11 9c 01 00]: process_stream+0x131f/0x3ac3 > http_process_req_common
Thread 54: id=0x7fde42f66640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/54 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1182073880601 now=1182073890199 diff=9598
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 55: id=0x7fde42765640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/55 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1025495397342 now=1025495411606 diff=14264
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 56: id=0x7fde416cc640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/56 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1165770825575 now=1165770901129 diff=75554
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 57: id=0x7fde40ecb640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/57 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1096591685645 now=1096591742304 diff=56659
curr_task=0
lock_hist: S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ R:LISTENER U:LISTENER R:TASK_WQ U:TASK_WQ
Thread 58: id=0x7fde33fff640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/58 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1135162015536 now=1135162092727 diff=77191
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 59: id=0x7fde32f66640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/59 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1015240302262 now=1015240315083 diff=12821
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ
Thread 60: id=0x7fde31ecd640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/60 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1173898733261 now=1173898801891 diff=68630
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 61: id=0x7fde316cc640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/61 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1137846400767 now=1137846581841 diff=181074
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 62: id=0x7fde27fff640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/62 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1061057745820 now=1061057767597 diff=21777
curr_task=0
lock_hist: R:PATEXP U:PATEXP R:TASK_WQ U:TASK_WQ S:IDLE_CONNS U:IDLE_CONNS R:TASK_WQ U:TASK_WQ
Thread 63: id=0x7fde26f66640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/63 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=1063036175022 now=1063036185277 diff=10255
curr_task=0
lock_hist: R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:TASK_WQ U:TASK_WQ
Thread 64: id=0x7fde26765640 act=1 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
1/64 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=1006493128324 now=1006543395861 diff=50267537
curr_task=0x7fde18f20d00 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fde19029790
stream=0x7fde19029790: [1754954774.623719] id=5490020 proto=tcpv4 source=179.43.49.8:52506
flags=0xc08, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=(nil), pend_pos=(nil) waiting=0 epoch=0
p_stc=1 p_req=1 p_res=0 p_prp=0
frontend=ssl (id=3 mode=http), listener=? (id=895) addr=179.96.0.90:80
backend=hyper_vod_tvs (id=15 mode=http)
server=<none> (id=0)
task=0x7fde18f20d00 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=63(1/63) age=0s)
txn=0x7fde18b367b0 flags=0x3000 meth=1 status=-1 req.st=MSG_BODY rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fde18b9c400 flags=0x00000422 ioto=1m40s state=EST endp=CONN,0x7fde18d7d850,0x40404001 sub=1 rex=<NEVER> wex=<NEVER> rto=1m40s wto=<NEVER>
h1s=0x7fde18d7d850 h1s.flg=0x100010 .req.state=MSG_DONE .res.state=MSG_RPBEFORE
.meth=GET status=0 .sd.flg=0x40404001 .sd.evts= .sc.flg=0x00000422 .sc.app=0x7fde19029790 .sc.evts=
.subs=0x7fde18b9c418(ev=1 tl=0x7fde18bac710 tl.calls=0 tl.ctx=0x7fde18b9c400 tl.fct=sc_conn_io_cb+0)
h1c=0x7fde18d7d780 h1c.flg=0x0 .sub=0 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fde18da3fb0 .exp=<NEVER>
co0=0x7fde18f46320 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=LISTENER:0x442fe8f0
flags=0x80000300 fd=55755 fd.state=120 updt=0 fd.tmask=0x8000000000000000
scb=0x7fde18f82a60 flags=0x00000821 ioto=0s state=INI endp=NONE,(nil),0x00000010 sub=0 rex=<NEVER> wex=<NEVER> rto=<NEVER> wto=<NEVER>
req=0x7fde190297b8 (f=0x808001 an=0x900 tofwd=0 total=625)
an_exp=<NEVER> buf=0x7fde190297c0 data=0x7fde18c21eb0 o=0 p=0 i=1048576 size=1048576
htx=0x7fde18c21eb0 flags=0x14 size=1048528 data=788 used=14 wrap=NO extra=0
res=0x7fde19029800 (f=0x80000000 an=0x0 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fde19029808 data=0x7fde18e1eed0 o=0 p=0 i=0 size=1048576
htx=0x7fde18e1eed0 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: R:TASK_WQ U:TASK_WQ R:LISTENER U:LISTENER R:TASK_WQ U:TASK_WQ R:PATEXP U:PATEXP
call trace(22):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db245 <2a 31 f6 e8 db f6 ff ff]: debug_handler+0x65/0xca > ha_thread_dump_one
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee447d947 <48 83 c9 01 48 89 4a 08]: libc:+0x8b947
| 0x7fdee447ee07 <7c 24 30 e8 b9 e9 ff ff]: libc:+0x8ce07 > libc:+0x8b7c0
| 0x7fdee4480871 <4c 89 e7 e8 0f d8 ff ff]: libc:__libc_malloc+0x1c1/0x327 > libc:+0x8c080
| 0x64517a <89 c0 c3 e8 e6 07 df ff]: pool_get_from_os_noinc+0xca/0xf0 > main-0xf40
| 0x645232 <48 89 fb e8 7e fe ff ff]: pool_alloc_nocache+0x12/0xaa > pool_get_from_os_noinc
| 0x6460df <4c 89 f7 e8 41 f1 ff ff]: __pool_alloc+0x16f/0x279 > pool_alloc_nocache
| 0x57a6f4 <89 04 24 e8 7c b8 0c 00]: main+0x143e54 > __pool_alloc
| 0x57961f <7b 18 4c 89 ee ff 51 08]: sample_process_cnv+0x6f/0x90
| 0x57969c <4c 89 ef e8 14 ff ff ff]: sample_process+0x5c/0xa7 > sample_process_cnv
| 0x691822 <4c 89 f7 e8 1e 7e ee ff]: acl_exec_cond+0x132/0x249 > sample_process
| 0x53c98d <89 7d c8 e8 63 4d 15 00]: process_stream+0x182dd > acl_exec_cond
| 0x53f696 <89 4d 98 e8 fa d1 ff ff]: http_process_req_common+0xb6/0x1590 > process_stream+0x181e0
| 0x5259cf <4c 89 ff e8 11 9c 01 00]: process_stream+0x131f/0x3ac3 > http_process_req_common
Thread 65: id=0x7fde256cc640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/1 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=785606100308 now=785606113585 diff=13277
curr_task=0
lock_hist: R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM S:IDLE_CONNS U:IDLE_CONNS
Thread 66: id=0x7fde13fff640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/2 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=805728430368 now=805728448593 diff=18225
curr_task=0
lock_hist: S:IDLE_CONNS U:IDLE_CONNS R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 67: id=0x7fde0b5ff640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/3 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=761401959773 now=761402035219 diff=75446
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 68: id=0x7fde126ce640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/4 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=810556606594 now=810556613780 diff=7186
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 69: id=0x7fde11ecd640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/5 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=782801938865 now=782801952916 diff=14051
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 70: id=0x7fde10e34640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/6 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=775300059964 now=775300086739 diff=26775
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 71: id=0x7fde0bfff640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/7 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=761538360746 now=761538372900 diff=12154
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 72: id=0x7fde0a566640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/8 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=774407518444 now=774407546221 diff=27777
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 73: id=0x7fde09d65640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/9 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=793954677442 now=793954722611 diff=45169
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 74: id=0x7fde08ccc640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/10 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=752062316484 now=752062330412 diff=13928
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 75: id=0x7fddfbfff640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/11 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=740818622597 now=740818676242 diff=53645
curr_task=0
lock_hist: R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM S:IDLE_CONNS U:IDLE_CONNS
Thread 76: id=0x7fddfb7fe640 act=1 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/12 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=700900151297 now=700950944926 diff=50793629
curr_task=0x7fdd38ea8840 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdd38b67b80
stream=0x7fdd38b67b80: [1754954774.657010] id=5490034 proto=tcpv4 source=164.163.99.28:6072
flags=0xc08, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=(nil), pend_pos=(nil) waiting=0 epoch=0
p_stc=1 p_req=1 p_res=0 p_prp=0
frontend=ssl (id=3 mode=http), listener=? (id=1291) addr=177.126.14.18:80
backend=hyper_vod_tvs (id=15 mode=http)
server=<none> (id=0)
task=0x7fdd38ea8840 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=75(2/11) age=0s)
txn=0x7fdd38e92100 flags=0x3000 meth=1 status=-1 req.st=MSG_BODY rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdd38e3b580 flags=0x00000422 ioto=1m40s state=EST endp=CONN,0x7fdd38b3ca70,0x40404001 sub=1 rex=<NEVER> wex=<NEVER> rto=1m40s wto=<NEVER>
h1s=0x7fdd38b3ca70 h1s.flg=0x100010 .req.state=MSG_DONE .res.state=MSG_RPBEFORE
.meth=GET status=0 .sd.flg=0x40404001 .sd.evts= .sc.flg=0x00000422 .sc.app=0x7fdd38b67b80 .sc.evts=
.subs=0x7fdd38e3b598(ev=1 tl=0x7fdd38bcdc10 tl.calls=0 tl.ctx=0x7fdd38e3b580 tl.fct=sc_conn_io_cb+0)
h1c=0x7fdd38be6820 h1c.flg=0x0 .sub=0 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdd38b2b620 .exp=<NEVER>
co0=0x7fdeb0deb160 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=LISTENER:0x44344750
flags=0x80000300 fd=81720 fd.state=120 updt=0 fd.tmask=0x800
scb=0x7fdd38d4ac70 flags=0x00000821 ioto=0s state=INI endp=NONE,(nil),0x00000010 sub=0 rex=<NEVER> wex=<NEVER> rto=<NEVER> wto=<NEVER>
req=0x7fdd38b67ba8 (f=0x808001 an=0x900 tofwd=0 total=522)
an_exp=<NEVER> buf=0x7fdd38b67bb0 data=0x7fdd38f14360 o=0 p=0 i=1048576 size=1048576
htx=0x7fdd38f14360 flags=0x14 size=1048528 data=691 used=14 wrap=NO extra=0
res=0x7fdd38b67bf0 (f=0x80000000 an=0x0 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdd38b67bf8 data=0x7fdd39014370 o=0 p=0 i=0 size=1048576
htx=0x7fdd39014370 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: W:LBPRM U:LBPRM S:IDLE_CONNS U:IDLE_CONNS R:LISTENER U:LISTENER R:PATEXP U:PATEXP
call trace(22):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db245 <2a 31 f6 e8 db f6 ff ff]: debug_handler+0x65/0xca > ha_thread_dump_one
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee447d947 <48 83 c9 01 48 89 4a 08]: libc:+0x8b947
| 0x7fdee447ee07 <7c 24 30 e8 b9 e9 ff ff]: libc:+0x8ce07 > libc:+0x8b7c0
| 0x7fdee4480871 <4c 89 e7 e8 0f d8 ff ff]: libc:__libc_malloc+0x1c1/0x327 > libc:+0x8c080
| 0x64517a <89 c0 c3 e8 e6 07 df ff]: pool_get_from_os_noinc+0xca/0xf0 > main-0xf40
| 0x645232 <48 89 fb e8 7e fe ff ff]: pool_alloc_nocache+0x12/0xaa > pool_get_from_os_noinc
| 0x6460df <4c 89 f7 e8 41 f1 ff ff]: __pool_alloc+0x16f/0x279 > pool_alloc_nocache
| 0x57a6f4 <89 04 24 e8 7c b8 0c 00]: main+0x143e54 > __pool_alloc
| 0x57961f <7b 18 4c 89 ee ff 51 08]: sample_process_cnv+0x6f/0x90
| 0x57969c <4c 89 ef e8 14 ff ff ff]: sample_process+0x5c/0xa7 > sample_process_cnv
| 0x691822 <4c 89 f7 e8 1e 7e ee ff]: acl_exec_cond+0x132/0x249 > sample_process
| 0x53c98d <89 7d c8 e8 63 4d 15 00]: process_stream+0x182dd > acl_exec_cond
| 0x53f696 <89 4d 98 e8 fa d1 ff ff]: http_process_req_common+0xb6/0x1590 > process_stream+0x181e0
| 0x5259cf <4c 89 ff e8 11 9c 01 00]: process_stream+0x131f/0x3ac3 > http_process_req_common
Thread 77: id=0x7fddfa765640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/13 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=814461504426 now=814461572972 diff=68546
curr_task=0
lock_hist: S:IDLE_CONNS U:IDLE_CONNS R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 78: id=0x7fddf9f64640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/14 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=805480178316 now=805480222661 diff=44345
curr_task=0
lock_hist: R:STK_SESS U:STK_SESS R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 79: id=0x7fddf9763640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/15 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=749708942719 now=749709124559 diff=181840
curr_task=0
lock_hist: S:IDLE_CONNS U:IDLE_CONNS R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 80: id=0x7fddf8f62640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/16 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=809269630998 now=809269654393 diff=23395
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 81: id=0x7fddef767640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/17 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=782261735892 now=782261790871 diff=54979
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 82: id=0x7fddeef66640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/18 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=758938452229 now=758938587205 diff=134976
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 83: id=0x7fddee765640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/19 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=773552920555 now=773553073832 diff=153277
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 84: id=0x7fdded6cc640 act=1 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/20 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=777030890062 now=777118394252 diff=87504190
curr_task=0x7fdd1cd95ae0 (task) calls=2 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdd1ce08090
stream=0x7fdd1ce08090: [1754954762.401769] id=5482018 proto=tcpv4 source=186.235.63.51:37883
flags=0x35c0a, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=(nil), pend_pos=(nil) waiting=0 epoch=0
p_stc=3 p_req=3 p_res=3 p_prp=2
frontend=ssl (id=3 mode=http), listener=? (id=1299) addr=177.126.14.18:80
backend=hyper_vod_tvs (id=15 mode=http)
server=tvs_hyper (id=1) addr=unix
task=0x7fdd1cd95ae0 (state=0x01 nice=0 calls=2 rate=0 exp=1m27s tid=83(2/19) age=13s)
txn=0x7fdd1ce08630 flags=0x3000 meth=1 status=502 req.st=MSG_DONE rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdd1cd95cc0 flags=0x000004a2 ioto=1m40s state=EST endp=CONN,0x7fdd1cd95b70,0x50404001 sub=1 rex=<NEVER> wex=<NEVER> rto=1m27s wto=<NEVER>
h1s=0x7fdd1cd95b70 h1s.flg=0x100010 .req.state=MSG_DONE .res.state=MSG_RPBEFORE
.meth=GET status=0 .sd.flg=0x50404001 .sd.evts= .sc.flg=0x000004a2 .sc.app=0x7fdd1ce08090 .sc.evts=
.subs=0x7fdd1cd95cd8(ev=1 tl=0x7fdd1ce08480 tl.calls=0 tl.ctx=0x7fdd1cd95cc0 tl.fct=sc_conn_io_cb+0)
h1c=0x7fdd1cd95930 h1c.flg=0x0 .sub=0 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdd1cd95a50 .exp=<NEVER>
co0=0x7fdd1cb57790 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=LISTENER:0x44345e50
flags=0x80000300 fd=61489 fd.state=120 updt=0 fd.tmask=0x80000
scb=0x7fdd1ce084d0 flags=0x0006103d ioto=1m40s state=CLO endp=CONN,0x7fdd1cdac2d0,0x50418a01 sub=0 rex=<NEVER> wex=<NEVER> rto=1m40s wto=<NEVER>
h1s=0x7fdd1cdac2d0 h1s.flg=0x14010 .req.state=MSG_DONE .res.state=MSG_RPBEFORE
.meth=GET status=0 .sd.flg=0x50418a01 .sd.evts=E5E1 .sc.flg=0x0006103d .sc.app=0x7fdd1ce08090 .sc.evts=S5 .subs=(nil)
h1c=0x7fdd1cdac120 h1c.flg=0x80000900 .sub=0 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts=M5M1 .task=0x7fdd1cdac240 .exp=4s
co1=0x7fdd1cdabf50 ctrl=abns_stream xprt=RAW mux=H1 data=STRM target=SERVER:0x440f2c00
flags=0x000c0300 fd=28689 fd.state=1122 updt=1 fd.tmask=0x80000
req=0x7fdd1ce080b8 (f=0x1840000 an=0x8000 tofwd=0 total=744)
an_exp=<NEVER> buf=0x7fdd1ce080c0 data=(nil) o=0 p=0 i=0 size=0
htx=0x7bcec0 flags=0x0 size=0 data=0 used=0 wrap=NO extra=0
res=0x7fdd1ce08100 (f=0x80008001 an=0x1400000 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdd1ce08108 data=0x7fdd1ce12f10 o=0 p=0 i=0 size=1048576
htx=0x7fdd1ce12f10 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: S:IDLE_CONNS U:IDLE_CONNS R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
call trace(15):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db245 <2a 31 f6 e8 db f6 ff ff]: debug_handler+0x65/0xca > ha_thread_dump_one
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee4564092 <83 f9 3f 76 73 48 89 d1]: libc:+0x172092
| 0x53b4d0 <45 31 e4 e8 20 9b ef ff]: http_reply_to_htx+0x8f0/0x901 > main-0x18b0
| 0x53dfef <48 89 df e8 f1 cb ff ff]: http_reply_message+0x9f/0x28b > http_reply_to_htx
| 0x53e1fe <48 89 d6 e8 52 fd ff ff]: http_reply_and_close+0x1e/0x2ff > http_reply_message
| 0x542df7 <48 89 c2 e8 e9 b3 ff ff]: http_wait_for_response+0x11e7/0x14d2 > http_reply_and_close
| 0x52514d <44 00 00 e8 c3 ca 01 00]: process_stream+0xa9d/0x3ac3 > http_wait_for_response
Thread 85: id=0x7fddececb640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/21 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=763904233647 now=763904249054 diff=15407
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 86: id=0x7fdde3767640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/22 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=813119097722 now=813119125403 diff=27681
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 87: id=0x7fdde26ce640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/23 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=792944783746 now=792944806888 diff=23142
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 88: id=0x7fdde1ecd640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/24 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=747522765930 now=747522838000 diff=72070
curr_task=0
lock_hist: R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM S:IDLE_CONNS U:IDLE_CONNS
Thread 89: id=0x7fdde0e34640 act=1 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/25 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=791604114279 now=791673755909 diff=69641630
curr_task=0x7fdcd4bd8350 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdcd4ba4510
stream=0x7fdcd4ba4510: [1754954775.020696] id=5490105 proto=tcpv4 source=179.0.119.168:5830
flags=0xc08, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=(nil), pend_pos=(nil) waiting=0 epoch=0
p_stc=1 p_req=1 p_res=0 p_prp=0
frontend=ssl (id=3 mode=http), listener=? (id=1304) addr=177.126.14.18:80
backend=hyper_vod_tvs (id=15 mode=http)
server=<none> (id=0)
task=0x7fdcd4bd8350 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=88(2/24) age=0s)
txn=0x7fdcd4ba4950 flags=0x3000 meth=1 status=-1 req.st=MSG_BODY rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdcd4b9f2c0 flags=0x00000422 ioto=1m40s state=EST endp=CONN,0x7fdcd4bd83e0,0x40404001 sub=1 rex=<NEVER> wex=<NEVER> rto=1m40s wto=<NEVER>
h1s=0x7fdcd4bd83e0 h1s.flg=0x100010 .req.state=MSG_DONE .res.state=MSG_RPBEFORE
.meth=GET status=0 .sd.flg=0x40404001 .sd.evts= .sc.flg=0x00000422 .sc.app=0x7fdcd4ba4510 .sc.evts=
.subs=0x7fdcd4b9f2d8(ev=1 tl=0x7fdcd4ba4900 tl.calls=0 tl.ctx=0x7fdcd4b9f2c0 tl.fct=sc_conn_io_cb+0)
h1c=0x7fdcd4bd81a0 h1c.flg=0x0 .sub=0 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdcd4bd82c0 .exp=<NEVER>
co0=0x7fdcd4bd7fe0 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=LISTENER:0x44346cb0
flags=0x80000300 fd=81829 fd.state=120 updt=0 fd.tmask=0x1000000
scb=0x7fdcd4b9f320 flags=0x00000821 ioto=0s state=INI endp=NONE,(nil),0x00000010 sub=0 rex=<NEVER> wex=<NEVER> rto=<NEVER> wto=<NEVER>
req=0x7fdcd4ba4538 (f=0x808001 an=0x900 tofwd=0 total=620)
an_exp=<NEVER> buf=0x7fdcd4ba4540 data=0x7fdcd4e14fc0 o=0 p=0 i=1048576 size=1048576
htx=0x7fdcd4e14fc0 flags=0x14 size=1048528 data=789 used=13 wrap=NO extra=0
res=0x7fdcd4ba4580 (f=0x80000000 an=0x0 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdcd4ba4588 data=0x7fdcd4d14fb0 o=0 p=0 i=0 size=1048576
htx=0x7fdcd4d14fb0 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP
call trace(19):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db245 <2a 31 f6 e8 db f6 ff ff]: debug_handler+0x65/0xca > ha_thread_dump_one
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee447d947 <48 83 c9 01 48 89 4a 08]: libc:+0x8b947
| 0x7fdee447ee07 <7c 24 30 e8 b9 e9 ff ff]: libc:+0x8ce07 > libc:+0x8b7c0
| 0x7fdee4480871 <4c 89 e7 e8 0f d8 ff ff]: libc:__libc_malloc+0x1c1/0x327 > libc:+0x8c080
| 0x64517a <89 c0 c3 e8 e6 07 df ff]: pool_get_from_os_noinc+0xca/0xf0 > main-0xf40
| 0x645232 <48 89 fb e8 7e fe ff ff]: pool_alloc_nocache+0x12/0xaa > pool_get_from_os_noinc
| 0x6460df <4c 89 f7 e8 41 f1 ff ff]: __pool_alloc+0x16f/0x279 > pool_alloc_nocache
| 0x63403c <00 31 f6 e8 34 1f 01 00]: task_run_applet+0x3e2dc > __pool_alloc
| 0x53ca1c <89 ef 4c 89 5d c8 ff d0]: process_stream+0x1836c
| 0x53f696 <89 4d 98 e8 fa d1 ff ff]: http_process_req_common+0xb6/0x1590 > process_stream+0x181e0
| 0x5259cf <4c 89 ff e8 11 9c 01 00]: process_stream+0x131f/0x3ac3 > http_process_req_common
Thread 90: id=0x7fddd7fff640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/26 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=804824536353 now=804824555902 diff=19549
curr_task=0
lock_hist: R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM S:IDLE_CONNS U:IDLE_CONNS
Thread 91: id=0x7fddd77fe640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/27 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=748551132297 now=748551151149 diff=18852
curr_task=0
lock_hist: R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM S:IDLE_CONNS U:IDLE_CONNS
Thread 92: id=0x7fddd6765640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/28 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=785338607506 now=785338640965 diff=33459
curr_task=0
lock_hist: S:IDLE_CONNS U:IDLE_CONNS R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 93: id=0x7fddd5f64640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/29 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=810254538169 now=810255292782 diff=754613
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 94: id=0x7fddd5763640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/30 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=788750995060 now=788751061569 diff=66509
curr_task=0
lock_hist: R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM S:IDLE_CONNS U:IDLE_CONNS
Thread 95: id=0x7fddcbfff640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/31 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=771327164114 now=771327312763 diff=148649
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 96: id=0x7fddcb7fe640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/32 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=744260082753 now=744260100499 diff=17746
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:PATEXP U:PATEXP W:LBPRM U:LBPRM S:IDLE_CONNS U:IDLE_CONNS
Thread 97: id=0x7fddcaffd640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/33 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=739821709946 now=739822600382 diff=890436
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 98: id=0x7fddc9f64640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/34 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=821193427128 now=821193508608 diff=81480
curr_task=0
lock_hist: R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM S:IDLE_CONNS U:IDLE_CONNS
Thread 99: id=0x7fddc8ecb640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/35 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=754032330565 now=754032390837 diff=60272
curr_task=0
lock_hist: U:LBPRM S:IDLE_CONNS R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM locked: IDLE_CONNS(S)
Thread 100: id=0x7fddb7767640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/36 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=747341611370 now=747341631824 diff=20454
curr_task=0
lock_hist: R:STK_TABLE U:STK_TABLE R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 101: id=0x7fddb6f66640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/37 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=780391871636 now=780391925816 diff=54180
curr_task=0
lock_hist: S:IDLE_CONNS U:IDLE_CONNS R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 102: id=0x7fddb5ecd640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/38 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=746156629622 now=746157388206 diff=758584
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 103: id=0x7fddb4e34640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/39 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=806842231349 now=806842242819 diff=11470
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 104: id=0x7fdda7767640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/40 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=766189999551 now=766190016972 diff=17421
curr_task=0
lock_hist: S:IDLE_CONNS U:IDLE_CONNS R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 105: id=0x7fdd9ffff640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/41 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=797428474161 now=797428519417 diff=45256
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:PATEXP U:PATEXP W:LBPRM U:LBPRM S:IDLE_CONNS U:IDLE_CONNS
Thread 106: id=0x7fdda66ce640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/42 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=852656015063 now=852656063917 diff=48854
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 107: id=0x7fdda5635640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/43 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=775424737167 now=775424746482 diff=9315
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 108: id=0x7fdd9f7fe640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/44 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=763661402097 now=763661447351 diff=45254
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 109: id=0x7fdd9effd640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/45 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=776274456762 now=776274926477 diff=469715
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 110: id=0x7fdd9df64640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/46 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=776531964961 now=776532110742 diff=145781
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 111: id=0x7fdd87fff640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/47 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=785846019086 now=785846069779 diff=50693
curr_task=0
lock_hist: S:IDLE_CONNS U:IDLE_CONNS R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 112: id=0x7fdd877fe640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/48 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=763541145414 now=763541198938 diff=53524
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 113: id=0x7fdd86765640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/49 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=785688682029 now=785688733117 diff=51088
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 114: id=0x7fdd85f64640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/50 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=787836785461 now=787836840432 diff=54971
curr_task=0
lock_hist: S:IDLE_CONNS U:IDLE_CONNS R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 115: id=0x7fdd85763640 act=1 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/51 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=783417248606 now=783544907999 diff=127659393
curr_task=0x7fdc74d2f680 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdc74d2f7e0
stream=0x7fdc74d2f7e0: [1754954774.841242] id=5490116 proto=tcpv4 source=187.19.251.132:18479
flags=0xc08, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=(nil), pend_pos=(nil) waiting=0 epoch=0
p_stc=1 p_req=1 p_res=0 p_prp=0
frontend=ssl (id=3 mode=http), listener=? (id=1330) addr=177.126.14.18:80
backend=hyper_vod_tvs (id=15 mode=http)
server=<none> (id=0)
task=0x7fdc74d2f680 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=114(2/50) age=0s)
txn=0x7fdc74d2fc20 flags=0x3000 meth=1 status=-1 req.st=MSG_BODY rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdc74c31070 flags=0x00000422 ioto=1m40s state=EST endp=CONN,0x7fdc74d2f710,0x40404001 sub=1 rex=<NEVER> wex=<NEVER> rto=1m40s wto=<NEVER>
h1s=0x7fdc74d2f710 h1s.flg=0x100010 .req.state=MSG_DONE .res.state=MSG_RPBEFORE
.meth=GET status=0 .sd.flg=0x40404001 .sd.evts= .sc.flg=0x00000422 .sc.app=0x7fdc74d2f7e0 .sc.evts=
.subs=0x7fdc74c31088(ev=1 tl=0x7fdc74d2fbd0 tl.calls=0 tl.ctx=0x7fdc74c31070 tl.fct=sc_conn_io_cb+0)
h1c=0x7fdc74d2f4d0 h1c.flg=0x0 .sub=0 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdc74d2f5f0 .exp=<NEVER>
co0=0x7fdc74d2f3e0 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=LISTENER:0x4434b770
flags=0x80000300 fd=81773 fd.state=20 updt=0 fd.tmask=0x4000000000000
scb=0x7fdc74c31010 flags=0x00000821 ioto=0s state=INI endp=NONE,(nil),0x00000010 sub=0 rex=<NEVER> wex=<NEVER> rto=<NEVER> wto=<NEVER>
req=0x7fdc74d2f808 (f=0x808001 an=0x900 tofwd=0 total=429)
an_exp=<NEVER> buf=0x7fdc74d2f810 data=0x7fdc74f37350 o=0 p=0 i=1048576 size=1048576
htx=0x7fdc74f37350 flags=0x14 size=1048528 data=600 used=14 wrap=NO extra=0
res=0x7fdc74d2f850 (f=0x80000000 an=0x0 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdc74d2f858 data=0x7fdc74e37340 o=0 p=0 i=0 size=1048576
htx=0x7fdc74e37340 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP
call trace(19):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db245 <2a 31 f6 e8 db f6 ff ff]: debug_handler+0x65/0xca > ha_thread_dump_one
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee447d947 <48 83 c9 01 48 89 4a 08]: libc:+0x8b947
| 0x7fdee447ee07 <7c 24 30 e8 b9 e9 ff ff]: libc:+0x8ce07 > libc:+0x8b7c0
| 0x7fdee4480871 <4c 89 e7 e8 0f d8 ff ff]: libc:__libc_malloc+0x1c1/0x327 > libc:+0x8c080
| 0x64517a <89 c0 c3 e8 e6 07 df ff]: pool_get_from_os_noinc+0xca/0xf0 > main-0xf40
| 0x645232 <48 89 fb e8 7e fe ff ff]: pool_alloc_nocache+0x12/0xaa > pool_get_from_os_noinc
| 0x6460df <4c 89 f7 e8 41 f1 ff ff]: __pool_alloc+0x16f/0x279 > pool_alloc_nocache
| 0x63403c <00 31 f6 e8 34 1f 01 00]: task_run_applet+0x3e2dc > __pool_alloc
| 0x53ca1c <89 ef 4c 89 5d c8 ff d0]: process_stream+0x1836c
| 0x53f696 <89 4d 98 e8 fa d1 ff ff]: http_process_req_common+0xb6/0x1590 > process_stream+0x181e0
| 0x5259cf <4c 89 ff e8 11 9c 01 00]: process_stream+0x131f/0x3ac3 > http_process_req_common
Thread 116: id=0x7fdd84f62640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/52 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=729859134202 now=729859168205 diff=34003
curr_task=0
lock_hist: R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM S:IDLE_CONNS U:IDLE_CONNS
Thread 117: id=0x7fdd7ffff640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/53 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=832635221505 now=832635283664 diff=62159
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 118: id=0x7fdd7f7fe640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/54 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=762551041809 now=762551075330 diff=33521
curr_task=0
lock_hist: R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:LISTENER U:LISTENER
Thread 119: id=0x7fdd7effd640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/55 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=797396104292 now=797396162945 diff=58653
curr_task=0
lock_hist: R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM S:IDLE_CONNS U:IDLE_CONNS
Thread 120: id=0x7fdd7e7fc640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/56 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=778611732402 now=778611949296 diff=216894
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 121: id=0x7fdd7dffb640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/57 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=755781850824 now=755781865947 diff=15123
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:STK_TABLE U:STK_TABLE R:STK_SESS U:STK_SESS
Thread 122: id=0x7fdd7d7fa640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/58 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=788214580632 now=788214589158 diff=8526
curr_task=0
lock_hist: R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM S:IDLE_CONNS U:IDLE_CONNS
Thread 123: id=0x7fdd7cff9640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/59 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=796318834272 now=796318841874 diff=7602
curr_task=0
lock_hist: R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM S:IDLE_CONNS U:IDLE_CONNS
Thread 124: id=0x7fdd7c7f8640 act=1 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/60 stuck=0 prof=0 harmless=0 isolated=0
cpu_ns: poll=793444384968 now=793476069324 diff=31684356
curr_task=0x7fdd28bac650 (task) calls=1 last=0
fct=0x5246b0(process_stream+0) ctx=0x7fdd28bac7b0
stream=0x7fdd28bac7b0: [1754954775.212025] id=5490141 proto=tcpv4 source=189.13.169.215:60684
flags=0xc08, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=(nil), pend_pos=(nil) waiting=0 epoch=0
p_stc=1 p_req=1 p_res=0 p_prp=0
frontend=ssl (id=3 mode=http), listener=? (id=1339) addr=177.126.14.18:80
backend=hyper_vod_tvs (id=15 mode=http)
server=<none> (id=0)
task=0x7fdd28bac650 (state=0x01 nice=0 calls=1 rate=0 exp=<NEVER> tid=123(2/59) age=0s)
txn=0x7fdd28d73a60 flags=0x3000 meth=1 status=-1 req.st=MSG_BODY rsp.st=MSG_RPBEFORE req.f=0x4c rsp.f=0x00
scf=0x7fdd28c0a660 flags=0x00000422 ioto=1m40s state=EST endp=CONN,0x7fdd28bac6e0,0x40404001 sub=1 rex=<NEVER> wex=<NEVER> rto=1m40s wto=<NEVER>
h1s=0x7fdd28bac6e0 h1s.flg=0x100010 .req.state=MSG_DONE .res.state=MSG_RPBEFORE
.meth=GET status=0 .sd.flg=0x40404001 .sd.evts= .sc.flg=0x00000422 .sc.app=0x7fdd28bac7b0 .sc.evts=
.subs=0x7fdd28c0a678(ev=1 tl=0x7fdd28bacba0 tl.calls=0 tl.ctx=0x7fdd28c0a660 tl.fct=sc_conn_io_cb+0)
h1c=0x7fdd28bac4a0 h1c.flg=0x0 .sub=0 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .evts= .task=0x7fdd28bac5c0 .exp=<NEVER>
co0=0x7fdd28bac2e0 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=LISTENER:0x4434d150
flags=0x80000300 fd=81860 fd.state=120 updt=0 fd.tmask=0x800000000000000
scb=0x7fdd28c0a600 flags=0x00000821 ioto=0s state=INI endp=NONE,(nil),0x00000010 sub=0 rex=<NEVER> wex=<NEVER> rto=<NEVER> wto=<NEVER>
req=0x7fdd28bac7d8 (f=0x808001 an=0x900 tofwd=0 total=546)
an_exp=<NEVER> buf=0x7fdd28bac7e0 data=0x7fdd28f12fd0 o=0 p=0 i=1048576 size=1048576
htx=0x7fdd28f12fd0 flags=0x14 size=1048528 data=717 used=14 wrap=NO extra=0
res=0x7fdd28bac820 (f=0x80000000 an=0x0 tofwd=0 total=0)
an_exp=<NEVER> buf=0x7fdd28bac828 data=0x7fdd28c12d70 o=0 p=0 i=0 size=1048576
htx=0x7fdd28c12d70 flags=0x0 size=1048528 data=0 used=0 wrap=NO extra=0
lock_hist: R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP
call trace(22):
| 0x5d82c0 <fc ff ff e8 60 da e5 ff]: ha_dump_backtrace+0x70/0x341 > main-0xb80
| 0x5db245 <2a 31 f6 e8 db f6 ff ff]: debug_handler+0x65/0xca > ha_thread_dump_one
| 0x7fdee45e33a0 <00 00 00 00 0f 1f 40 00]: libpthread:+0x123a0
| 0x7fdee447d947 <48 83 c9 01 48 89 4a 08]: libc:+0x8b947
| 0x7fdee447ee07 <7c 24 30 e8 b9 e9 ff ff]: libc:+0x8ce07 > libc:+0x8b7c0
| 0x7fdee4480871 <4c 89 e7 e8 0f d8 ff ff]: libc:__libc_malloc+0x1c1/0x327 > libc:+0x8c080
| 0x64517a <89 c0 c3 e8 e6 07 df ff]: pool_get_from_os_noinc+0xca/0xf0 > main-0xf40
| 0x645232 <48 89 fb e8 7e fe ff ff]: pool_alloc_nocache+0x12/0xaa > pool_get_from_os_noinc
| 0x6460df <4c 89 f7 e8 41 f1 ff ff]: __pool_alloc+0x16f/0x279 > pool_alloc_nocache
| 0x57a735 <89 45 00 e8 3b b8 0c 00]: main+0x143e95 > __pool_alloc
| 0x57961f <7b 18 4c 89 ee ff 51 08]: sample_process_cnv+0x6f/0x90
| 0x57969c <4c 89 ef e8 14 ff ff ff]: sample_process+0x5c/0xa7 > sample_process_cnv
| 0x691822 <4c 89 f7 e8 1e 7e ee ff]: acl_exec_cond+0x132/0x249 > sample_process
| 0x53c98d <89 7d c8 e8 63 4d 15 00]: process_stream+0x182dd > acl_exec_cond
| 0x53f696 <89 4d 98 e8 fa d1 ff ff]: http_process_req_common+0xb6/0x1590 > process_stream+0x181e0
| 0x5259cf <4c 89 ff e8 11 9c 01 00]: process_stream+0x131f/0x3ac3 > http_process_req_common
Thread 125: id=0x7fdd7bff7640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/61 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=771690736491 now=771690745962 diff=9471
curr_task=0
lock_hist: S:IDLE_CONNS U:IDLE_CONNS R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 126: id=0x7fdd7b7f6640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/62 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=809973285186 now=809973296880 diff=11694
curr_task=0
lock_hist: R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM S:IDLE_CONNS U:IDLE_CONNS
Thread 127: id=0x7fdd7aff5640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/63 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=814019953273 now=814019963904 diff=10631
curr_task=0
lock_hist: R:PATEXP U:PATEXP W:LBPRM U:LBPRM R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Thread 128: id=0x7fdd7a7f4640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
2/64 stuck=0 prof=0 harmless=1 isolated=0
cpu_ns: poll=747152412141 now=747152581345 diff=169204
curr_task=0
lock_hist: W:LBPRM U:LBPRM R:LISTENER U:LISTENER R:PATEXP U:PATEXP W:LBPRM U:LBPRM
Hint: when reporting this bug to developers, please check if a core file was
produced, open it with 'gdb', issue 't a a bt full', check that the
output does not contain sensitive data, then join it with the bug report.
For more info, please see https://github.com/haproxy/haproxy/issues/2374
Oh thank you, this is super helpful, it's even the first time the warnings are so useful! So it shows that you have 3 threads (13, 42, 48) which are all blocked at exactly the same place and were caught not making any progress (up to one second for 48): cli_io_handler+0x2da75.
Could you please open the binary with gdb and issue "list *(cli_io_handler+0x2da75)" ? It should show the exact line number. I hope it will not report the contents of an inline function!
I suspect it could be related to the monitoring scripts, maybe they're causing some locking somewhere that hinders the other threads as well, because we've seen 111, 113 and a few others block in totally random areas, indicating they've possibly experienced a long pause. I suspect that a few dump sessions are hindering each other and causing other threads to sometimes wait long on a lock held by one of them. But with a bit of luck we'll manage to figure what exactly it is above. The nice thing is that most CLI commands can yield so we may possibly consider replacing a lock by a retryable trylock, and/or perform less operations per loop if other threads are hindered too often in a single loop.
Hi Willy,
Here you go:
(gdb) list *(cli_io_handler+0x2da75)
0x5c90c5 is in sc_conn_send (src/stconn.c:1718).
1713 s->txn->flags &= ~TX_L7_RETRY;
1714 else {
1715 memcpy(b_orig(&s->txn->l7_buffer),
1716 b_orig(&oc->buf),
1717 b_size(&oc->buf));
1718 s->txn->l7_buffer.head = co_data(oc);
1719 b_add(&s->txn->l7_buffer, co_data(oc));
1720 }
1721
1722 }
As for the locking, maybe an RCU? After all, whatever precision one might expect from a highly-parallel (and possibly loaded) system is just a faint moment in time.
Thanks Ricardo. So it's not even a locking issue, and there's nothing waiting here. I simply don't understand how it's possible to stay stick here. It's just reading oc->output. Most likely the reason is in fact that it's the return pointer of the memcpy() call then. Oh, I'm just realizing that you're running with 1 MB buffers:
buf=0x7fddd8bc11e0 data=0x7fddd8f3a200 o=160 p=160 i=1048416 size=1048576
So actually that memcpy() clearly is the reason why we're always there. It doesn't necessarily mean that no progress was made, but it may mean that every time we interrupt this stuck thread it's statistically there. I don't see why it would loop there. Normally the loops is limited by the configured number of retries. Strangely I'm always seeing conn_retries=0 so it would indicate it's the very first request and it's not looping either. Also the stream's calls count is 1, proving that we didn't leave the function to re-enter it.
I hardly see why such a request would make no progress though. And I think it's physically impossible to spend one second copying 1 MB of data. So the only other explanation I'm seeing now is that we're looping in process_stream() and constantly hitting that place. Maybe due to a synchronous connect() error or such a thing that wouldn't be properly handled for example. I think I'll first need to be able to reproduce it here as it will otherwise be very difficult to analyse on production or on a core.
BTW regarding locking, RCU etc, we use very little locking nowadays (that's how we can scale with the number of threads), because sessions and streams are affine to a thread once created and do not migrate. Some places still do use some locking, but that wouldn't necessarily be addressed by RCU as it's more a matter of consistency between operations than data modifications, and these ones are not on critical paths (e.g. server going down etc).
But it's true that my first reflex when seeing a wathdog is still to think about a potential locking issue!
Oh I had not noticed, but all stuck threads are the http client! Maybe we're facing a special case related to DNS resolution. Do you know what calls it ? I guess it's from Lua ?
Oh I had not noticed, but all stuck threads are the http client! Maybe we're facing a special case related to DNS resolution. Do you know what calls it ? I guess it's from Lua ?
Yes, calling/called from a Lua script, performing some complex L7+ access control.
Logged stderr shows a number of network errors; summarized below (each one happened at a call into the Redis Lua module):
[ALERT] (1859117) : Lua function 'redis-valida': burst timeout.
[ALERT] (1859117) : Lua function 'redis-valida': runtime error: /usr/share/lua/5.4/redis.lua:309: connection error: connection closed. from [C]: in global 'error', /usr/share/lua/5.4/redis.lua:851: in field 'error', /usr/share/lua/5.4/redis.lua:309: in field 'read', /usr/share/lua/5.4/redis.lua:315: in field 'read', /usr/share/lua/5.4/redis.lua:425: in method 'expire', /etc/haproxy/valida_conjunto.lua:77: in function line 4.
[ALERT] (1859117) : Lua function 'redis-valida': runtime error: /usr/share/lua/5.4/redis.lua:309: connection error: connection closed. from [C]: in global 'error', /usr/share/lua/5.4/redis.lua:851: in field 'error', /usr/share/lua/5.4/redis.lua:309: in field 'read', /usr/share/lua/5.4/redis.lua:315: in field 'read', /usr/share/lua/5.4/redis.lua:425: in method 'sadd', /etc/haproxy/valida_conjunto.lua:73: in function line 4.
[ALERT] (1859117) : Lua function 'redis-valida': runtime error: /usr/share/lua/5.4/redis.lua:309: connection error: connection closed. from [C]: in global 'error', /usr/share/lua/5.4/redis.lua:851: in field 'error', /usr/share/lua/5.4/redis.lua:309: in field 'read', /usr/share/lua/5.4/redis.lua:315: in field 'read', /usr/share/lua/5.4/redis.lua:425: in method 'scard', /etc/haproxy/valida_conjunto.lua:31: in function line 4.
[ALERT] (1859117) : Lua function 'redis-valida': runtime error: /usr/share/lua/5.4/redis.lua:309: connection error: connection closed. from [C]: in global 'error', /usr/share/lua/5.4/redis.lua:851: in field 'error', /usr/share/lua/5.4/redis.lua:309: in field 'read', /usr/share/lua/5.4/redis.lua:315: in field 'read', /usr/share/lua/5.4/redis.lua:425: in method 'smembers', /etc/haproxy/valida_conjunto.lua:52: in function line 4.
[ALERT] (1859117) : Lua function 'redis-valida': runtime error: /usr/share/lua/5.4/redis.lua:309: connection error: connection closed. from [C]: in global 'error', /usr/share/lua/5.4/redis.lua:851: in field 'error', /usr/share/lua/5.4/redis.lua:309: in field 'read', /usr/share/lua/5.4/redis.lua:315: in field 'read', /usr/share/lua/5.4/redis.lua:425: in method 'srem', /etc/haproxy/valida_conjunto.lua:71: in function line 4.
[ALERT] (1859117) : Lua function 'redis-valida': runtime error: /usr/share/lua/5.4/redis.lua:309: connection error: connection closed. from [C]: in global 'error', /usr/share/lua/5.4/redis.lua:851: in field 'error', /usr/share/lua/5.4/redis.lua:309: in field 'read', /usr/share/lua/5.4/redis.lua:315: in field 'read', /usr/share/lua/5.4/redis.lua:425: in method 'ttl', /etc/haproxy/valida_conjunto.lua:76: in function line 4.
Strangely, none is reported at the single place that could trigger a DNS resolution.
(The lack of timestamping (even better a full seconds.nanoseconds) makes it a bit had to guess how disperse or concentrated were those network errors---there is, though, an intermix of [ALERT] and thread dumping, so they could be very well entangled just as the output suggests.)
Indeed, errors could be more helpful if they would come with a timestamp :-/ OK so at least now we know that something (possibly network errors) can trigger a misbehavior at connect time when using http client from Lua. I'm not particularly fluent in these areas unfortunately :-/
@wtarreau we were able to bypass lua completely (inspired by your recent post on the Mailing list with João Morais for stick-tables updates across peers), as we were able to configure 25 peers for stick-table replication and aggregation (with variables + add). So we're good for now (and frankly a way better solution!).
We used to use Lua for that....but indeed those errors indicate that some network condition under heavy load caused problems.
One of the side benefits of forcing high load on this new kind of setup is uncovering these corner cases, I suppose :-)
We'll know tonight when the traffic increases. But most likely that was the culprit given the errors of Lua timeout + nothing that useful on the stderr output.
I'd say the "trigger" nor "the culprit" since the culprit is still a bug somewhere in the code, that we'll need to find out. But at least now we have some elements, knowing that it can be triggered from Lua under high loads. Will not be easy though!