seg fault errors during span extraction

[Dhivyakumaresan]

Hi,

we are receiving below seg faults. i have given the gdb backtrace, info args, info register, disas output from core dump. could someone look at and provide some insights why the error occurs. we suspect the error is due to missing span headers or it could be something else.

we are using Datadog tracer.

Jul 18 18:32:45 kong-enterprise-data-i-0d8579c86c063a722 kernel: nginx[14525]: segfault at 4 ip 00007f9342372a1b sp 00007fff545d4e20 error 4 in libdd_opentracing_plugin.so[7f9342320000+f0000]

gdb outputs:

[``` dh-api-admin@kong-enterprise-data-i-0d8579c86c063a722 cores]$ sudo gdb /usr/local/openresty/nginx/sbin/nginx ./core.14525 GNU gdb (GDB) Red Hat Enterprise Linux 8.0.1-36.amzn2.0.1 Copyright (C) 2017 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/local/openresty/nginx/sbin/nginx...done. [New LWP 14525] [New LWP 14526] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `nginx: worker process '. Program terminated with signal SIGSEGV, Segmentation fault. #0 std::_Function_handler<opentracing::v3::expected<void, std::error_code>(opentracing::v3::string_view, opentracing::v3::string_view), datadog::opentracing::SpanContext::deserialize(std::shared_ptr<const datadog::opentracing::Logger>, const opentracing::v3::TextMapReader&, const datadog::opentracing::HeadersImpl&)::<lambda(opentracing::v3::string_view, opentracing::v3::string_view)> >::_M_invoke(const std::_Any_data &, opentracing::v3::string_view &&, opentracing::v3::string_view &&) (__functor=..., __args#0=..., __args#1=...) at /usr/include/c++/7/bits/std_function.h:302

warning: Source file is more recent than executable. 302 std::forward<_ArgTypes>(__args)...); [Current thread is 1 (Thread 0x7f9344d4f780 (LWP 14525))] Missing separate debuginfos, use: debuginfo-install kong-enterprise-edition-2.8.1.1-1.noarch

(gdb) bt #0 std::_Function_handler<opentracing::v3::expected<void, std::error_code>(opentracing::v3::string_view, opentracing::v3::string_view), datadog::opentracing::SpanContext::deserialize(std::shared_ptr<const datadog::opentracing::Logger>, const opentracing::v3::TextMapReader&, const datadog::opentracing::HeadersImpl&)::<lambda(opentracing::v3::string_view, opentracing::v3::string_view)> >::_M_invoke(const std::_Any_data &, opentracing::v3::string_view &&, opentracing::v3::string_view &&) (__functor=..., __args#0=..., __args#1=...) at /usr/include/c++/7/bits/std_function.h:302 #1 0x0000000000552950 in std::function<opentracing::v3::expected<void, std::error_code> (opentracing::v3::string_view, opentracing::v3::string_view)>::operator()(opentracing::v3::string_view, opentracing::v3::string_view) const (__args#1=..., __args#0=..., this=0x7fff545d4f80) at /usr/include/c++/4.8.2/functional:2471 #2 ngx_opentracing::(anonymous namespace)::NgxHeaderCarrierReader::__lambda0::operator() (header=..., __closure=) at /tmp/nginx-opentracing/opentracing/src/extract_span_context.cpp:36 #3 ngx_opentracing::for_each<ngx_table_elt_t, ngx_opentracing::(anonymous namespace)::NgxHeaderCarrierReader::ForeachKey(std::function<opentracing::v3::expected(opentracing::v3::string_view, opentracing::v3::string_view)>) const::__lambda0> ( f=..., list=...) at /tmp/nginx-opentracing/opentracing/src/utility.h:74 #4 ngx_opentracing::(anonymous namespace)::NgxHeaderCarrierReader::ForeachKey(std::function<opentracing::v3::expected<void, std::error_code>(opentracing::v3::string_view, opentracing::v3::string_view)>) const (this=this@entry=0x7fff545d5120, f=...) at /tmp/nginx-opentracing/opentracing/src/extract_span_context.cpp:37 #5 0x00007f9342370326 in datadog::opentracing::SpanContext::deserialize (logger=std::shared_ptr (count 4, weak 0) 0x3e65310, reader=..., headers_impl=...) at /root/dd-opentracing-cpp/src/propagation.cpp:469 #6 0x00007f9342371b6e in datadog::opentracing::SpanContext::deserialize (logger=std::shared_ptr (count 4, weak 0) 0x3e65310, reader=..., styles=std::set with 2 elements = {...}) at /root/dd-opentracing-cpp/src/propagation.cpp:438 #7 0x00007f934239c1da in datadog::opentracing::Tracer::Extract (this=0x3e53ca0, reader=...) at /root/dd-opentracing-cpp/src/tracer.cpp:368 #8 0x0000000000552a3a in ngx_opentracing::extract_span_context (tracer=..., request=0x3cf4ab0) at /tmp/nginx-opentracing/opentracing/src/extract_span_context.cpp:52 #9 0x00000000005581ff in ngx_opentracing::RequestTracing::RequestTracing (this=0x443ae30, request=0x3cf4ab0, core_loc_conf=, loc_conf=, parent_span_context=0x0) at /tmp/nginx-opentracing/opentracing/src/request_tracing.cpp:101 #10 0x00000000005563da in __gnu_cxx::new_allocator<ngx_opentracing::RequestTracing>::construct<ngx_opentracing::RequestTracing, ngx_http_request_s*&, ngx_http_core_loc_conf_s*&, ngx_opentracing::opentracing_loc_conf_t*&> (__p=, this=0x3df76f0) at /usr/include/c++/4.8.2/ext/new_allocator.h:120 #11 std::allocator_traits<std::allocator<ngx_opentracing::RequestTracing> >::_S_construct<ngx_opentracing::RequestTracing, ngx_http_request_s*&, ngx_http_core_loc_conf_s*&, ngx_opentracing::opentracing_loc_conf_t*&> (__p=, __a=...) at /usr/include/c++/4.8.2/bits/alloc_traits.h:254 #12 std::allocator_traits<std::allocator<ngx_opentracing::RequestTracing> >::construct<ngx_opentracing::RequestTracing, ngx_http_request_s*&, ngx_http_core_loc_conf_s*&, ngx_opentracing::opentracing_loc_conf_t*&> (__p=, __a=...) at /usr/include/c++/4.8.2/bits/alloc_traits.h:393 #13 std::vector<ngx_opentracing::RequestTracing, std::allocator<ngx_opentracing::RequestTracing> >::_M_emplace_back_aux<ngx_http_request_s*&, ngx_http_core_loc_conf_s*&, ngx_opentracing::opentracing_loc_conf_t*&> (this=this@entry=0x3df76f0) at /usr/include/c++/4.8.2/bits/vector.tcc:408 #14 0x0000000000555f73 in std::vector<ngx_opentracing::RequestTracing, std::allocator<ngx_opentracing::RequestTracing> >::emplace_back<ngx_http_request_s*&, ngx_http_core_loc_conf_s*&, ngx_opentracing::opentracing_loc_conf_t*&> (this=0x3df76f0) at /usr/include/c++/4.8.2/bits/vector.tcc:101 #15 ngx_opentracing::OpenTracingContext::OpenTracingContext (this=0x3df76f0, request=0x3cf4ab0, core_loc_conf=0x1fc4cb8, loc_conf=0x1fc63e0) at /tmp/nginx-opentracing/opentracing/src/opentracing_context.cpp:21 #16 0x00000000005552ce in ngx_opentracing::on_enter_block (request=0x3cf4ab0) at /tmp/nginx-opentracing/opentracing/src/opentracing_handler.cpp:39 #17 0x000000000046508c in ngx_http_core_rewrite_phase (r=0x3cf4ab0, ph=) at src/http/ngx_http_core_module.c:932 #18 0x0000000000460a85 in ngx_http_core_run_phases (r=r@entry=0x3cf4ab0) at src/http/ngx_http_core_module.c:878 #19 0x0000000000460b6c in ngx_http_handler (r=r@entry=0x3cf4ab0) at src/http/ngx_http_core_module.c:861 #20 0x000000000046b583 in ngx_http_process_request (r=r@entry=0x3cf4ab0) at src/http/ngx_http_request.c:2106 #21 0x000000000046bac1 in ngx_http_process_request_headers (rev=rev@entry=0x7f931f99d6d0) at src/http/ngx_http_request.c:1508 #22 0x000000000046be14 in ngx_http_process_request_line (rev=0x7f931f99d6d0) at src/http/ngx_http_request.c:1175 #23 0x00000000004543a7 in ngx_epoll_process_events (cycle=, timer=, flags=) at src/event/modules/ngx_epoll_module.c:901 #24 0x000000000044b673 in ngx_process_events_and_timers (cycle=cycle@entry=0x1f4ca60) at src/event/ngx_event.c:257 #25 0x0000000000452722 in ngx_worker_process_cycle (cycle=cycle@entry=0x1f4ca60, data=data@entry=0x0) at src/os/unix/ngx_process_cycle.c:782 #26 0x00000000004510c0 in ngx_spawn_process (cycle=cycle@entry=0x1f4ca60, proc=proc@entry=0x4526b0 <ngx_worker_process_cycle>, data=data@entry=0x0, name=name@entry=0x5a21dd "worker process", respawn=respawn@entry=-3) at src/os/unix/ngx_process.c:199 #27 0x0000000000452b7c in ngx_start_worker_processes (cycle=cycle@entry=0x1f4ca60, n=1, type=type@entry=-3) at src/os/unix/ngx_process_cycle.c:382 #28 0x0000000000453328 in ngx_master_process_cycle (cycle=cycle@entry=0x1f4ca60) at src/os/unix/ngx_process_cycle.c:135 #29 0x000000000042afa6 in main (argc=, argv=) at src/core/nginx.c:386

(gdb) info args __functor = @0x7fff545d4f80: {_M_unused = {_M_object = 0x4443080, _M_const_object = 0x4443080, _M_function_pointer = 0x4443080, _M_member_pointer = (void (std::_Undefined_class::*)(std::_Undefined_class * const)) 0x4443080}, _M_pod_data = "\200\060D\004", '\000' <repeats 11 times>} args#0 = @0x3cf5718: {data = 0x7265737574736f68 <error: Cannot access memory at address 0x7265737574736f68>, length = 7161132899992297773} __args#1 =

(gdb) info register rax 0x4 4 rbx 0x7fff545d4e70 140734608789104 rcx 0x4 4 rdx 0x3cf5718 63919896 rsi 0x7fff545d4f80 140734608789376 rdi 0x7fff545d4e70 140734608789104 rbp 0x3cf4b20 0x3cf4b20 rsp 0x7fff545d4e20 0x7fff545d4e20 r8 0x3d6974c 64395084 r9 0xe 14 r10 0x70 112 r11 0x7f934399a5ca 140270471062986 r12 0x7fff545d4f00 140734608789248 r13 0x3d88d10 64523536 r14 0x7fff545d4f80 140734608789376 r15 0x5528d0 5581008 rip 0x7f9342372a1b 0x7f9342372a1b <std::_Function_handler<opentracing::v3::expected<void, std::error_code>(opentracing::v3::string_view, opentracing::v3::string_view), datadog::opentracing::SpanContext::deserialize(std::shared_ptr<const datadog::opentracing::Logger>, const opentracing::v3::TextMapReader&, const datadog::opentracing::HeadersImpl&)::<lambda(opentracing::v3::string_view, opentracing::v3::string_view)> >::_M_invoke(const std::_Any_data &, opentracing::v3::string_view &&, opentracing::v3::string_view &&)+11> eflags 0x10202 [ IF RF ] cs 0x33 51 ss 0x2b 43 ds 0x0 0 es 0x0 0 fs 0x0 0 gs 0x0 0 k0 0x0 0 k1 0x0 0 k2 0x0 0 k3 0x0 0 k4 0x0 0 k5 0x0 0 k6 0x0 0 k7 0x0 0

(gdb) disas Dump of assembler code for function std::_Function_handler<opentracing::v3::expected<void, std::error_code>(opentracing::v3::string_view, opentracing::v3::string_view), datadog::opentracing::SpanContext::deserialize(std::shared_ptr<const datadog::opentracing::Logger>, const opentracing::v3::TextMapReader&, const datadog::opentracing::HeadersImpl&)::<lambda(opentracing::v3::string_view, opentracing::v3::string_view)> >::_M_invoke(const std::_Any_data &, opentracing::v3::string_view &&, opentracing::v3::string_view &&): 0x00007f9342372a10 <+0>: push %rbx 0x00007f9342372a11 <+1>: mov %rcx,%rax 0x00007f9342372a14 <+4>: mov %rdi,%rbx 0x00007f9342372a17 <+7>: sub $0x20,%rsp => 0x00007f9342372a1b <+11>: mov (%rax),%r8 0x00007f9342372a1e <+14>: mov 0x8(%rax),%r9 0x00007f9342372a22 <+18>: mov %fs:0x28,%rcx 0x00007f9342372a2b <+27>: mov %rcx,0x18(%rsp) 0x00007f9342372a30 <+32>: xor %ecx,%ecx 0x00007f9342372a32 <+34>: mov (%rsi),%rsi 0x00007f9342372a35 <+37>: mov 0x8(%rdx),%rcx 0x00007f9342372a39 <+41>: mov %rsp,%rdi 0x00007f9342372a3c <+44>: mov (%rdx),%rdx 0x00007f9342372a3f <+47>: callq 0x7f93423721d0 <datadog::opentracing::SpanContext::<lambda(opentracing::v3::string_view, opentracing::v3::string_view)>::operator()(opentracing::v3::string_view, opentracing::v3::string_view) const> 0x00007f9342372a44 <+52>: movzbl (%rsp),%eax 0x00007f9342372a48 <+56>: test %al,%al 0x00007f9342372a4a <+58>: mov %al,(%rbx) 0x00007f9342372a4c <+60>: jne 0x7f9342372a58 <std::_Function_handler<opentracing::v3::expected<void, std::error_code>(opentracing::v3::string_view, opentracing::v3::string_view), datadog::opentracing::SpanContext::deserialize(std::shared_ptr<const datadog::opentracing::Logger>, const opentracing::v3::TextMapReader&, const datadog::opentracing::HeadersImpl&)::<lambda(opentracing::v3::string_view, opentracing::v3::string_view)> >::_M_invoke(const std::_Any_data &, opentracing::v3::string_view &&, opentracing::v3::string_view &&)+72> 0x00007f9342372a4e <+62>: movdqu 0x8(%rsp),%xmm0 0x00007f9342372a54 <+68>: movups %xmm0,0x8(%rbx) 0x00007f9342372a58 <+72>: mov 0x18(%rsp),%rcx 0x00007f9342372a5d <+77>: xor %fs:0x28,%rcx 0x00007f9342372a66 <+86>: mov %rbx,%rax 0x00007f9342372a69 <+89>: jne 0x7f9342372a71 <std::_Function_handler<opentracing::v3::expected<void, std::error_code>(opentracing::v3::string_view, opentracing::v3::string_view), datadog::opentracing::SpanContext::deserialize(std::shared_ptr<const datadog::opentracing::Logger>, const opentracing::v3::TextMapReader&, const datadog::opentracing::HeadersImpl&)::<lambda(opentracing::v3::string_view, opentracing::v3::string_view)> >::_M_invoke(const std::_Any_data &, opentracing::v3::string_view &&, opentracing::v3::string_view &&)+97> 0x00007f9342372a6b <+91>: add $0x20,%rsp 0x00007f9342372a6f <+95>: pop %rbx 0x00007f9342372a70 <+96>: retq 0x00007f9342372a71 <+97>: callq 0x7f934234f1f0 __stack_chk_fail@plt End of assembler dump.

Thanks in advance

[dgoffredo]

It looks like you're using the dd-opentracing-cpp OpenTracing plugin with Kong.

The crash seems to be due to memory corruption. The std::function object that is invoked for each HTTP header in an incoming request refers to an invalid address. It's likely that the underlying issue lies elsewhere.

In order to debug this, I'd need to be able to reproduce the crash.

Is the crash happening with any regularity? If so, have you found a way to trigger the crash?