grpc icon indicating copy to clipboard operation
grpc copied to clipboard

SEGFAULT in grpc_core::RefcountedMdBase::hash

Open soumik0108 opened this issue 1 year ago • 3 comments

What version of gRPC and what language are you using?

1.39

What operating system (Linux, Windows,...) and version?

BSD12

What runtime / compiler are you using (e.g. python version or version of gcc)

clang version 10.0.1

What did you do?

initalmetadata array passed to grpc_call_start_batch. shutdown of cq usinggrpc_completion_queue_shutdown. draining cq using grpc_completion_queue_next. Deletion of grpc call usinggrpc_call_unref Deltion of Initial metadata array using grpc_metadata_array_destroy.

What did you expect to see?

No Core

What did you see instead?

Segfault and valgrind pointing Invalid read

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

BT

#0 grpc_core::RefcountedMdBase::hash (this=0x5a5a5a58) at ../../../../../../../../src/external/apache2/grpc/dist/src/core/lib/transport/metadata.h:239 #1 grpc_mdelem_unref (gmd=...) at ../../../../../../../../src/external/apache2/grpc/dist/src/core/lib/transport/metadata.h:390 #2 grpc_metadata_batch_destroy (batch=0x2461b44) at ../../../../../../../../src/external/apache2/grpc/dist/src/core/lib/transport/metadata_batch.cc:93 #3 0x0188ac43 in post_batch_completion (bctl=0x2462060) at ../../../../../../../../src/external/apache2/grpc/dist/src/core/lib/surface/call.cc:1189 #4 finish_batch_step (bctl=0x2462060) at ../../../../../../../../src/external/apache2/grpc/dist/src/core/lib/surface/call.cc:1254 #5 0x0188a4ac in receiving_initial_metadata_ready (bctlp=0x2462060, error=0x0) at ../../../../../../../../src/external/apache2/grpc/dist/src/core/lib/surface/call.cc:1526 #6 0x018322bd in grpc_core::Closure::Run (location=..., closure=0x2461ef8, error=0x0) at ../../../../../../../../src/external/apache2/grpc/dist/src/core/lib/iomgr/closure.h:245 #7 grpc_core::ClientChannel::CallData::RecvInitialMetadataReadyForConfigSelectorCommitCallback (arg=0x2461f90, error=0x0) at ../../../../../../../../src/external/apache2/grpc/dist/src/core/ext/filters/client_channel/client_channel.cc:2287 #8 0x018d8a90 in grpc_core::Closure::Run (location=..., closure=0x2462024, error=0x0) at ../../../../../../../../src/external/apache2/grpc/dist/src/core/lib/iomgr/closure.h:245 #9 recv_initial_metadata_ready (user_data=0x3c9e098, error=0x0) at ../../../../../../../../src/external/apache2/grpc/dist/src/core/ext/filters/http/client/http_client_filter.cc:205 #10 0x018d93a3 in grpc_core::Closure::Run (location=..., closure=0x3c9e1b0, error=0x0) at ../../../../../../../../src/external/apache2/grpc/dist/src/core/lib/iomgr/closure.h:245 #11 grpc_core::(anonymous namespace)::CallData::OnRecvInitialMetadataReady (arg=0x3c9e2d0, error=0x0) at ../../../../../../../../src/external/apache2/grpc/dist/src/core/ext/filters/http/message_compress/message_decompress_filter.cc:165 #12 0x019b65f2 in exec_ctx_run (closure=0x2, error=0x0) at ../../../../../../../../src/external/apache2/grpc/dist/src/core/lib/iomgr/exec_ctx.cc:43 #13 grpc_core::ExecCtx::Flush (this=0x25a5f00) at ../../../../../../../../src/external/apache2/grpc/dist/src/core/lib/iomgr/exec_ctx.cc:165 #14 0x01999c10 in grpc_core::Executor::RunClosures (executor_name=0x16d0961 "default-executor", list=...) at ../../../../../../../../src/external/apache2/grpc/dist/src/core/lib/iomgr/executor.cc:130 #15 0x01999f17 in grpc_core::Executor::ThreadMain (arg=0x21ba060) at ../../../../../../../../src/external/apache2/grpc/dist/src/core/lib/iomgr/executor.cc:248 #16 0x019b608a in grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void ()(void), void*, bool*, grpc_core::Thread::Options const&)::{lambda(void*)#1}::operator()(void*) const (this=, v=) at ../../../../../../../../src/external/apache2/grpc/dist/src/core/lib/gprpp/thd_posix.cc:140 #17 grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void ()(void), void*, bool*, grpc_core::Thread::Options const&)::{lambda(void*)#1}::__invoke(void*) (v=0x21e0200) at ../../../../../../../../src/external/apache2/grpc/dist/src/core/lib/gprpp/thd_posix.cc:110 #18 0x01f372f2 in thread_start (curthread=0x2169300) at /src/lib/libthr/thread/thr_create.c:292

Valgrind

==23804== Invalid read of size 4 ==23804== at 0x8A2FD67: grpc_metadata_batch_destroy(grpc_metadata_batch*) (metadata_batch.cc:93) ==23804== by 0x894BC42: post_batch_completion (call.cc:1189) ==23804== by 0x894BC42: finish_batch_step(batch_control*) (call.cc:1254) ==23804== by 0x894B4AB: receiving_initial_metadata_ready(void*, grpc_error*) (call.cc:1526) ==23804== by 0x88F32BC: Run (closure.h:245) ==23804== by 0x88F32BC: grpc_core::ClientChannel::CallData::RecvInitialMetadataReadyForConfigSelectorCommitCallback(void*, grpc_error*) (client_channel.cc:2287) ==23804== by 0x8999A8F: Run (closure.h:245) ==23804== by 0x8999A8F: recv_initial_metadata_ready(void*, grpc_error*) (http_client_filter.cc:205) ==23804== by 0x899A3A2: Run (closure.h:245) ==23804== by 0x899A3A2: grpc_core::(anonymous namespace)::CallData::OnRecvInitialMetadataReady(void*, grpc_error*) (message_decompress_filter.cc:165) ==23804== by 0x8A775F1: exec_ctx_run (exec_ctx.cc:43) ==23804== by 0x8A775F1: grpc_core::ExecCtx::Flush() (exec_ctx.cc:165) ==23804== by 0x8A5AC0F: grpc_core::Executor::RunClosures(char const*, grpc_closure_list) (executor.cc:130) ==23804== by 0x8A5AF16: grpc_core::Executor::ThreadMain(void*) (executor.cc:248) ==23804== by 0x8A77089: operator() (thd_posix.cc:140) ==23804== by 0x8A77089: grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void ()(void), void*, bool*, grpc_core::Thread::Options con st&)::{lambda(void*)#1}::__invoke(void*) (thd_posix.cc:110) ==23804== by 0x91212F1: ??? (in /packages/mnt/os-libs-compat32-12/usr/lib32/libthr.so.3) ==23804== Address 0xba6c914 is 36 bytes inside a block of size 156 free'd ==23804== at 0x73EBCC8: free (src/paul-floyd-317-fbsd12/valgrind-freebsd/coregrind/m_replacemalloc/vg_replace_malloc.c:602) ==23804== by 0x8A8AEB7: gpr_free (alloc.cc:52) ==23804== by 0x89C4617: grpc_metadata_array_destroy (metadata_array.cc:35) ==23804== by 0x7F15EF1: GrpcCall::~GrpcCall() (GrpcCall.cc:1344) ==23804== by 0x91212F1: ??? (in /packages/mnt/os-libs-compat32-12/usr/lib32/libthr.so.3) ==23804== Block was alloc'd at ==23804== at 0x73EAB42: malloc (src/paul-floyd-317-fbsd12/valgrind-freebsd/coregrind/m_replacemalloc/vg_replace_malloc.c:311) ==23804== by 0x8A8AE5C: gpr_malloc (alloc.cc:32) ==23804== by 0x7F13A66: GrpcCall::GrpcSendInitialMetadata(grpc_client_info_s*, std::__1::map<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator<char

, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits, std::__1::all ocator > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const, std::__1::basic_string<char, std ::__1::char_traits, std::__1::allocator > > > >*) (GrpcCall.cc:303)

Anything else we should know about your project / environment?

soumik0108 avatar Jan 17 '24 22:01 soumik0108

Clarification of the statement is not enough. You should explain more and after that we think more on the problem.

AbhiSharmaNIT avatar Jan 20 '24 17:01 AbhiSharmaNIT

The version that you are using is pretty old. The code has seen a lot of changes and I don't think that branch is going to see any fixes. Can you please using the latest version and see if the issue still persists?

Also, if you can provide the exact repro that you used, that would make it easier for us to debug.

(I'll also note that you're using BSD. I'm not sure if that might be a factor here but just something to keep in mind.)

yashykt avatar Jan 23 '24 20:01 yashykt

Ok I will try to reproduce on 1.52 with asan enabled image.

soumik0108 avatar Jan 25 '24 08:01 soumik0108

More than 30 days have passed since label "disposition/requires reporter action" was added. Closing this issue. Please feel free to re-open/create a new issue if this is still relevant.

grpc-bot avatar Feb 23 '24 00:02 grpc-bot