grpc icon indicating copy to clipboard operation
grpc copied to clipboard

grpc crash at LoadBalancedCall::PickSubchannelImpl

Open henryluoheng opened this issue 1 year ago • 6 comments

What version of gRPC and what language are you using?

grpc1.62.2 C++

What operating system (Linux, Windows,...) and version?

Windows 10

What runtime / compiler are you using (e.g. python version or version of gcc)

msvc visual studio 2022

What did you do?

It's a crash reported by our users, I'm not able to find out what exactly he did when the crash happens, and I have not found a way to reproduce it.

What did you expect to see?

grpc works correctly

What did you see instead?

by debugging the dmp file I got, this is the calling stack

image

Anything else we should know about your project / environment?

Sorry for the limit of info I provided, I am not able discover more currently. I just try to seek for some possibility if you team can find any clue about this crash, if not, that's ok. Within a month after we release our newest application version with updating grpc to 1.62.2, three users reported crash on this calling stack.

henryluoheng avatar Jul 17 '24 09:07 henryluoheng

I looked through the code and did not see anything suspicious. There is little we can do with the limited info provided. We have not seen other reports even though the code in question have not changed a lot in the last several releases and is covered with tests on supported platforms.

Please try to get more information from your clients. See this guide - https://github.com/grpc/grpc/blob/v1.62.x/TROUBLESHOOTING.md. Please note that you need to follow the guide from 1.62 branch as there were significant changes in logging since 1.62. This reference provides more details on setting up tracing: https://github.com/grpc/grpc/blob/v1.62.x/doc/environment_variables.md

eugeneo avatar Jul 17 '24 14:07 eugeneo

I looked through the code and did not see anything suspicious. There is little we can do with the limited info provided. We have not seen other reports even though the code in question have not changed a lot in the last several releases and is covered with tests on supported platforms.

Please try to get more information from your clients. See this guide - https://github.com/grpc/grpc/blob/v1.62.x/TROUBLESHOOTING.md. Please note that you need to follow the guide from 1.62 branch as there were significant changes in logging since 1.62. This reference provides more details on setting up tracing: https://github.com/grpc/grpc/blob/v1.62.x/doc/environment_variables.md

OK,thank you very much. I'll follow the guide and have a try. And if I get more information, I'll update.

henryluoheng avatar Jul 18 '24 02:07 henryluoheng

not sure if this info helps but seemingly the same crash happens on firebase-ios-sdk as well. https://github.com/firebase/firebase-ios-sdk/issues/12968

Kaz-flex avatar Aug 08 '24 17:08 Kaz-flex

@Kaz-flex is this failing on Mac or iOS? Is there any chance to use a build with debug symbols or tracing?

eugeneo avatar Aug 08 '24 18:08 eugeneo

It's happening on iOS, Unfortunately I did not reproduce it manually but we observed it through crash tracking service. You can find stack trace of all the threads here if that's something that can help. https://github.com/firebase/firebase-ios-sdk/issues/13459

Kaz-flex avatar Aug 08 '24 19:08 Kaz-flex

@eugeneo I'm trying to release our app with GRPC_VERBOSITY=debug but not sure what to set for GRPC_TRACE because all would be just too many logs. Do you have any suggestions on what to set for GRPC_TRACE and GRPC_VERBOSITY?

Kaz-flex avatar Oct 11 '24 22:10 Kaz-flex

My recommendation would be - GRPC_TRACE=call,channel,client_channel,client_channel_call,client_channel_lb_call,pick_first GRPC_VERBOSITY=DEBUG (or initialize absl logging and set verbosity that way.)

yashykt avatar Nov 05 '24 20:11 yashykt

More than 30 days have passed since label "disposition/requires reporter action" was added. Closing this issue. Please feel free to re-open/create a new issue if this is still relevant.

grpc-bot avatar Dec 05 '24 21:12 grpc-bot