grpc
Grpc.Core.Internal.AsyncCall<>.UnaryCall hangs (malformed call status?)
Ver 1.45 (1.46 code is the same) The symptom is hanging in Grpc.Core.Internal.AsyncCall<>.UnaryCall on return unaryResponseTcs.Task.GetAwaiter().GetResult(); line.
The stack trace example is:
[Waiting on Async Operation, double-click or press enter to view Async Call Stacks] Grpc.Core.dll!Grpc.Core.Internal.AsyncCall<GenericRequest, GenericResponse>.UnaryCall(GenericRequest msg) Line 116 C# Grpc.Core.dll!Grpc.Core.Calls.BlockingUnaryCall<GenericRequest, GenericResponse>(Grpc.Core.CallInvocationDetails<GenericRequest, GenericResponse> call, GenericRequest req) Line 25 C# Grpc.Core.dll!Grpc.Core.DefaultCallInvoker.BlockingUnaryCall<GenericRequest, GenericResponse>(Grpc.Core.Method<GenericRequest, GenericResponse> method, string host, Grpc.Core.CallOptions options, GenericRequest request) Line 16 C# Grpc.Core.Api.dll!Grpc.Core.Interceptors.InterceptingCallInvoker.BlockingUnaryCall.AnonymousMethod__3_0<GenericRequest, MGenericResponse>(GenericRequest req, Grpc.Core.Interceptors.ClientInterceptorContext<GenericRequest, GenericResponse> ctx) Line 51 C# Grpc.Core.Api.dll!Grpc.Core.ClientBase.ClientBaseConfiguration.ClientBaseConfigurationInterceptor.BlockingUnaryCall<GenericRequest, GenericResponse>(GenericRequest request, Grpc.Core.Interceptors.ClientInterceptorContext<GenericRequest, GenericResponse> context, Grpc.Core.Interceptors.Interceptor.BlockingUnaryCallContinuation<GenericRequest, GenericResponse> continuation) Line 174 C# Grpc.Core.Api.dll!Grpc.Core.Interceptors.InterceptingCallInvoker.BlockingUnaryCall<GenericRequest, GenericResponse>(Grpc.Core.Method<GenericRequest, GenericResponse> method, string host, Grpc.Core.CallOptions options, GenericRequest request) Line 48 C#
It is happening periodically, the problem how I think is in the next code in of the UnaryCall method:
try
{
using (profiler.NewScope("AsyncCall.UnaryCall.HandleBatch"))
{
HandleUnaryResponse(success, ctx.GetReceivedStatusOnClient(), ctx.GetReceivedMessageReader(), ctx.GetReceivedInitialMetadata());
}
}
catch (Exception e)
{
Logger.Error(e, "Exception occurred while invoking completion delegate.");
**!!! here should be "throw;" !!!**
}
Without that line, in case of HandleUnaryResponse error, unaryResponseTcs will never get neither error, nor value and will wait infinitely.
Also, a test could be added to ensure, the unaryResponseTcs state is not WaitingForActivationcan prior to call GetResult().
@jtattermusch, @hassox, @beccasaurus, @timburks, could you please fix the issue asap? :) It is critical in terms of nature of the bug - it hangs threads. I already tested the fix on production, and it behaves as expected.
The causing the exception message:
System.ArgumentException: The output char buffer is too small to contain the decoded characters, encoding 'Unicode (UTF-8)' fallback 'System.Text.DecoderReplacementFallback'.
Parameter name: chars
at System.Text.Encoding.ThrowCharsOverflow()
at System.Text.Encoding.ThrowCharsOverflow(DecoderNLS decoder, Boolean nothingDecoded)
at System.Text.UTF8Encoding.GetChars(Byte* bytes, Int32 byteCount, Char* chars, Int32 charCount, DecoderNLS baseDecoder)
at System.Text.UTF8Encoding.GetChars(Byte* bytes, Int32 byteCount, Char* chars, Int32 charCount)
at Grpc.Core.Api.Utils.EncodingExtensions.GetString(Encoding encoding, Byte* source, Int32 byteCount)
at Grpc.Core.Api.Utils.EncodingExtensions.GetString(Encoding encoding, IntPtr ptr, Int32 len)
at Grpc.Core.Internal.BatchContextSafeHandle.**GetReceivedStatusOnClient**()
at Grpc.Core.Internal.AsyncCall'2.UnaryCall(TRequest msg)
at Grpc.Core.Calls.BlockingUnaryCall[TRequest,TResponse](CallInvocationDetails'2 call, TRequest req)
at Grpc.Core.DefaultCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method'2 method, String host, CallOptions options, TRequest request)
at Grpc.Core.Interceptors.InterceptingCallInvoker.<BlockingUnaryCall>b__3_0[TRequest,TResponse](TRequest req, ClientInterceptorContext'2 ctx)
at Grpc.Core.ClientBase.ClientBaseConfiguration.ClientBaseConfigurationInterceptor.BlockingUnaryCall[TRequest,TResponse](TRequest request, ClientInterceptorContext'2 context, BlockingUnaryCallContinuation'2 continuation)
at Grpc.Core.Interceptors.InterceptingCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method'2 method, String host, CallOptions options, TRequest request)
FTR Looks like this is the code location the user is referring to: https://github.com/grpc/grpc/blob/53d69cc581c5b7305708587f4f1939278477c28a/src/csharp/Grpc.Core/Internal/AsyncCall.cs#L124
The causing the exception message:
System.ArgumentException: The output char buffer is too small to contain the decoded characters, encoding 'Unicode (UTF-8)' fallback 'System.Text.DecoderReplacementFallback'. Parameter name: chars at System.Text.Encoding.ThrowCharsOverflow() at System.Text.Encoding.ThrowCharsOverflow(DecoderNLS decoder, Boolean nothingDecoded) at System.Text.UTF8Encoding.GetChars(Byte* bytes, Int32 byteCount, Char* chars, Int32 charCount, DecoderNLS baseDecoder) at System.Text.UTF8Encoding.GetChars(Byte* bytes, Int32 byteCount, Char* chars, Int32 charCount) at Grpc.Core.Api.Utils.EncodingExtensions.GetString(Encoding encoding, Byte* source, Int32 byteCount) at Grpc.Core.Api.Utils.EncodingExtensions.GetString(Encoding encoding, IntPtr ptr, Int32 len) at Grpc.Core.Internal.BatchContextSafeHandle.**GetReceivedStatusOnClient**() at Grpc.Core.Internal.AsyncCall'2.UnaryCall(TRequest msg) at Grpc.Core.Calls.BlockingUnaryCall[TRequest,TResponse](CallInvocationDetails'2 call, TRequest req) at Grpc.Core.DefaultCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method'2 method, String host, CallOptions options, TRequest request) at Grpc.Core.Interceptors.InterceptingCallInvoker.<BlockingUnaryCall>b__3_0[TRequest,TResponse](TRequest req, ClientInterceptorContext'2 ctx) at Grpc.Core.ClientBase.ClientBaseConfiguration.ClientBaseConfigurationInterceptor.BlockingUnaryCall[TRequest,TResponse](TRequest request, ClientInterceptorContext'2 context, BlockingUnaryCallContinuation'2 continuation) at Grpc.Core.Interceptors.InterceptingCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method'2 method, String host, CallOptions options, TRequest request)
Ok, that's quite odd. Can you share the code that reproduces the above problem?
It looks like decoding the call's status (likely the "details" field) is failing - what is the contents of the call's status? https://github.com/grpc/grpc/blob/53d69cc581c5b7305708587f4f1939278477c28a/src/csharp/Grpc.Core/Internal/BatchContextSafeHandle.cs#L90 AFAIK this is a situation that should never happen (receiving malformed details message)
@jtattermusch, I see the issue on production with bad Wi-Fi network access few times per a day, i.e. I cannot reproduce it via a test program. However, as I wrote, because of this catch (Exception e) without throw (or unaryResponseTcs.SetException) inside, the code hangs forever. We patch the code locally by adding throw, it stopped hanging and we were able to see this stacktrace, which I added previously. So the exception is happening in the GetReceivedStatusOnClient method, not sure why, but I think a broken connection is the reason.
Also, imho, as an additional protection from possible similar issues, this unaryResponseTcs TaskCompletionSource status could be checked prior to GetResult(), for a case of future possible changes, preventing setting a result/exception to the unaryResponseTcs.
Anyway, adding throw and releasing a new fix version - all we desire now :)
I've created a Pull Request to handle the exception in AsyncCall.UnaryCall().
PR https://github.com/grpc/grpc/pull/30429
Rather than just throwing the exception I have created an RcpException and set the exception in unaryResponseTcs.
I was unable to reproduce the original corrupt status code. Looking at the marshalling code it is unclear how the UTF-8 decoding would fail.
There is no obvious way to create a unit test with a mock failure and the failure happens at quite a low level. I was able to test locally by temporarily modifying AsyncCall.UnaryCall() and explicitly throwing an exception and checking the behaviour of both the fixed and non-fixed code.
I commented on the proposed fix and I think we need to understand more why exactly does the status from ctx.GetReceivedStatusOnClient() seem to be malformed.
@zhenomansh can you dump the contents of the "detailsLength" bytes starting at "detailsPtr" in the invocation of
https://github.com/grpc/grpc/blob/c34b22eda3e5770f6dc970885feaa9d06eae5d76/src/csharp/Grpc.Core/Internal/BatchContextSafeHandle.cs#L90 when the exception System.ArgumentException: The output char buffer is too small to contain the decoded characters, encoding 'Unicode (UTF-8)' fallback 'System.Text.DecoderReplacementFallback'. happens?
Also, what is the "status"?
We need to understand why the status delivered by the internal logic seems corrupt (it should not be) in order to come up with the right fix.
There are two issues to address:
- why does the decoding of the status fail?
- how to handle a low level error and report the error rather than causing the call to hang
I've tried to address the second issue, albeit without the clean up. I could add some resource clean up in a finally block.
For the first issue (why does the decoding fail?) - code inspection looks like it shouldn't happen - in EncodingExtensions.GetString the size of the required buffer is obtained with Encoding.GetCharCount() before calling Encoding.GetChars() so there shouldn't be an exception with The output char buffer is too small to contain the decoded characters.
public static unsafe string GetString(this Encoding encoding, byte* source, int byteCount)
{
if (byteCount == 0) return ""; // most callers will have already checked, but: make sure
// allocate a right-sized string and decode into it
int charCount = encoding.GetCharCount(source, byteCount);
string s = new string('\0', charCount);
fixed (char* cPtr = s)
{
encoding.GetChars(source, byteCount, cPtr, charCount);
}
return s;
}
The fallback System.Text.DecoderReplacementFallback() is being called meaning the bytes aren't valid UTF-8 - so unless the replacement string has been defined elsewhere in the application that should just mean that a single char '?' is being substituted.
Since we are dealing with unmanaged code in an error situation it is always possible that something elsewhere in the application is overwriting memory while we are trying to decode it.
@jtattermusch, finally I have managed to collect the info from production :)
public ClientSideStatus GetReceivedStatusOnClient()
{
StatusCode statusCode = StatusCode.Unknown;
int detailsLengthInt = -123;
try
{
statusCode = Native.grpcsharp_batch_context_recv_status_on_client_status(this);
UIntPtr detailsLength;
IntPtr detailsPtr = Native.grpcsharp_batch_context_recv_status_on_client_details(this, out detailsLength);
detailsLengthInt = (int)detailsLength.ToUInt32();
string details = MarshalUtils.PtrToStringUTF8(detailsPtr, detailsLengthInt);
string debugErrorString = Marshal.PtrToStringAnsi(Native.grpcsharp_batch_context_recv_status_on_client_error_string(this));
var status = new Status(statusCode, details, debugErrorString != null ? new CoreErrorDetailException(debugErrorString) : null);
IntPtr metadataArrayPtr = Native.grpcsharp_batch_context_recv_status_on_client_trailing_metadata(this);
var metadata = MetadataArraySafeHandle.ReadMetadataFromPtrUnsafe(metadataArrayPtr);
return new ClientSideStatus(status, metadata);
}
catch (Exception ex)
{
throw new Exception($"GetReceivedStatusOnClient failed with statusCode={statusCode}, detailsLength={detailsLengthInt}, {ex}");
}
}
And the result is always:
GetReceivedStatusOnClient failed with statusCode=Unknown, detailsLength=14, System.ArgumentException: The output char buffer is too small to contain the decoded characters, encoding 'Unicode (UTF-8)' fallback 'System.Text.DecoderReplacementFallback'.
Parameter name: chars
at System.Text.Encoding.ThrowCharsOverflow()
at System.Text.Encoding.ThrowCharsOverflow(DecoderNLS decoder, Boolean nothingDecoded)
at System.Text.UTF8Encoding.GetChars(Byte* bytes, Int32 byteCount, Char* chars, Int32 charCount, DecoderNLS baseDecoder)
at System.String.CreateStringFromEncoding(Byte* bytes, Int32 byteLength, Encoding encoding)
at System.Text.Encoding.GetString(Byte* bytes, Int32 byteCount)
at Grpc.Core.Api.Utils.EncodingExtensions.GetString(Encoding encoding, IntPtr ptr, Int32 len)
at Grpc.Core.Internal.BatchContextSafeHandle.GetReceivedStatusOnClient()
at Grpc.Core.Internal.BatchContextSafeHandle.GetReceivedStatusOnClient()
at Grpc.Core.Internal.AsyncCall`2.UnaryCall(TRequest msg)
at Grpc.Core.Calls.BlockingUnaryCall[TRequest,TResponse](CallInvocationDetails`2 call, TRequest req)
at Grpc.Core.DefaultCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method`2 method, String host, CallOptions options, TRequest request)
at Grpc.Core.Interceptors.InterceptingCallInvoker.<BlockingUnaryCall>b__3_0[TRequest,TResponse](TRequest req, ClientInterceptorContext`2 ctx)
at Grpc.Core.ClientBase.ClientBaseConfiguration.ClientBaseConfigurationInterceptor.BlockingUnaryCall[TRequest,TResponse](TRequest request, ClientInterceptorContext`2 context, BlockingUnaryCallContinuation`2 continuation)
at Grpc.Core.Interceptors.InterceptingCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method`2 method, String host, CallOptions options, TRequest request)
@zhenomansh thank you. Is there any chance you could modify the code for GetReceivedStatusOnClient to give a few more details in the exception message? Things that would be useful are:
- Int64 value of detailsPtr , e.g.
detailsPtr.toInt64()- so can check it isn't zero or some other strange value etc - value of
detailsPtr.Size- so we know if it is 4 (32 bit) or 8 (64 bit) - hex string of dump of bytes that detailsPtr is pointing at, e.g. in the case above it would be the 14 bytes as hex. I might be able to help you with some code if you are not sure how to do that.
@zhenomansh I've created a draft PR with additional debug info in GetReceivedStatusOnClient()
https://github.com/grpc/grpc/pull/31343
public ClientSideStatus GetReceivedStatusOnClient()
{
StatusCode statusCode = StatusCode.Unknown;
int detailsLengthInt = -123;
IntPtr detailsPtr = IntPtr.Zero;
try
{
statusCode = Native.grpcsharp_batch_context_recv_status_on_client_status(this);
UIntPtr detailsLength;
detailsPtr = Native.grpcsharp_batch_context_recv_status_on_client_details(this, out detailsLength);
detailsLengthInt = (int)detailsLength.ToUInt32();
string details = MarshalUtils.PtrToStringUTF8(detailsPtr, (int)detailsLength.ToUInt32());
string debugErrorString = Marshal.PtrToStringAnsi(Native.grpcsharp_batch_context_recv_status_on_client_error_string(this));
var status = new Status(Native.grpcsharp_batch_context_recv_status_on_client_status(this), details, debugErrorString != null ? new CoreErrorDetailException(debugErrorString) : null);
IntPtr metadataArrayPtr = Native.grpcsharp_batch_context_recv_status_on_client_trailing_metadata(this);
var metadata = MetadataArraySafeHandle.ReadMetadataFromPtrUnsafe(metadataArrayPtr);
return new ClientSideStatus(status, metadata);
}
catch (Exception ex)
{
long detailsPtrLong = detailsPtr.ToInt64();
String detailsHex = PtrToHexString(detailsPtr, detailsLengthInt);
throw new Exception($"GetReceivedStatusOnClient failed with " +
$"statusCode={statusCode}, detailsLength={detailsLengthInt}, ptr={detailsPtrLong}, ptr size={IntPtr.Size}" +
$"hex={detailsHex}\nException: {ex}");
}
}
private static unsafe string PtrToHexString(IntPtr ptr, int length)
{
if (ptr == IntPtr.Zero)
{
return "<pointer is zero>";
}
if (length <= 0)
{
return $"<length {length}>";
}
StringBuilder sb = new StringBuilder();
if (length > 1024)
{
sb.Append("<truncated> ");
length = 1024;
}
byte* bytes = (byte*)ptr.ToPointer();
for (int i=0; i<length; i++)
{
byte b = bytes[i];
sb.AppendFormat("{0:x2} ", b);
}
return sb.ToString();
}
@tonydnewell here it is:
statusCode=Unknown, detailsLength=14, detailsPtrInt=210233972, detailsMessage=0x18 B4 9B 0A 38 2F FF 07 40 4D 5C 00 90 4D statusCode=Unknown, detailsLength=14, detailsPtrInt=212854564, detailsMessage=0x50 9E C3 16 D8 E4 BB 16 20 C8 D6 16 06 00 statusCode=Unknown, detailsLength=14, detailsPtrInt=227006628, detailsMessage=0xD0 C9 90 12 38 0F 6D 0A 98 C3 A2 12 FF FF statusCode=Unknown, detailsLength=14, detailsPtrInt=210497156, detailsMessage=0x18 9F 94 00 90 BD 63 05 90 47 39 00 E0 47
@zhenomansh thank you for the update.
The data does look totally random - not UTF-8 or ASCII. This might be because memory is being re-used after freeing or some other memory corruption. I'll continue to look at the grpc core code to see if there is anything obvious, but without being able to easily reproduce the problem it is going to be difficult to diagnose.
Does your application have any other native or unmanaged code? Is an error being sent by the server? If so what should it be? Or is this a client-side only error?
@tonydnewell, I tried to test the data and the Encoding.GetString method works fine with data examples above, however it doesn't look like a text. So, may be, the problem is in pointers?
The app is quite big, but it doesn't use unmanaged code at all. It has several links to external dlls, however, they all work for many years and no memory problems have been met so far.
The bug is happening on different PC for many clients.
As I noticed, it happens mostly in weak wi-fi zones.
I understand that it is a goal to get to the truth, however, the real problem of Grpc.Core is the hanging scenario, which we overrode by a custom build and we wait for the proper fix to be included to the official nuget.
I will try to collect more info if you are interested, but is there a chance that your pending fix will be accepted by @jtattermusch? :)
It is already several months passed, and the official version still hangs.
#30429
@tonydnewell, I tried to test the data and the Encoding.GetString method works fine with data examples above, however it doesn't look like a text. So, may be, the problem is in pointers?
The app is quite big, but it doesn't use unmanaged code at all. It has several links to external dlls, however, they all work for many years and no memory problems have been met so far.
The bug is happening on different PC for many clients.
As I noticed, it happens mostly in weak wi-fi zones.
I understand that it is a goal to get to the truth, however, the real problem of Grpc.Core is the hanging scenario, which we overrode by a custom build and we wait for the proper fix to be included to the official nuget.
I will try to collect more info if you are interested, but is there a chance that your pending fix will be accepted by @jtattermusch? :)
The fix will be accepted as soon as we have a reasonable fix and once we have a better understanding of what's actually going on. The problem with the current "fix" is that it simply add some extra handling where in theory we should never see a problem - and blindly accepting a fix that gets rid of a suspiciously odd situation by papering over it IMHO isn't a very good strategy. We need to understand why data data gets corrupted (it at least seems like it does) in your app because this could be a problem in the native C-core layer (and if there's e.g. a use-after-free bug in the native layer we should really fix the actual rootcause, since no amount of papering over it is going to help much).
Does that sound alright to you?
It is already several months passed, and the official version still hangs.
@jtattermusch, I believe there are two different problems.
One is, probably, related to the native source code, or other low-level part (driver, Windows etc.) However we see the problem quite rare, and in general it doesn't affect the reliability of communication, as any communication could become unstable due to wiring/signal reasons.
The second one, is the critical mistake in this Grpc.Core.Internal.AsyncCall<>.UnaryCall, which doesn't set a state to the unaryResponseTcs, which hangs the executing thread forever.
We don't know a reason of the native mistake, and we even don't have ideas how to figure that out at the moment, but it is not as important, comparing to the second critical problem, which makes using of the official grpc nuget impossible for us.
So, sorry for opposing, but postponing the existing fix for the critical problem until we find a rare bug in the native code, doesn't sound "alright" to me :)
Applying this critical part fix won't hide the native code problem, so we would be able to continue the investigation.
That is why I really see no obstacles in giving a green light to the pending pool request from @tonydnewell :)
@zhenomansh I understand your frustration. I will revisit my fix to add the missing resource clean up but that won't be this week. Also not being able reproduce the problem makes it difficult for me to test any fix.
We do need to get a better understanding of what is happening. Are you able to describe in high level terms how your application uses gRPC and the environment it is running it? Anything that helps reproduce or narrow down the problem would be helpful.
- You talk about weak wi-fi zones - how does that manifest itself? Connections lost? Slow?
- Are the unary calls long lived? E.g. send a request and wait a long time for a response? Are they frequent (several a second) or infrequent (one now and then)?
- Are you creating a new connection (Channel) for each call?
- Do you have other gRPC calls that work OK? (streaming calls etc)
- Is the application multithreaded with gRPC calls made from different threads?
- What is the size of the messages being passed to and fro?
- Is the server seeing any errors?
@tonydnewell, sure :)
Our solution is about collecting telemetry on ore mining sites, underground, around a world. gRPC is used as the only low-level transport, wrapped by our high-level code. We have plenty of calls to microservices from different client apps which are desktop apps or on-board apps on underground equipment (trucks, loaders etc.). All Windows-based (.Net Framework 4.7). We never saw the issue in inter-service communications or between desktop client app and a service where is a reliable network connection persists. It happens only when an underground unit communicating to services and that is why there is an assumption that it relates to a weak wi-fi signal. A number of calls from an underground unit to services could be 10000-100000 per a day. The issue happens on some units once per a week, on some several times a day, on some never and I would say it is the mining site-specific thing. We see that in most cases the issue happens in a period of connection problems, but it also could happen when an equipment is online. All sites equipment have similar on-board PC with same set of drivers, so, the problem could be specific to the standard PC configuration.
- You talk about weak wi-fi zones - how does that manifest itself? Connections lost? Slow? Equipment moves in/outside the wi-fi coverage are and the connection could be unstable, also there cold be switching between wi-fi access points.
- Are the unary calls long lived? E.g. send a request and wait a long time for a response? Are they frequent (several a second) or infrequent (one now and then)? some calls could wait up to a minute, but in most cases it takes milliseconds, in average it makes a call about once per 2 seconds (comparing to services, which could make 1000 calls per a second and work fine).
- Are you creating a new connection (Channel) for each call? We re-creating a channel only after communication errors, as there is another issue: When PC restores the connection to intranet, gRPC calls could fail on requests up to several minutes, reporting "Unavailable" state. Re-creation of a channel fix the problem and enables communication immediately.
- Do you have other gRPC calls that work OK? (streaming calls etc) We only use this type of calls, async and sync, streaming is not used. They all works well except of this rare cases as explained above.
- Is the application multithreaded with gRPC calls made from different threads? Yes, it is, calls could be done simultaneously from different threads, but less likely to the same service/endpoint.
- What is the size of the messages being passed to and from? Request is about 2-2.5 KB, response is in the range of 1-20KB.
- Is the server seeing any errors? No, services don't report errors in case of issue.
I've created a new PR to fix the hanging in a different way: https://github.com/grpc/grpc/pull/31450
This does not address why the status is corrupt when reading it. I cannot reproduce that. It looks like the memory is being overwritten while the UTF-8 bytes are being decoded.
This fix should stop any hanging in a unary call by wrapping the reading of the status (and metadata) and not throwing an exception. A new error status is produced instead containing the original exception and also a message is logged.
It was not possible to produce a unit test. The only way I could test this was to alter the code for BatchContextSafeHandle.GetReceivedStatusOnClient() and throw an exception:
// TODO REMOVE - FOR TESTING ONLY
if (details.Equals("Stream removed"))
{
// Fake an exception
throw new System.ArgumentException("forced exception for Stream removed");
}
and kill the server during a unary call to get the "Stream removed" error.
I reviewed #31450 and came up with a proposal with what change needs to be made to work around (=not fix) the issue. See https://github.com/grpc/grpc/pull/31450#pullrequestreview-1209793631
@zhenomansh Grpc 2.46.6 has been released. It contains the workaround. Are you able to test it?
Thank you guys! It will take some time, but I will definitely provide feedback.
More than 30 days have passed since label "disposition/requires reporter action" was added. Closing this issue. Please feel free to re-open/create a new issue if this is still relevant.
@tonydnewell, the production testing showed the fix is working. Guys, thank you again.