error serializing large responses on client with remote pipe

[zachrybaker]

see issue 48 for context.

Granted this case is probably not the norm:

Remote piping (for now, going to be local soon for security reasons). The server code is 100% synchronous re-platformed mainframe code, that I bridge the async gap via Task.FromResult(data);

When I replay two week's worth of API calls, 0.19% of the calls - 1283 of them - failed on the client side (server sent "normal data"). These calls can be replayed on a local pipe successfully between a .Net6.0 server and a .Net6.0 client. Replaying them in isolation using the same client, to a remote .Net6.0 server, however, they will consistently fail, regardless of load.

A sampling of these, when traced back to the request that originated the failure, all happened to be deserializing a large-ish payload, say around 200KB when serialized to JSON. When pulling back the same collection but with a smaller size, no problem deserializing.

So I think there is something that gets fouled with the bytes payload when it is a remote pipe that isn't happening when it is not remote. Seems unlikely that I have stumbled upon a framing issue, but I guess that is possible.

System.AggregateException HResult=0x80131500 Message=One or more errors occurred. (Protocol message contained an invalid tag (zero).) Source=System.Private.CoreLib StackTrace: at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) at System.Threading.Tasks.Task1.get_Result() at GrpcDotNetNamedPipes.NamedPipeChannel.BlockingUnaryCall[TRequest,TResponse](Method2 method, String host, CallOptions callOptions, TRequest request) in C:\code\patterns\GrpcDotNetNamedPipes\NamedPipeChannel.cs:line 93 at Grpc.Core.Interceptors.InterceptingCallInvoker.<BlockingUnaryCall>b__3_0[TRequest,TResponse](TRequest req, ClientInterceptorContext2 ctx) at Grpc.Core.ClientBase.ClientBaseConfiguration.ClientBaseConfigurationInterceptor.BlockingUnaryCall[TRequest,TResponse](TRequest request, ClientInterceptorContext2 context, BlockingUnaryCallContinuation2 continuation) at Grpc.Core.Interceptors.InterceptingCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method2 method, String host, CallOptions options, TRequest request) at Tamus.Famis.Protos.Shadow.Iasys.IasysSvc.IasysSvcClient.GetSysTableDictionaryCollection(GetSysTableDictionaryRqst request, CallOptions options) in C:\code\patterns\s\Tamus.Famis.Protos.Client\obj\Debug\netstandard2.0\protos\shadow\IasysGrpc.cs:line 1153 at Tamus.Famis.Protos.Shadow.Iasys.IasysSvc.IasysSvcClient.GetSysTableDictionaryCollection(GetSysTableDictionaryRqst request, Metadata headers, Nullable1 deadline, CancellationToken cancellationToken) in C:\code\patterns\s\Tamus.Famis.Protos.Client\obj\Debug\netstandard2.0\protos\shadow\IasysGrpc.cs:line 1142 at Tamus.Integration.Shadow.ShadowServerService.<>c__DisplayClass632_0.<GetSysTableDictionaryCollection>b__0() in C:\code\patterns\s\Tamus.Integration.Shadow.Client\ShadowServerService_Calls\ShadowServerService.Iasys.cs:line 57 at Tamus.Famis.Protos.Client.Helpers.ServiceExtensions.ThenRun[O](StartPipeFn InitializeFn, Func1 do) in C:\code\patterns\s\Tamus.Famis.Protos.Client\Helpers\ServiceExtensions.cs:line 52 at Tamus.Integration.Shadow.ShadowServerService.RunIasys[O](Func1 do) in C:\code\patterns\s\Tamus.Integration.Shadow.Client\ShadowServerService.cs:line 73 at Tamus.Integration.Shadow.ShadowServerService.GetSysTableDictionaryCollection(GetSysTableDictionaryRqst request) in C:\code\patterns\s\Tamus.Integration.Shadow.Client\ShadowServerService_Calls\ShadowServerService.Iasys.cs:line 57 at Famis.DataLayer.Sys.TableDictionaryCollection.Fill(String campusCd) in C:\code\patterns\s\Tamus.NaturalMigration.Iasys.Code.ShadowSubprograms\ZNNPDICT.TableDictionary.cs:line 86

This exception was originally thrown at this call stack: [External Code] Tamus.Famis.Protos.Shadow.Iasys.IasysSvc.__Helper_DeserializeMessage<T>(Grpc.Core.DeserializationContext, Google.Protobuf.MessageParser<T>) in IasysGrpc.cs Tamus.Famis.Protos.Shadow.Iasys.IasysSvc..cctor.AnonymousMethod__136_23(Grpc.Core.DeserializationContext) in IasysGrpc.cs GrpcDotNetNamedPipes.Internal.Helpers.SerializationHelpers.Deserialize<T>(Grpc.Core.Marshaller<T>, byte[]) in SerializationHelpers.cs GrpcDotNetNamedPipes.Internal.MessageReader<TMessage>.Current.get() in MessageReader.cs GrpcDotNetNamedPipes.Internal.MessageReader<TMessage>.ReadNextMessage.AnonymousMethod__0() in MessageReader.cs

Inner Exception 1: InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).