Multi-threaded TopicSession use on OSX causes unmanaged "pointer being freed was not allocated" error

[chuckb]

Actual Behavior

On OSX, when I have numerous (> 300) sessions to process and fetch them in a loop, processing them async, then receiving and processing all messages within them async, I randomly (most runs crash...only 1-2 have run to completion) get:

dotnet(79250,0x700005110000) malloc: *** error for object 0x7ff72230a010: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug

Expected Behavior

When I run the same code on Windows 10 (build 14393), I get no error and the process runs to completion (tried more than 10 runs).

Versions

• Environment

charles-benedicts-macbook:~ chuck_benedict$ dotnet --info .NET Command Line Tools (2.0.0)

Product Information: Version: 2.0.0 Commit SHA-1 hash: cdcd1928c9

Runtime Environment: OS Name: Mac OS X OS Version: 10.12 OS Platform: Darwin RID: osx.10.12-x64 Base Path: /usr/local/share/dotnet/sdk/2.0.0/

Microsoft .NET Core Shared Framework Host

Version : 2.0.0 Build : e8b8861ac7faf042c87a5c2f9f2d04c98b69f28d

• NuGet package version or commit ID:

"Microsoft.Azure.ServiceBus/1.0.0": { "dependencies": { "Microsoft.Azure.Amqp": "2.1.0", "NETStandard.Library": "2.0.0", "System.Runtime.Serialization.Xml": "4.3.0"

Additional Information

I was able to use lldb to get a complete (managed and unmanaged) stack trace of the failing thread. I have also been in communication with the team doing work on the System.Net.Security library and they tell me that SslStream (and indeed it is documented here []https://msdn.microsoft.com/en-us/library/system.net.security.sslstream(v=vs.110).aspx#Thread%20Safety) is not threadsafe for concurrent access. While I did not find any other thread in the backtrace access a SslStream, I did see re-entrant access to the System.Net.Security.SslStreamInternal.ProcessFrameBody method within the thread below.

(lldb) sos DumpStack
OS Thread Id: 0x16c6855 (38)
TEB information is not available so a stack size of 0xFFFF is assumed
Current frame: libsystem_malloc.dylib!malloc_error_break
Child-SP         RetAddr          Caller, Callee
00007000109E1F50 00007fffaa2b6f6e libsystem_malloc.dylib!free + 0x199, calling libsystem_malloc.dylib!malloc_error_break
00007000109E1FA0 00007fff9ae58b41 Security!SSLRecordServiceWriteQueueInternal + 0x80, calling Security!sslFree
00007000109E1FE0 00007fff9ac8bc3b Security!SSLServiceWriteQueue + 0xf
00007000109E1FF0 00007fff9ac8f441 Security!SSLRead + 0x99, calling Security!SSLServiceWriteQueue
00007000109E2090 00000001085e5bf4 System.Security.Cryptography.Native.Apple.dylib!AppleCryptoNative_SslRead + 0x34, calling System.Security.Cryptography.Native.Apple.dylib!SSLRead
00007000109E20D0 00000001146eeee5 (MethodDesc 0000000114ac6f88 + 0xa5 DomainBoundILStubClass.IL_STUB_PInvoke(System.Net.SafeSslHandle, Byte*, Int32, Int32 ByRef))
00007000109E2110 00000001146eeee5 (MethodDesc 0000000114ac6f88 + 0xa5 DomainBoundILStubClass.IL_STUB_PInvoke(System.Net.SafeSslHandle, Byte*, Int32, Int32 ByRef))
00007000109E2170 00000001149d318a (MethodDesc 0000000114aa0510 + 0x8a System.Net.Security.SslStreamPal.DecryptMessage(System.Net.Security.SafeDeleteContext, Byte[], Int32 ByRef, Int32 ByRef)), calling 00000001146df528 (stub for Interop+AppleCrypto.SslRead(System.Net.SafeSslHandle, Byte*, Int32, Int32 ByRef))
00007000109E21E0 00000001149c1667 (MethodDesc 000000011494ab68 + 0xd7 System.Net.Security.SecureChannel.Decrypt(Byte[], Int32 ByRef, Int32 ByRef)), calling (MethodDesc 0000000114aa0510 + 0 System.Net.Security.SslStreamPal.DecryptMessage(System.Net.Security.SafeDeleteContext, Byte[], Int32 ByRef, Int32 ByRef))
00007000109E21F0 00000001149bf69a (MethodDesc 000000011494aa88 + 0x1a System.Net.Security.SecureChannel.get_IsValidContext())
00007000109E2250 00000001149c90a8 (MethodDesc 0000000114ac5880 + 0x98 System.Net.Security.SslStreamInternal.ProcessFrameBody(Int32, Byte[], Int32, Int32, System.Net.AsyncProtocolRequest)), calling (MethodDesc 000000011494ab68 + 0 System.Net.Security.SecureChannel.Decrypt(Byte[], Int32 ByRef, Int32 ByRef))
00007000109E2260 00000001149b798d (MethodDesc 0000000114aa3070 + 0x5d System.Net.FixedSizeReader.ReadPacketAsync(System.IO.Stream, System.Net.AsyncProtocolRequest)), calling (MethodDesc 0000000114aa3308 + 0 System.Runtime.CompilerServices.AsyncVoidMethodBuilder.Start[[System.Net.FixedSizeReader+<ReadPacketAsync>d__1, System.Net.Security]](<ReadPacketAsync>d__1 ByRef))
00007000109E22F0 00000001149c8fc3 (MethodDesc 0000000114ac5870 + 0x123 System.Net.Security.SslStreamInternal.StartFrameBody(Int32, Byte[], Int32, Int32, System.Net.AsyncProtocolRequest)), calling (MethodDesc 0000000114ac5880 + 0 System.Net.Security.SslStreamInternal.ProcessFrameBody(Int32, Byte[], Int32, Int32, System.Net.AsyncProtocolRequest))
00007000109E2340 00000001149c8e82 (MethodDesc 0000000114ac5860 + 0xc2 System.Net.Security.SslStreamInternal.StartFrameHeader(Byte[], Int32, Int32, System.Net.AsyncProtocolRequest)), calling (MethodDesc 0000000114ac5870 + 0 System.Net.Security.SslStreamInternal.StartFrameBody(Int32, Byte[], Int32, Int32, System.Net.AsyncProtocolRequest))
00007000109E2380 00000001149c8da2 (MethodDesc 0000000114ac5850 + 0x112 System.Net.Security.SslStreamInternal.StartReading(Byte[], Int32, Int32, System.Net.AsyncProtocolRequest)), calling (MethodDesc 0000000114ac5860 + 0 System.Net.Security.SslStreamInternal.StartFrameHeader(Byte[], Int32, Int32, System.Net.AsyncProtocolRequest))
00007000109E23D0 00000001149c8b49 (MethodDesc 0000000114ac5840 + 0xe9 System.Net.Security.SslStreamInternal.ProcessRead(Byte[], Int32, Int32, System.Net.BufferAsyncResult)), calling (MethodDesc 0000000114ac5850 + 0 System.Net.Security.SslStreamInternal.StartReading(Byte[], Int32, Int32, System.Net.AsyncProtocolRequest))
00007000109E2420 00000001149c438a (MethodDesc 0000000114946da0 + 0x8a System.Net.Security.SslStream.BeginRead(Byte[], Int32, Int32, System.AsyncCallback, System.Object)), calling (MethodDesc 0000000114ac5840 + 0 System.Net.Security.SslStreamInternal.ProcessRead(Byte[], Int32, Int32, System.Net.BufferAsyncResult))
00007000109E2470 0000000111b9448e (MethodDesc 00000001116c1038 + 0xce System.Threading.Tasks.TaskFactory`1[[System.Int32, System.Private.CoreLib]].FromAsyncTrim[[System.__Canon, System.Private.CoreLib],[System.IO.Stream+ReadWriteParameters, System.Private.CoreLib]](System.__Canon, ReadWriteParameters, System.Func`5<System.__Canon,ReadWriteParameters,System.AsyncCallback,System.Object,System.IAsyncResult>, System.Func`3<System.__Canon,System.IAsyncResult,Int32>))
00007000109E24E0 0000000111b4de8a (MethodDesc 00000001116f5ae8 + 0x18a System.IO.Stream.BeginEndReadAsync(Byte[], Int32, Int32)), calling (MethodDesc 00000001116c1038 + 0 System.Threading.Tasks.TaskFactory`1[[System.Int32, System.Private.CoreLib]].FromAsyncTrim[[System.__Canon, System.Private.CoreLib],[System.IO.Stream+ReadWriteParameters, System.Private.CoreLib]](System.__Canon, ReadWriteParameters, System.Func`5<System.__Canon,ReadWriteParameters,System.AsyncCallback,System.Object,System.IAsyncResult>, System.Func`3<System.__Canon,System.IAsyncResult,Int32>))
00007000109E2500 0000000114b3fc8d (MethodDesc 0000000114642ce0 + 0xdd Microsoft.Azure.Amqp.AmqpConnectionBase.OnReceiveFrameBuffer(Microsoft.Azure.Amqp.ByteBuffer))
00007000109E2550 0000000111b4dcd9 (MethodDesc 00000001116f5ac8 + 0x29 System.IO.Stream.ReadAsync(Byte[], Int32, Int32, System.Threading.CancellationToken)), calling (MethodDesc 00000001116f5ae8 + 0 System.IO.Stream.BeginEndReadAsync(Byte[], Int32, Int32))
00007000109E2570 0000000111b4dca8 (MethodDesc 00000001116f5ab8 + 0x28 System.IO.Stream.ReadAsync(Byte[], Int32, Int32))
00007000109E2590 00000001146efec7 (MethodDesc 000000011494a580 + 0x67 Microsoft.Azure.Amqp.StreamExtensions.BeginRead(System.IO.Stream, Byte[], Int32, Int32, System.AsyncCallback, System.Object)), calling (MethodDesc 00000001116f5ab8 + 0 System.IO.Stream.ReadAsync(Byte[], Int32, Int32))
00007000109E25F0 00000001146efdce (MethodDesc 0000000114944cb0 + 0xae Microsoft.Azure.Amqp.Transport.TlsTransport.ReadAsync(Microsoft.Azure.Amqp.Transport.TransportAsyncCallbackArgs)), calling (MethodDesc 000000011494a580 + 0 Microsoft.Azure.Amqp.StreamExtensions.BeginRead(System.IO.Stream, Byte[], Int32, Int32, System.AsyncCallback, System.Object))
00007000109E2660 00000001146f2eee (MethodDesc 0000000114ac8f10 + 0x5e Microsoft.Azure.Amqp.Sasl.SaslTransport.ReadAsync(Microsoft.Azure.Amqp.Transport.TransportAsyncCallbackArgs))
00007000109E26A0 0000000114b39eaa (MethodDesc 0000000114b27b60 + 0x13a Microsoft.Azure.Amqp.AsyncIO+AsyncReader.ReadBuffer())
00007000109E2740 0000000114b3f62d (MethodDesc 0000000114b27b50 + 0xcd Microsoft.Azure.Amqp.AsyncIO+AsyncReader.OnReadBufferComplete(Microsoft.Azure.Amqp.Transport.TransportAsyncCallbackArgs)), calling (MethodDesc 0000000114b27b60 + 0 Microsoft.Azure.Amqp.AsyncIO+AsyncReader.ReadBuffer())
00007000109E27C0 00000001146ef2cf (MethodDesc 0000000114944d30 + 0x20f Microsoft.Azure.Amqp.Transport.TlsTransport.HandleOperationComplete(System.IAsyncResult, Boolean, Boolean))
00007000109E2890 00000001146f0be8 (MethodDesc 0000000114944d00 + 0xd8 Microsoft.Azure.Amqp.Transport.TlsTransport.OnReadComplete(System.IAsyncResult)), calling (MethodDesc 0000000114944d30 + 0 Microsoft.Azure.Amqp.Transport.TlsTransport.HandleOperationComplete(System.IAsyncResult, Boolean, Boolean))
00007000109E28E0 00000001146f0aeb (MethodDesc 0000000114ac7368 + 0x16b Microsoft.Azure.Amqp.TaskHelpers+<>c__DisplayClass8_0`1[[System.Int32, System.Private.CoreLib]].<ToAsyncResult>b__1(System.Threading.Tasks.Task`1<Int32>))
00007000109E2970 0000000111b0d051 (MethodDesc 00000001116edae8 + 0x71 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007000109E29C0 0000000111b46945 (MethodDesc 00000001116f5428 + 0x1f5 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)), calling (MethodDesc 00000001116edae8 + 0 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007000109E29F0 00000001149bd66e (MethodDesc 000000011494c2c0 + 0x6e System.Net.LazyAsyncResult.get_CompletedSynchronously())
00007000109E2A10 00000001032f3045 libcoreclr.dylib!JIT_GetSharedGCThreadStaticBase(unsigned long, unsigned int) + 0x35
00007000109E2A80 0000000111c7a808 (MethodDesc 0000000111716ce0 + 0x48 System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(System.Threading.Tasks.Task, Boolean)), calling (MethodDesc 00000001116f5408 + 0 System.Threading.Tasks.Task.ExecuteEntryUnsafe())
00007000109E2AC0 0000000111b7a148 (MethodDesc 00000001116fa5c8 + 0xb8 System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean))
00007000109E2B00 0000000111bb5090 (MethodDesc 0000000111701bb8 + 0x60 System.Threading.Tasks.TaskContinuation.InlineIfPossibleOrElseQueue(System.Threading.Tasks.Task, Boolean)), calling (MethodDesc 00000001116fa5c8 + 0 System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean))
00007000109E2B30 0000000111b48f45 (MethodDesc 00000001116f49e0 + 0x175 System.Threading.Tasks.Task.ContinueWithCore(System.Threading.Tasks.Task, System.Threading.Tasks.TaskScheduler, System.Threading.CancellationToken, System.Threading.Tasks.TaskContinuationOptions)), calling (MethodDesc 00000001116fbfb8 + 0 System.Threading.Tasks.StandardTaskContinuation.Run(System.Threading.Tasks.Task, Boolean))
00007000109E2B90 0000000111b4c8d6 (MethodDesc 00000001116f57a0 + 0x106 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].ContinueWith(System.Action`1<System.Threading.Tasks.Task`1<Int32>>, System.Threading.Tasks.TaskScheduler, System.Threading.CancellationToken, System.Threading.Tasks.TaskContinuationOptions)), calling (MethodDesc 00000001116f49e0 + 0 System.Threading.Tasks.Task.ContinueWithCore(System.Threading.Tasks.Task, System.Threading.Tasks.TaskScheduler, System.Threading.CancellationToken, System.Threading.Tasks.TaskContinuationOptions))
00007000109E2C00 0000000111b4c933 (MethodDesc 00000001116f5790 + 0x43 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].ContinueWith(System.Action`1<System.Threading.Tasks.Task`1<Int32>>, System.Threading.Tasks.TaskContinuationOptions)), calling (MethodDesc 00000001116f57a0 + 0 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].ContinueWith(System.Action`1<System.Threading.Tasks.Task`1<Int32>>, System.Threading.Tasks.TaskScheduler, System.Threading.CancellationToken, System.Threading.Tasks.TaskContinuationOptions))
00007000109E2C40 00000001146f04e0 (MethodDesc 0000000114ac7040 + 0x1c0 Microsoft.Azure.Amqp.TaskHelpers.ToAsyncResult[[System.Int32, System.Private.CoreLib]](System.Threading.Tasks.Task`1<Int32>, System.AsyncCallback, System.Object)), calling (MethodDesc 00000001116f5790 + 0 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].ContinueWith(System.Action`1<System.Threading.Tasks.Task`1<Int32>>, System.Threading.Tasks.TaskContinuationOptions))
00007000109E2CE0 00000001146efedc (MethodDesc 000000011494a580 + 0x7c Microsoft.Azure.Amqp.StreamExtensions.BeginRead(System.IO.Stream, Byte[], Int32, Int32, System.AsyncCallback, System.Object)), calling (MethodDesc 0000000114ac7040 + 0 Microsoft.Azure.Amqp.TaskHelpers.ToAsyncResult[[System.Int32, System.Private.CoreLib]](System.Threading.Tasks.Task`1<Int32>, System.AsyncCallback, System.Object))
00007000109E2D40 00000001146efdce (MethodDesc 0000000114944cb0 + 0xae Microsoft.Azure.Amqp.Transport.TlsTransport.ReadAsync(Microsoft.Azure.Amqp.Transport.TransportAsyncCallbackArgs)), calling (MethodDesc 000000011494a580 + 0 Microsoft.Azure.Amqp.StreamExtensions.BeginRead(System.IO.Stream, Byte[], Int32, Int32, System.AsyncCallback, System.Object))
00007000109E2DB0 00000001146f2eee (MethodDesc 0000000114ac8f10 + 0x5e Microsoft.Azure.Amqp.Sasl.SaslTransport.ReadAsync(Microsoft.Azure.Amqp.Transport.TransportAsyncCallbackArgs))
00007000109E2DF0 0000000114b39eaa (MethodDesc 0000000114b27b60 + 0x13a Microsoft.Azure.Amqp.AsyncIO+AsyncReader.ReadBuffer())
00007000109E2E90 0000000114b3f62d (MethodDesc 0000000114b27b50 + 0xcd Microsoft.Azure.Amqp.AsyncIO+AsyncReader.OnReadBufferComplete(Microsoft.Azure.Amqp.Transport.TransportAsyncCallbackArgs)), calling (MethodDesc 0000000114b27b60 + 0 Microsoft.Azure.Amqp.AsyncIO+AsyncReader.ReadBuffer())
00007000109E2F10 00000001146ef2cf (MethodDesc 0000000114944d30 + 0x20f Microsoft.Azure.Amqp.Transport.TlsTransport.HandleOperationComplete(System.IAsyncResult, Boolean, Boolean))
00007000109E2FE0 00000001146f0be8 (MethodDesc 0000000114944d00 + 0xd8 Microsoft.Azure.Amqp.Transport.TlsTransport.OnReadComplete(System.IAsyncResult)), calling (MethodDesc 0000000114944d30 + 0 Microsoft.Azure.Amqp.Transport.TlsTransport.HandleOperationComplete(System.IAsyncResult, Boolean, Boolean))
00007000109E3030 00000001146f0aeb (MethodDesc 0000000114ac7368 + 0x16b Microsoft.Azure.Amqp.TaskHelpers+<>c__DisplayClass8_0`1[[System.Int32, System.Private.CoreLib]].<ToAsyncResult>b__1(System.Threading.Tasks.Task`1<Int32>))
00007000109E30C0 0000000111b0d051 (MethodDesc 00000001116edae8 + 0x71 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007000109E30D0 00000001149b7f5d (MethodDesc 000000011494c958 + 0x2d System.Net.AsyncProtocolRequest.CompleteUser(Int32)), calling (MethodDesc 000000011494c300 + 0 System.Net.LazyAsyncResult.ProtectedInvokeCallback(System.Object, IntPtr))
00007000109E3110 0000000111b46945 (MethodDesc 00000001116f5428 + 0x1f5 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)), calling (MethodDesc 00000001116edae8 + 0 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007000109E3160 00000001032f3045 libcoreclr.dylib!JIT_GetSharedGCThreadStaticBase(unsigned long, unsigned int) + 0x35
00007000109E31D0 0000000111c7a808 (MethodDesc 0000000111716ce0 + 0x48 System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(System.Threading.Tasks.Task, Boolean)), calling (MethodDesc 00000001116f5408 + 0 System.Threading.Tasks.Task.ExecuteEntryUnsafe())
00007000109E3210 0000000111b7a148 (MethodDesc 00000001116fa5c8 + 0xb8 System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean))
00007000109E3250 0000000111bb5090 (MethodDesc 0000000111701bb8 + 0x60 System.Threading.Tasks.TaskContinuation.InlineIfPossibleOrElseQueue(System.Threading.Tasks.Task, Boolean)), calling (MethodDesc 00000001116fa5c8 + 0 System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean))
00007000109E3280 0000000111b48f45 (MethodDesc 00000001116f49e0 + 0x175 System.Threading.Tasks.Task.ContinueWithCore(System.Threading.Tasks.Task, System.Threading.Tasks.TaskScheduler, System.Threading.CancellationToken, System.Threading.Tasks.TaskContinuationOptions)), calling (MethodDesc 00000001116fbfb8 + 0 System.Threading.Tasks.StandardTaskContinuation.Run(System.Threading.Tasks.Task, Boolean))
00007000109E32E0 0000000111b4c8d6 (MethodDesc 00000001116f57a0 + 0x106 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].ContinueWith(System.Action`1<System.Threading.Tasks.Task`1<Int32>>, System.Threading.Tasks.TaskScheduler, System.Threading.CancellationToken, System.Threading.Tasks.TaskContinuationOptions)), calling (MethodDesc 00000001116f49e0 + 0 System.Threading.Tasks.Task.ContinueWithCore(System.Threading.Tasks.Task, System.Threading.Tasks.TaskScheduler, System.Threading.CancellationToken, System.Threading.Tasks.TaskContinuationOptions))
00007000109E3350 0000000111b4c933 (MethodDesc 00000001116f5790 + 0x43 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].ContinueWith(System.Action`1<System.Threading.Tasks.Task`1<Int32>>, System.Threading.Tasks.TaskContinuationOptions)), calling (MethodDesc 00000001116f57a0 + 0 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].ContinueWith(System.Action`1<System.Threading.Tasks.Task`1<Int32>>, System.Threading.Tasks.TaskScheduler, System.Threading.CancellationToken, System.Threading.Tasks.TaskContinuationOptions))
00007000109E3390 00000001146f04e0 (MethodDesc 0000000114ac7040 + 0x1c0 Microsoft.Azure.Amqp.TaskHelpers.ToAsyncResult[[System.Int32, System.Private.CoreLib]](System.Threading.Tasks.Task`1<Int32>, System.AsyncCallback, System.Object)), calling (MethodDesc 00000001116f5790 + 0 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].ContinueWith(System.Action`1<System.Threading.Tasks.Task`1<Int32>>, System.Threading.Tasks.TaskContinuationOptions))
00007000109E3430 00000001146efedc (MethodDesc 000000011494a580 + 0x7c Microsoft.Azure.Amqp.StreamExtensions.BeginRead(System.IO.Stream, Byte[], Int32, Int32, System.AsyncCallback, System.Object)), calling (MethodDesc 0000000114ac7040 + 0 Microsoft.Azure.Amqp.TaskHelpers.ToAsyncResult[[System.Int32, System.Private.CoreLib]](System.Threading.Tasks.Task`1<Int32>, System.AsyncCallback, System.Object))
00007000109E3490 00000001146efdce (MethodDesc 0000000114944cb0 + 0xae Microsoft.Azure.Amqp.Transport.TlsTransport.ReadAsync(Microsoft.Azure.Amqp.Transport.TransportAsyncCallbackArgs)), calling (MethodDesc 000000011494a580 + 0 Microsoft.Azure.Amqp.StreamExtensions.BeginRead(System.IO.Stream, Byte[], Int32, Int32, System.AsyncCallback, System.Object))
00007000109E3500 00000001146f2eee (MethodDesc 0000000114ac8f10 + 0x5e Microsoft.Azure.Amqp.Sasl.SaslTransport.ReadAsync(Microsoft.Azure.Amqp.Transport.TransportAsyncCallbackArgs))
00007000109E3540 0000000114b39eaa (MethodDesc 0000000114b27b60 + 0x13a Microsoft.Azure.Amqp.AsyncIO+AsyncReader.ReadBuffer())
00007000109E35E0 0000000114b3f62d (MethodDesc 0000000114b27b50 + 0xcd Microsoft.Azure.Amqp.AsyncIO+AsyncReader.OnReadBufferComplete(Microsoft.Azure.Amqp.Transport.TransportAsyncCallbackArgs)), calling (MethodDesc 0000000114b27b60 + 0 Microsoft.Azure.Amqp.AsyncIO+AsyncReader.ReadBuffer())
00007000109E3660 00000001146ef2cf (MethodDesc 0000000114944d30 + 0x20f Microsoft.Azure.Amqp.Transport.TlsTransport.HandleOperationComplete(System.IAsyncResult, Boolean, Boolean))
00007000109E3730 00000001146f0be8 (MethodDesc 0000000114944d00 + 0xd8 Microsoft.Azure.Amqp.Transport.TlsTransport.OnReadComplete(System.IAsyncResult)), calling (MethodDesc 0000000114944d30 + 0 Microsoft.Azure.Amqp.Transport.TlsTransport.HandleOperationComplete(System.IAsyncResult, Boolean, Boolean))
00007000109E3780 00000001146f0aeb (MethodDesc 0000000114ac7368 + 0x16b Microsoft.Azure.Amqp.TaskHelpers+<>c__DisplayClass8_0`1[[System.Int32, System.Private.CoreLib]].<ToAsyncResult>b__1(System.Threading.Tasks.Task`1<Int32>))
00007000109E3810 0000000111b0d051 (MethodDesc 00000001116edae8 + 0x71 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007000109E3860 0000000111b46945 (MethodDesc 00000001116f5428 + 0x1f5 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)), calling (MethodDesc 00000001116edae8 + 0 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007000109E3890 00000001149bd66e (MethodDesc 000000011494c2c0 + 0x6e System.Net.LazyAsyncResult.get_CompletedSynchronously())
00007000109E38B0 00000001032f3045 libcoreclr.dylib!JIT_GetSharedGCThreadStaticBase(unsigned long, unsigned int) + 0x35
00007000109E3920 0000000111c7a808 (MethodDesc 0000000111716ce0 + 0x48 System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(System.Threading.Tasks.Task, Boolean)), calling (MethodDesc 00000001116f5408 + 0 System.Threading.Tasks.Task.ExecuteEntryUnsafe())
00007000109E3960 0000000111b7a148 (MethodDesc 00000001116fa5c8 + 0xb8 System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean))
00007000109E39A0 0000000111bb5090 (MethodDesc 0000000111701bb8 + 0x60 System.Threading.Tasks.TaskContinuation.InlineIfPossibleOrElseQueue(System.Threading.Tasks.Task, Boolean)), calling (MethodDesc 00000001116fa5c8 + 0 System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean))
00007000109E39D0 0000000111b48f45 (MethodDesc 00000001116f49e0 + 0x175 System.Threading.Tasks.Task.ContinueWithCore(System.Threading.Tasks.Task, System.Threading.Tasks.TaskScheduler, System.Threading.CancellationToken, System.Threading.Tasks.TaskContinuationOptions)), calling (MethodDesc 00000001116fbfb8 + 0 System.Threading.Tasks.StandardTaskContinuation.Run(System.Threading.Tasks.Task, Boolean))
00007000109E3A30 0000000111b4c8d6 (MethodDesc 00000001116f57a0 + 0x106 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].ContinueWith(System.Action`1<System.Threading.Tasks.Task`1<Int32>>, System.Threading.Tasks.TaskScheduler, System.Threading.CancellationToken, System.Threading.Tasks.TaskContinuationOptions)), calling (MethodDesc 00000001116f49e0 + 0 System.Threading.Tasks.Task.ContinueWithCore(System.Threading.Tasks.Task, System.Threading.Tasks.TaskScheduler, System.Threading.CancellationToken, System.Threading.Tasks.TaskContinuationOptions))
00007000109E3AA0 0000000111b4c933 (MethodDesc 00000001116f5790 + 0x43 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].ContinueWith(System.Action`1<System.Threading.Tasks.Task`1<Int32>>, System.Threading.Tasks.TaskContinuationOptions)), calling (MethodDesc 00000001116f57a0 + 0 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].ContinueWith(System.Action`1<System.Threading.Tasks.Task`1<Int32>>, System.Threading.Tasks.TaskScheduler, System.Threading.CancellationToken, System.Threading.Tasks.TaskContinuationOptions))
00007000109E3AE0 00000001146f04e0 (MethodDesc 0000000114ac7040 + 0x1c0 Microsoft.Azure.Amqp.TaskHelpers.ToAsyncResult[[System.Int32, System.Private.CoreLib]](System.Threading.Tasks.Task`1<Int32>, System.AsyncCallback, System.Object)), calling (MethodDesc 00000001116f5790 + 0 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].ContinueWith(System.Action`1<System.Threading.Tasks.Task`1<Int32>>, System.Threading.Tasks.TaskContinuationOptions))
00007000109E3B80 00000001146efedc (MethodDesc 000000011494a580 + 0x7c Microsoft.Azure.Amqp.StreamExtensions.BeginRead(System.IO.Stream, Byte[], Int32, Int32, System.AsyncCallback, System.Object)), calling (MethodDesc 0000000114ac7040 + 0 Microsoft.Azure.Amqp.TaskHelpers.ToAsyncResult[[System.Int32, System.Private.CoreLib]](System.Threading.Tasks.Task`1<Int32>, System.AsyncCallback, System.Object))
00007000109E3BE0 00000001146efdce (MethodDesc 0000000114944cb0 + 0xae Microsoft.Azure.Amqp.Transport.TlsTransport.ReadAsync(Microsoft.Azure.Amqp.Transport.TransportAsyncCallbackArgs)), calling (MethodDesc 000000011494a580 + 0 Microsoft.Azure.Amqp.StreamExtensions.BeginRead(System.IO.Stream, Byte[], Int32, Int32, System.AsyncCallback, System.Object))
00007000109E3C50 00000001146f2eee (MethodDesc 0000000114ac8f10 + 0x5e Microsoft.Azure.Amqp.Sasl.SaslTransport.ReadAsync(Microsoft.Azure.Amqp.Transport.TransportAsyncCallbackArgs))
00007000109E3C90 0000000114b39eaa (MethodDesc 0000000114b27b60 + 0x13a Microsoft.Azure.Amqp.AsyncIO+AsyncReader.ReadBuffer())
00007000109E3D30 0000000114b3f62d (MethodDesc 0000000114b27b50 + 0xcd Microsoft.Azure.Amqp.AsyncIO+AsyncReader.OnReadBufferComplete(Microsoft.Azure.Amqp.Transport.TransportAsyncCallbackArgs)), calling (MethodDesc 0000000114b27b60 + 0 Microsoft.Azure.Amqp.AsyncIO+AsyncReader.ReadBuffer())
00007000109E3DB0 00000001146ef2cf (MethodDesc 0000000114944d30 + 0x20f Microsoft.Azure.Amqp.Transport.TlsTransport.HandleOperationComplete(System.IAsyncResult, Boolean, Boolean))
00007000109E3E80 00000001146f0be8 (MethodDesc 0000000114944d00 + 0xd8 Microsoft.Azure.Amqp.Transport.TlsTransport.OnReadComplete(System.IAsyncResult)), calling (MethodDesc 0000000114944d30 + 0 Microsoft.Azure.Amqp.Transport.TlsTransport.HandleOperationComplete(System.IAsyncResult, Boolean, Boolean))
00007000109E3ED0 00000001146f0aeb (MethodDesc 0000000114ac7368 + 0x16b Microsoft.Azure.Amqp.TaskHelpers+<>c__DisplayClass8_0`1[[System.Int32, System.Private.CoreLib]].<ToAsyncResult>b__1(System.Threading.Tasks.Task`1<Int32>))
00007000109E3F60 0000000111b0d051 (MethodDesc 00000001116edae8 + 0x71 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007000109E3F70 00000001149b7f5d (MethodDesc 000000011494c958 + 0x2d System.Net.AsyncProtocolRequest.CompleteUser(Int32)), calling (MethodDesc 000000011494c300 + 0 System.Net.LazyAsyncResult.ProtectedInvokeCallback(System.Object, IntPtr))
00007000109E3FB0 0000000111b46945 (MethodDesc 00000001116f5428 + 0x1f5 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)), calling (MethodDesc 00000001116edae8 + 0 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007000109E4000 00000001032f3045 libcoreclr.dylib!JIT_GetSharedGCThreadStaticBase(unsigned long, unsigned int) + 0x35
00007000109E4070 0000000111c7a808 (MethodDesc 0000000111716ce0 + 0x48 System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(System.Threading.Tasks.Task, Boolean)), calling (MethodDesc 00000001116f5408 + 0 System.Threading.Tasks.Task.ExecuteEntryUnsafe())
00007000109E40B0 0000000111b7a148 (MethodDesc 00000001116fa5c8 + 0xb8 System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean))
00007000109E40F0 0000000111bb5090 (MethodDesc 0000000111701bb8 + 0x60 System.Threading.Tasks.TaskContinuation.InlineIfPossibleOrElseQueue(System.Threading.Tasks.Task, Boolean)), calling (MethodDesc 00000001116fa5c8 + 0 System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean))
00007000109E4120 0000000111b48f45 (MethodDesc 00000001116f49e0 + 0x175 System.Threading.Tasks.Task.ContinueWithCore(System.Threading.Tasks.Task, System.Threading.Tasks.TaskScheduler, System.Threading.CancellationToken, System.Threading.Tasks.TaskContinuationOptions)), calling (MethodDesc 00000001116fbfb8 + 0 System.Threading.Tasks.StandardTaskContinuation.Run(System.Threading.Tasks.Task, Boolean))
00007000109E4180 0000000111b4c8d6 (MethodDesc 00000001116f57a0 + 0x106 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].ContinueWith(System.Action`1<System.Threading.Tasks.Task`1<Int32>>, System.Threading.Tasks.TaskScheduler, System.Threading.CancellationToken, System.Threading.Tasks.TaskContinuationOptions)), calling (MethodDesc 00000001116f49e0 + 0 System.Threading.Tasks.Task.ContinueWithCore(System.Threading.Tasks.Task, System.Threading.Tasks.TaskScheduler, System.Threading.CancellationToken, System.Threading.Tasks.TaskContinuationOptions))
00007000109E41F0 0000000111b4c933 (MethodDesc 00000001116f5790 + 0x43 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].ContinueWith(System.Action`1<System.Threading.Tasks.Task`1<Int32>>, System.Threading.Tasks.TaskContinuationOptions)), calling (MethodDesc 00000001116f57a0 + 0 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].ContinueWith(System.Action`1<System.Threading.Tasks.Task`1<Int32>>, System.Threading.Tasks.TaskScheduler, System.Threading.CancellationToken, System.Threading.Tasks.TaskContinuationOptions))
00007000109E4230 00000001146f04e0 (MethodDesc 0000000114ac7040 + 0x1c0 Microsoft.Azure.Amqp.TaskHelpers.ToAsyncResult[[System.Int32, System.Private.CoreLib]](System.Threading.Tasks.Task`1<Int32>, System.AsyncCallback, System.Object)), calling (MethodDesc 00000001116f5790 + 0 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].ContinueWith(System.Action`1<System.Threading.Tasks.Task`1<Int32>>, System.Threading.Tasks.TaskContinuationOptions))
00007000109E42D0 00000001146efedc (MethodDesc 000000011494a580 + 0x7c Microsoft.Azure.Amqp.StreamExtensions.BeginRead(System.IO.Stream, Byte[], Int32, Int32, System.AsyncCallback, System.Object)), calling (MethodDesc 0000000114ac7040 + 0 Microsoft.Azure.Amqp.TaskHelpers.ToAsyncResult[[System.Int32, System.Private.CoreLib]](System.Threading.Tasks.Task`1<Int32>, System.AsyncCallback, System.Object))
00007000109E4330 00000001146efdce (MethodDesc 0000000114944cb0 + 0xae Microsoft.Azure.Amqp.Transport.TlsTransport.ReadAsync(Microsoft.Azure.Amqp.Transport.TransportAsyncCallbackArgs)), calling (MethodDesc 000000011494a580 + 0 Microsoft.Azure.Amqp.StreamExtensions.BeginRead(System.IO.Stream, Byte[], Int32, Int32, System.AsyncCallback, System.Object))
00007000109E43A0 00000001146f2eee (MethodDesc 0000000114ac8f10 + 0x5e Microsoft.Azure.Amqp.Sasl.SaslTransport.ReadAsync(Microsoft.Azure.Amqp.Transport.TransportAsyncCallbackArgs))
00007000109E43E0 0000000114b39eaa (MethodDesc 0000000114b27b60 + 0x13a Microsoft.Azure.Amqp.AsyncIO+AsyncReader.ReadBuffer())
00007000109E4480 0000000114b3f62d (MethodDesc 0000000114b27b50 + 0xcd Microsoft.Azure.Amqp.AsyncIO+AsyncReader.OnReadBufferComplete(Microsoft.Azure.Amqp.Transport.TransportAsyncCallbackArgs)), calling (MethodDesc 0000000114b27b60 + 0 Microsoft.Azure.Amqp.AsyncIO+AsyncReader.ReadBuffer())
00007000109E4500 00000001146ef2cf (MethodDesc 0000000114944d30 + 0x20f Microsoft.Azure.Amqp.Transport.TlsTransport.HandleOperationComplete(System.IAsyncResult, Boolean, Boolean))
00007000109E45D0 00000001146f0be8 (MethodDesc 0000000114944d00 + 0xd8 Microsoft.Azure.Amqp.Transport.TlsTransport.OnReadComplete(System.IAsyncResult)), calling (MethodDesc 0000000114944d30 + 0 Microsoft.Azure.Amqp.Transport.TlsTransport.HandleOperationComplete(System.IAsyncResult, Boolean, Boolean))
00007000109E4620 00000001146f0aeb (MethodDesc 0000000114ac7368 + 0x16b Microsoft.Azure.Amqp.TaskHelpers+<>c__DisplayClass8_0`1[[System.Int32, System.Private.CoreLib]].<ToAsyncResult>b__1(System.Threading.Tasks.Task`1<Int32>))
00007000109E46B0 0000000111b0d051 (MethodDesc 00000001116edae8 + 0x71 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007000109E4700 0000000111b46945 (MethodDesc 00000001116f5428 + 0x1f5 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)), calling (MethodDesc 00000001116edae8 + 0 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007000109E4730 00000001149bd66e (MethodDesc 000000011494c2c0 + 0x6e System.Net.LazyAsyncResult.get_CompletedSynchronously())
00007000109E4750 00000001032f3045 libcoreclr.dylib!JIT_GetSharedGCThreadStaticBase(unsigned long, unsigned int) + 0x35
00007000109E47C0 0000000111c7a808 (MethodDesc 0000000111716ce0 + 0x48 System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(System.Threading.Tasks.Task, Boolean)), calling (MethodDesc 00000001116f5408 + 0 System.Threading.Tasks.Task.ExecuteEntryUnsafe())
00007000109E4800 0000000111b7a148 (MethodDesc 00000001116fa5c8 + 0xb8 System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean))
00007000109E4840 0000000111bb5090 (MethodDesc 0000000111701bb8 + 0x60 System.Threading.Tasks.TaskContinuation.InlineIfPossibleOrElseQueue(System.Threading.Tasks.Task, Boolean)), calling (MethodDesc 00000001116fa5c8 + 0 System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean))
00007000109E4870 0000000111b48f45 (MethodDesc 00000001116f49e0 + 0x175 System.Threading.Tasks.Task.ContinueWithCore(System.Threading.Tasks.Task, System.Threading.Tasks.TaskScheduler, System.Threading.CancellationToken, System.Threading.Tasks.TaskContinuationOptions)), calling (MethodDesc 00000001116fbfb8 + 0 System.Threading.Tasks.StandardTaskContinuation.Run(System.Threading.Tasks.Task, Boolean))
00007000109E48D0 0000000111b4c8d6 (MethodDesc 00000001116f57a0 + 0x106 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].ContinueWith(System.Action`1<System.Threading.Tasks.Task`1<Int32>>, System.Threading.Tasks.TaskScheduler, System.Threading.CancellationToken, System.Threading.Tasks.TaskContinuationOptions)), calling (MethodDesc 00000001116f49e0 + 0 System.Threading.Tasks.Task.ContinueWithCore(System.Threading.Tasks.Task, System.Threading.Tasks.TaskScheduler, System.Threading.CancellationToken, System.Threading.Tasks.TaskContinuationOptions))
00007000109E4940 0000000111b4c933 (MethodDesc 00000001116f5790 + 0x43 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].ContinueWith(System.Action`1<System.Threading.Tasks.Task`1<Int32>>, System.Threading.Tasks.TaskContinuationOptions)), calling (MethodDesc 00000001116f57a0 + 0 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].ContinueWith(System.Action`1<System.Threading.Tasks.Task`1<Int32>>, System.Threading.Tasks.TaskScheduler, System.Threading.CancellationToken, System.Threading.Tasks.TaskContinuationOptions))
00007000109E4980 00000001146f04e0 (MethodDesc 0000000114ac7040 + 0x1c0 Microsoft.Azure.Amqp.TaskHelpers.ToAsyncResult[[System.Int32, System.Private.CoreLib]](System.Threading.Tasks.Task`1<Int32>, System.AsyncCallback, System.Object)), calling (MethodDesc 00000001116f5790 + 0 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].ContinueWith(System.Action`1<System.Threading.Tasks.Task`1<Int32>>, System.Threading.Tasks.TaskContinuationOptions))
00007000109E4A20 00000001146efedc (MethodDesc 000000011494a580 + 0x7c Microsoft.Azure.Amqp.StreamExtensions.BeginRead(System.IO.Stream, Byte[], Int32, Int32, System.AsyncCallback, System.Object)), calling (MethodDesc 0000000114ac7040 + 0 Microsoft.Azure.Amqp.TaskHelpers.ToAsyncResult[[System.Int32, System.Private.CoreLib]](System.Threading.Tasks.Task`1<Int32>, System.AsyncCallback, System.Object))
00007000109E4A80 00000001146efdce (MethodDesc 0000000114944cb0 + 0xae Microsoft.Azure.Amqp.Transport.TlsTransport.ReadAsync(Microsoft.Azure.Amqp.Transport.TransportAsyncCallbackArgs)), calling (MethodDesc 000000011494a580 + 0 Microsoft.Azure.Amqp.StreamExtensions.BeginRead(System.IO.Stream, Byte[], Int32, Int32, System.AsyncCallback, System.Object))
00007000109E4AF0 00000001146f2eee (MethodDesc 0000000114ac8f10 + 0x5e Microsoft.Azure.Amqp.Sasl.SaslTransport.ReadAsync(Microsoft.Azure.Amqp.Transport.TransportAsyncCallbackArgs))
00007000109E4B30 0000000114b39eaa (MethodDesc 0000000114b27b60 + 0x13a Microsoft.Azure.Amqp.AsyncIO+AsyncReader.ReadBuffer())
00007000109E4BD0 0000000114b3f62d (MethodDesc 0000000114b27b50 + 0xcd Microsoft.Azure.Amqp.AsyncIO+AsyncReader.OnReadBufferComplete(Microsoft.Azure.Amqp.Transport.TransportAsyncCallbackArgs)), calling (MethodDesc 0000000114b27b60 + 0 Microsoft.Azure.Amqp.AsyncIO+AsyncReader.ReadBuffer())
00007000109E4C50 00000001146ef2cf (MethodDesc 0000000114944d30 + 0x20f Microsoft.Azure.Amqp.Transport.TlsTransport.HandleOperationComplete(System.IAsyncResult, Boolean, Boolean))
00007000109E4D20 00000001146f0be8 (MethodDesc 0000000114944d00 + 0xd8 Microsoft.Azure.Amqp.Transport.TlsTransport.OnReadComplete(System.IAsyncResult)), calling (MethodDesc 0000000114944d30 + 0 Microsoft.Azure.Amqp.Transport.TlsTransport.HandleOperationComplete(System.IAsyncResult, Boolean, Boolean))
00007000109E4D70 00000001146f0aeb (MethodDesc 0000000114ac7368 + 0x16b Microsoft.Azure.Amqp.TaskHelpers+<>c__DisplayClass8_0`1[[System.Int32, System.Private.CoreLib]].<ToAsyncResult>b__1(System.Threading.Tasks.Task`1<Int32>))
00007000109E4E00 0000000111b0d051 (MethodDesc 00000001116edae8 + 0x71 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007000109E4E50 0000000111b46945 (MethodDesc 00000001116f5428 + 0x1f5 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)), calling (MethodDesc 00000001116edae8 + 0 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007000109E4E80 00000001032a7ceb libcoreclr.dylib!ThreadNative::GetThreadState(ThreadBaseObject*) + 0x18b, calling libcoreclr.dylib!Frame::PopIfChained()
00007000109E4EA0 00000001032f3045 libcoreclr.dylib!JIT_GetSharedGCThreadStaticBase(unsigned long, unsigned int) + 0x35
00007000109E4F10 0000000111c7a808 (MethodDesc 0000000111716ce0 + 0x48 System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(System.Threading.Tasks.Task, Boolean)), calling (MethodDesc 00000001116f5408 + 0 System.Threading.Tasks.Task.ExecuteEntryUnsafe())
00007000109E4F50 0000000111b7a148 (MethodDesc 00000001116fa5c8 + 0xb8 System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean))
00007000109E4F90 0000000111bb5090 (MethodDesc 0000000111701bb8 + 0x60 System.Threading.Tasks.TaskContinuation.InlineIfPossibleOrElseQueue(System.Threading.Tasks.Task, Boolean)), calling (MethodDesc 00000001116fa5c8 + 0 System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean))
00007000109E4FC0 0000000111b47c85 (MethodDesc 00000001116f5590 + 0x1f5 System.Threading.Tasks.Task.RunContinuations(System.Object))
00007000109E4FD0 00000001149bd6d1 (MethodDesc 000000011494c2c8 + 0x21 System.Net.LazyAsyncResult.get_IsCompleted())
00007000109E4FF0 00000001149bdd93 (MethodDesc 000000011494c360 + 0x193 System.Net.LazyAsyncResult.WaitForCompletion(Boolean))
00007000109E5070 0000000111b4c250 (MethodDesc 00000001116f5708 + 0x60 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].TrySetResult(Int32)), calling (MethodDesc 00000001116f5580 + 0 System.Threading.Tasks.Task.FinishContinuations())
00007000109E5090 0000000111bb824c (MethodDesc 00000001117027e8 + 0x3c System.Threading.Tasks.TaskFactory`1+FromAsyncTrimPromise`1[[System.Int32, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].Complete(System.__Canon, System.Func`3<System.__Canon,System.IAsyncResult,Int32>, System.IAsyncResult, Boolean)), calling (MethodDesc 00000001116f5708 + 0 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].TrySetResult(Int32))
00007000109E50D0 00000001149bdab9 (MethodDesc 000000011494c330 + 0x159 System.Net.LazyAsyncResult.Complete(IntPtr))
00007000109E5130 00000001149bd8ca (MethodDesc 000000011494c300 + 0xea System.Net.LazyAsyncResult.ProtectedInvokeCallback(System.Object, IntPtr))
00007000109E5140 00000001032b0e4b libcoreclr.dylib!Buffer::BlockCopy(ArrayBase*, int, ArrayBase*, int, int) + 0x2fb, calling libcoreclr.dylib!memmove
00007000109E5170 00000001149b7f5d (MethodDesc 000000011494c958 + 0x2d System.Net.AsyncProtocolRequest.CompleteUser(Int32)), calling (MethodDesc 000000011494c300 + 0 System.Net.LazyAsyncResult.ProtectedInvokeCallback(System.Object, IntPtr))
00007000109E5190 00000001149c91e6 (MethodDesc 0000000114ac5880 + 0x1d6 System.Net.Security.SslStreamInternal.ProcessFrameBody(Int32, Byte[], Int32, Int32, System.Net.AsyncProtocolRequest)), calling (MethodDesc 000000011494c958 + 0 System.Net.AsyncProtocolRequest.CompleteUser(Int32))
00007000109E51A0 0000000111b86532 (MethodDesc 00000001116fb858 + 0x62 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef))
00007000109E5230 00000001149c979b (MethodDesc 0000000114ac58e0 + 0x5b System.Net.Security.SslStreamInternal.ReadFrameCallback(System.Net.AsyncProtocolRequest)), calling (MethodDesc 0000000114ac5880 + 0 System.Net.Security.SslStreamInternal.ProcessFrameBody(Int32, Byte[], Int32, Int32, System.Net.AsyncProtocolRequest))
00007000109E5260 00000001149b7e68 (MethodDesc 000000011494c918 + 0x38 System.Net.AsyncProtocolRequest.CompleteRequest(Int32))
00007000109E5280 00000001149b7bce (MethodDesc 0000000114aa31a0 + 0x22e System.Net.FixedSizeReader+<ReadPacketAsync>d__1.MoveNext()), calling (MethodDesc 000000011494c918 + 0 System.Net.AsyncProtocolRequest.CompleteRequest(Int32))
00007000109E5320 0000000111b0d051 (MethodDesc 00000001116edae8 + 0x71 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007000109E5370 0000000111b86532 (MethodDesc 00000001116fb858 + 0x62 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef))
00007000109E53A0 0000000111b47b8e (MethodDesc 00000001116f5590 + 0xfe System.Threading.Tasks.Task.RunContinuations(System.Object)), calling (MethodDesc 00000001116fb858 + 0 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef))
00007000109E5450 0000000111b4c250 (MethodDesc 00000001116f5708 + 0x60 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].TrySetResult(Int32)), calling (MethodDesc 00000001116f5580 + 0 System.Threading.Tasks.Task.FinishContinuations())
00007000109E5470 0000000111b95044 (MethodDesc 00000001116fd9a0 + 0x184 System.Threading.Tasks.TaskFactory`1[[System.Int32, System.Private.CoreLib]].FromAsyncCoreLogic(System.IAsyncResult, System.Func`2<System.IAsyncResult,Int32>, System.Action`1<System.IAsyncResult>, System.Threading.Tasks.Task`1<Int32>, Boolean)), calling (MethodDesc 00000001116f5708 + 0 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].TrySetResult(Int32))
00007000109E5490 0000000111b94f1d (MethodDesc 00000001116fd9a0 + 0x5d System.Threading.Tasks.TaskFactory`1[[System.Int32, System.Private.CoreLib]].FromAsyncCoreLogic(System.IAsyncResult, System.Func`2<System.IAsyncResult,Int32>, System.Action`1<System.IAsyncResult>, System.Threading.Tasks.Task`1<Int32>, Boolean)), calling (MethodDesc 00000001116fd9a0 + 0xa5 System.Threading.Tasks.TaskFactory`1[[System.Int32, System.Private.CoreLib]].FromAsyncCoreLogic(System.IAsyncResult, System.Func`2<System.IAsyncResult,Int32>, System.Action`1<System.IAsyncResult>, System.Threading.Tasks.Task`1<Int32>, Boolean))
00007000109E54A0 00000001146e2684 (MethodDesc 00000001147b4fe0 + 0x34 Microsoft.Azure.Amqp.Transport.TransportAsyncCallbackArgs.System.IAsyncResult.get_CompletedSynchronously()), calling (MethodDesc 00000001147b4f50 + 0 Microsoft.Azure.Amqp.Transport.TransportAsyncCallbackArgs.get_CompletedSynchronously())
00007000109E54E0 00000001146e25d6 (MethodDesc 0000000114946620 + 0x106 Microsoft.Azure.Amqp.Transport.TransportStream.CompleteOperation(Microsoft.Azure.Amqp.Transport.TransportAsyncCallbackArgs))
00007000109E5560 00000001146e4cce (MethodDesc 0000000114946610 + 0x9e Microsoft.Azure.Amqp.Transport.TransportStream.OnIOComplete(Microsoft.Azure.Amqp.Transport.TransportAsyncCallbackArgs)), calling (MethodDesc 0000000114946620 + 0 Microsoft.Azure.Amqp.Transport.TransportStream.CompleteOperation(Microsoft.Azure.Amqp.Transport.TransportAsyncCallbackArgs))
00007000109E55A0 00000001146e4777 (MethodDesc 00000001148fe398 + 0x527 Microsoft.Azure.Amqp.Transport.TcpTransport.HandleReadComplete(Microsoft.Azure.Amqp.Transport.TransportAsyncCallbackArgs, Boolean, Boolean))
00007000109E55B0 00007fffaa34b4ce libsystem_pthread.dylib!_pthread_cond_signal + 0x285, calling libsystem_pthread.dylib!_pthread_cond_updateval
00007000109E5710 00000001146e4185 (MethodDesc 00000001148fe378 + 0xb5 Microsoft.Azure.Amqp.Transport.TcpTransport.OnReadComplete(System.Object, System.Net.Sockets.SocketAsyncEventArgs)), calling (MethodDesc 00000001148fe398 + 0 Microsoft.Azure.Amqp.Transport.TcpTransport.HandleReadComplete(Microsoft.Azure.Amqp.Transport.TransportAsyncCallbackArgs, Boolean, Boolean))
00007000109E5760 0000000114841996 (MethodDesc 00000001147b8180 + 0x36 System.Net.Sockets.SocketAsyncEventArgs.OnCompleted(System.Net.Sockets.SocketAsyncEventArgs))
00007000109E5780 000000011484201d (MethodDesc 00000001147b82f8 + 0x1d System.Net.Sockets.SocketAsyncEventArgs.ExecutionCallback(System.Object))
00007000109E5790 0000000111b0d051 (MethodDesc 00000001116edae8 + 0x71 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007000109E57E0 0000000114843237 (MethodDesc 00000001147b8478 + 0x57 System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncSuccess(Int32, System.Net.Sockets.SocketFlags)), calling (MethodDesc 00000001116edae8 + 0 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007000109E5800 0000000114844984 (MethodDesc 00000001147b8708 + 0x14 System.Net.Sockets.SocketAsyncEventArgs.CompletionCallback(Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError)), calling (MethodDesc 00000001147b8478 + 0 System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncSuccess(Int32, System.Net.Sockets.SocketFlags))
00007000109E5810 00000001032f452f libcoreclr.dylib!JIT_ChkCastAny + 0x1f, calling libcoreclr.dylib!ObjIsInstanceOfNoGC
00007000109E5820 00000001148438ce (MethodDesc 00000001147b8598 + 0x2e System.Net.Sockets.SocketAsyncEventArgs.TransferCompletionCallbackCore(Int32, Byte[], Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError)), calling (MethodDesc 00000001147b8708 + 0 System.Net.Sockets.SocketAsyncEventArgs.CompletionCallback(Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError))
00007000109E5830 000000011484ff7b (MethodDesc 0000000114aa4488 + 0x2b System.Net.Sockets.SocketAsyncContext+ReceiveOperation.InvokeCallback())
00007000109E5840 000000011484fd7d (MethodDesc 00000001148fac68 + 0x1d System.Net.Sockets.SocketAsyncContext+AsyncOperation+<>c.<TryCompleteOrAbortAsync>b__14_0(System.Object))
00007000109E5850 0000000111b0d051 (MethodDesc 00000001116edae8 + 0x71 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007000109E58A0 0000000111b87ce0 (MethodDesc 00000001116fbf80 + 0x1a0 System.Threading.ThreadPoolWorkQueue.Dispatch())
00007000109E58E0 00000001030c5296 libcoreclr.dylib!CorUnix::CSynchCache<CorUnix::CSynchWaitController>::Get(CorUnix::CPalThread*, int, CorUnix::CSynchWaitController**) + 0x76, calling libcoreclr.dylib!CorUnix::InternalLeaveCriticalSection(CorUnix::CPalThread*, _CRITICAL_SECTION*)
00007000109E5910 00000001033a0911 libcoreclr.dylib!CallDescrWorkerInternal + 0x7c
00007000109E5930 00000001032907ed libcoreclr.dylib!MethodDescCallSite::CallTargetWorker(unsigned long const*, unsigned long*, int) + 0x3ad, calling libcoreclr.dylib!CallDescrWorkerInternal
00007000109E5A20 00000001032904f5 libcoreclr.dylib!MethodDescCallSite::CallTargetWorker(unsigned long const*, unsigned long*, int) + 0xb5, calling libcoreclr.dylib!ArgIteratorTemplate<ArgIteratorBase>::SizeOfArgStack()
00007000109E5B00 00000001032ab0d5 libcoreclr.dylib!QueueUserWorkItemManagedCallback(void*) + 0xa5, calling libcoreclr.dylib!MethodDescCallSite::CallTargetWorker(unsigned long const*, unsigned long*, int)
00007000109E5C10 00000001032526da libcoreclr.dylib!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 0x17a
00007000109E5D30 0000000103252d93 libcoreclr.dylib!ManagedThreadBase::ThreadPool(ADID, void (*)(void*), void*) + 0x33, calling libcoreclr.dylib!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)
00007000109E5D70 0000000103247a0c libcoreclr.dylib!ManagedPerAppDomainTPCount::DispatchWorkItem(bool*, bool*) + 0x10c, calling libcoreclr.dylib!ManagedThreadBase::ThreadPool(ADID, void (*)(void*), void*)
00007000109E5D90 00007fffaa16a2af libsystem_c.dylib!clock_gettime + 0xc6, calling libsystem_c.dylib!_mach_boottime_usec
00007000109E5E20 000000010324739a libcoreclr.dylib!PerAppDomainTPCountList::GetAppDomainIndexForThreadpoolDispatch() + 0x7a
00007000109E5E60 0000000103271d40 libcoreclr.dylib!ThreadpoolMgr::WorkerThreadStart(void*) + 0x3f0
00007000109E5E70 00000001032da281 libcoreclr.dylib!EEHeapFreeInProcessHeap(unsigned int, void*) + 0x31, calling libcoreclr.dylib!HeapFree
00007000109E5EC0 00000001030cedd8 libcoreclr.dylib!CorUnix::CPalThread::ThreadEntry(void*) + 0x148
00007000109E5EF0 00007fffaa34c93b libsystem_pthread.dylib!_pthread_body + 0xb4
00007000109E5F10 00007fffaa34c887 libsystem_pthread.dylib!_pthread_body, calling libsystem_pthread.dylib!_pthread_body
00007000109E5F50 00007fffaa34c08d libsystem_pthread.dylib!thread_start + 0xd, calling libsystem_pthread.dylib!_pthread_start
(lldb) 

[nemakam]

@chuckb This seems to be an interesting one. However, don't think this is anything to do with our code. Should be a dotnet core on OSX bug. https://github.com/dotnet/coreclr should be a better place to file this issue.

[chuckb]

Happy to file it over there, but I would think that this would pique some curiosity since it severely limits scalable use of your library on OSX anyway (and perhaps linux - not confirmed - too since code is similar). Would you mind replying with the basis of your analysis? While it is clear that the error is occurring in unmanaged code, my preliminary analysis of re-entrancy into SSLStreamInternal class by Amqp code (whether by that library or by your use of it) seems like a reasonable hypothesis, given the restrictions cited in the documentation.

[clemensv]

Linking comment https://github.com/dotnet/corefx/issues/24142#issuecomment-330671855 - asked for clarification. This clearly looks like a NETCore issue from the stack trace. SslStream should be safe to call in the fashion we do it; the caller here is the stack that we're using for NETFX everywhere, so it appears the NETStd port regressed the behavior on OSX in some fashion.

We'll leave this open here for tracking the issue; discussed should be on the CoreFX bug thread.

[chuckb]

Cross posting here on the result of testing my app against the work being done by corefx team to resolve this issue. When I run my app with their fix against service bus 2.0.0, I get:

Unhandled Exception: System.AggregateException: One or more errors occurred. (Method not found: 'Void* System.Buffers.MemoryHandle.get_PinnedPointer()'.) (Method not found: 'Void* System.Buffers.MemoryHandle.get_PinnedPointer()'.) ---> System.MissingMethodException: Method not found: 'Void* System.Buffers.MemoryHandle.get_PinnedPointer()'.
   at Microsoft.Azure.ServiceBus.Core.MessageSender.<OnSendAsync>d__44.MoveNext() in D:\azure-messaging-dotnet\src\Microsoft.Azure.ServiceBus\Core\MessageSender.cs:line 414
--- End of stack trace from previous location where exception was thrown ---

This error is the result of this commit: https://github.com/dotnet/corefx/commit/4c5a4efb652236a41ee701f1ee268e361b793489#diff-79239bb9a51ce78e8baaadc3a6f2f503

Not sure it requires immediate action for authors of this library, but you might let corefx folks know that what they are tinkering with will cause a breaking change for you.

Also, another thing that bothers me is that if I look at line 414 of MessageSender.cs in github for the 2.0.0 tag, I don't see a line referencing directly the PinnedPointer property, which makes me wonder which version of the servicebus dll I have (I am referencing 2.0.0 in my project).

Note for clarity I am referencing 1.0.0 in my original post, but I changed it to 2.0.0 for this test.