microsoft-graph-comms-samples icon indicating copy to clipboard operation
microsoft-graph-comms-samples copied to clipboard
verified publisher icon microsoftgraph

EchoBot throws exceptions during the initialization and terminates the call on joining.

Open genemgh opened this issue 1 year ago • 31 comments

We try to run the EchoBot example (GitHub: microsoftgraph/microsoft-graph-comms-samples/Samples/PublicSamples/EchoBot/). We have a Microsoft 365 Developer sandbox as a part of Visual Studio subscription, and a tenant in the sandbox. We use Postman to hit the bot.

Versioning: - We use Windows 11, Visual Studio 2022 Professional, C#, and .NET version 6.0. - README in the EchoBot example recommends .NET 4.7.1, but it looks like it's just wrong. (The project file for the EchoBot example contains <TargetFramework>net6.0</TargetFramework>). - We downloaded .NET 4.7.1 SDK from the NET web site, installed it, switched to .NET Framework 4.7.1, did the NuGet clean and restore. Does not compile.

Ngrok: - We have Ngrok (free version) running and providing two endpoints locally, one for HTTPS, and one for TCP.

Certificates: - We created a self-signed certificate for the TCP domain and installed it in the local certificate store for both User / Personal and Local Machine / Personal. - The certificate is being picked up by the bot (without the certificate installed, the bot does not start at all).

Graph Application and Azure Bot registrations: - The application is registered in MS Entra for our sandbox tenant, all the permissions are granted and consented. - The Azure Bot is also registered in the Azure portal. - We also added the Microsoft Teams Channel to the bot channels in the bot registration, enabled calling in the Calling tab, and added the correct domain name to Webhook (for calling) field: https://<XXXXXXXXXXXX>.ngrok-free.app/api/calling

Local environment: - The application / client ID and the application / client secret are added to the .env file in the bot project. - The Ngrok endpoints and ports are added to the .env file. - The certificate thumbprint is added to the .env file.

When we run the bot from Visual Studio in the debug mode, during the bot initialization, the following 12 different types of exceptions are logged: 1. onecore\net\netprofiles\service\src\nsp\dll\namespaceserviceprovider.cpp(613)\nlansp_c.dll!00007FFF45F1F6BD: (caller: 00007FFF619FACF6) LogHr(1) tid(59e8) 8007277C No such service is known. The service cannot be found in the specified name space. (3) 2. Microsoft C++ exception: std::system_error at memory location 0x00000050203BA850. 3. Exception thrown at 0x00007FFF5FE1FABC (KernelBase.dll) in EchoBot.exe: WinRT originate error - 0x80040155 : 'Failed to find proxy registration for IID: {79EAC9E4-BAF9-11CE-8C82-00AA004BA90B}.'. 4. Exception thrown at 0x00007FFF5FE1FABC (KernelBase.dll) in EchoBot.exe: 0x80040155: Interface not registered. (2) 5. MediaPerf is not registered: no key found at SYSTEM\CurrentControlSet\Services\MediaPerf\Performance 6. Exception thrown at 0x00007FFF5FE1FABC (KernelBase.dll) in EchoBot.exe: 0x000006D9: There are no more endpoints available from the endpoint mapper. 7. Exception thrown at 0x00007FFF5FE1FABC (KernelBase.dll) in EchoBot.exe: 0x000006BA: The RPC server is unavailable. (3) 8. 'System.InvalidOperationException' in System.Diagnostics.PerformanceCounter.dll: Category does not exist. 9. 'System.IO.IOException' in System.Net.Sockets.dll, System.Net.Security.dll, System.Private.CoreLib.dll 10. 'System.InvalidOperationException' in Unity.Container.dll. No public constructor is available for type Microsoft.Extensions.Options.IPostConfigureOptions`1[Microsoft.Extensions.Logging.LoggerFilterOptions] (and for other interfaces, about 60 of them). Inner Exception: InvalidRegistrationException: Exception of type 'Unity.Exceptions.InvalidRegistrationException' was thrown. 11. System.IO.IOException in System.Net.Security.dll, System.Private.CoreLib.dll. Received an unexpected EOF or 0 bytes from the transport stream. (5) 12. System.IO.IOException: 'Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..'

But despite of all the exceptions, the bot starts: Microsoft.Hosting.Lifetime: Information: Now listening on: http://[::]:9442 Microsoft.Extensions.Hosting.Internal.Host: Debug: Hosting started. And the GET health check request from Postman hits Ngrok, hits the bot, and returns 200 OK to the Postman. During the bot initialization, the BotMediaStream is created, the audio socket is initialized, the bot sends status active for media, and the media player is created.

Testing; - We start the bot in Visual Studio (as administrator) locally. - We start the Teams client locally, in our sandbox tenant. - We start a private scheduled meeting in Teams (Calendar -> Schedule -> Join Meeting). - We use the meeting JoinURL for the bot to join the meeting call from Postman with a JSON POST request { "joinURL": <JOIN_URL> }. - It hits Ngrok, hits the bot, the bot returns 200 OK and reports that it joined the call: CallController: Joined the call, call ID: 1e005980-8e2c-4831..., URL: https://teams.microsoft.com/l/meetup-join/19%3ameeting_NjZh... - The bot also receives two notifications via api/calling/notification endpoint after the join call request and replies with 202 Accepted: PlatformCallController: Notification result: StatusCode: 202, ReasonPhrase: 'Accepted', Version: 1.1, Content: System.Net.Http.EmptyContent. - We monitor the wire in WireShark for the media TCP ports.

The problem: It tries to establish the call: 17:01:28:954 CallHandler: Call status updated to Establishing. And then immediately terminates it: 17:01:29:954 CallHandler: Call status updated to Terminated - Server Internal Error. DiagCode: 500#1203002.@. And then throws the following exceptions: Exception thrown: 'System.Threading.Tasks.TaskCanceledException' in System.Private.CoreLib.dll A task was canceled. Exception thrown: 'System.IO.IOException' in System.Net.Sockets.dll Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request. THE BOT DOES NOT APPEAR IN THE MEETING, and NO MEDIA IS SENT TO THE BOT.

Questions: 1. All the exceptions are triggered deeply in the native libraries. Can it mean that there are some versioning issues? 2. What can cause the exceptions in the bot during the initialization? Are those exceptions significant? 3. Can those exceptions cause that the bot is not joining the call and also the media stream issues? 4. Is it OK to use a self-signed certificate? 5. What would you recommend to further debug the issue?

Thank you!

genemgh avatar Sep 11 '24 22:09 genemgh

I'll start with question 4 it is not ok to use a self signed certificate. Regarding the other questions probably they resolve themself if you use a certificate with a certificate to the root chain. Maybe as tip with ngrok try to get a connection to 0.tcp.-endpoint of ngrok by restarting on the other endpoints there sometimes seem to be problems with the firewall settings from ngrok.

1fabi0 avatar Sep 13 '24 06:09 1fabi0

Thank you for the advice! We'll try a root chain certificate.

Our Ngrok is configured to generate the following two endpoints: TCP: tcp://N.tcp.ngrok.io:PORT -> localhost:8445, N = 0,2,4.6,8, PORT is a random int usually between 10000 and 30000. HTTP: https://DOMAIN.ngrok-free.app -> http://localhost:9442, DOMAIN is a random string like d7b9-63-209-137-19

We generate the certificates for domain N.tcp.ngrok.io, the TCP connection domain, no wildcard *. Is that correct?

=========================================================

Another possible reason of the issue: Our application registration and the Azure bot registration technically belong to two different tenants.

We have a sandbox available as a part of our Microsoft 365 Developer Subscription for Visual Studio Professional. So, the application is registered in the sandbox tenant (and it's multi-tenant). The Azure bot registration is a part of our organization tenant (but it's based on the application registration, and it's also multi-tenant). That's because the real Azure subscription is under our organization tenant as a part of 365 Subscription.

We run the Teams client from our sandbox tenant. During the call joining process, the bot does not ask for any admin consent. Can it (the registration in two different tenants) be the reason of the call termination?

=========================================================

And would anybody please confirm that it's supposed to work for .NET 6.0 :-)

Thank you!

genemgh avatar Sep 13 '24 19:09 genemgh

I think your tenant situation is not a problem as long as you gave admin permission for the teams operating tenant. Regarding the certificate for the tcp ngrok address, as it might be rather complicated to get a valid certificate for the ngrok domain you can use a cname on a custom domain that then points to e.g. 0.tcp.ngrok.io and get the certificate for your custom domain(this implies using the custom domain instead of the ngrok domain in the bot).

Yes .Net 6 works, but hosting with IIS doesn't

1fabi0 avatar Sep 13 '24 21:09 1fabi0

Thank you for your help! We'll try it.

A deployment question. Does it have to be deployed in Azure VMSS? Is it possible to deploy it on-premises on a Windows Server in Kubernetes?

genemgh avatar Sep 16 '24 03:09 genemgh

Yes, it is possible to deploy it on-premises in a Kubernetes cluster, but keep in mind it is not officially supported by Microsoft. It is recommended to run the bot as close as possible to the Azure datacenter location where the teams meeting is hosted to reduce packet loss and roundtrip times.

However regarding Kubernetes, you can have a look at the Azure Kubernetes Service Sample that I also worked on. The k8s charts and deployment steps should be compatible/portable.

1fabi0 avatar Sep 17 '24 07:09 1fabi0

In our case, we'll have to redirect the media stream to our on-premises platform for processing and analysis anyway, so the distance is not very important. Anyway, thank you for the advice, we'll consider it.

genemgh avatar Sep 18 '24 01:09 genemgh

BTW, is it possible to use HTTPS for the media stream instead of TCP? Web Sockets, for example? Thank you!

genemgh avatar Sep 19 '24 00:09 genemgh

No, it seems like the media platform is somewhere based on WCF nowadays core WCF so you can not replace the TCP endpoint with websockets, hopefully Microsoft will refactor the media platform one day and pay some technical debt and remove the dependency to WCF.

1fabi0 avatar Sep 19 '24 07:09 1fabi0

Finally, we have our root chain certificate issued and the DNS cname records added. It works now (in the non-debug mode). Thank you very much for your help!

In the debug mode it triggers the same exceptions. Is it supposed to work in the debug mode? It would provide a lot of useful information...

Does the audio stream provide the participant diarization?

Thank you!

genemgh avatar Sep 30 '24 22:09 genemgh

Our real-time media bot throws the following exception. Could anybody help us to explain what we do wrong? The MS servers sometimes forcibly closes connection. (The bot is able to join a Teams meeting and it receives the audio.) Thank you!

[24-12-04 17:02:56.5681] fail: PTBotService.Bot.MediaLogger[0] [AvMP][AppId:63b1acf6-88cb-4b73-a92a-e9eb26872450] TL_ERROR(TF_COMPONENT) 30684.122::12/04/2024-17:02:56.567.00000009 (MPAZAPPHOST,LogError:AzAppMPHostLogger.cs(39))

[EventManager] Sending of notification message failed to Uri : https://002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701/in_7/media/v2/mpproxy/1/mpcontext/9c869ec7-5161-4b0c-9ed5-67e79cf96eeb/mpEventPostback for HttpMethod: POST

with exception : System.AggregateException: One or more errors occurred. (The SSL connection could not be established, see inner exception.) (The SSL connection could not be established, see inner exception.) (The SSL connection could not be established, see inner exception.) ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.. ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
at System.Net.Security.SslStream.<FillHandshakeBufferAsync>g__InternalFillHandshakeBufferAsync|189_0[TIOAdapter](TIOAdapter adap, ValueTask1 task, Int32 minSize) at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter) at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request) at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.LoggingMessageHandler.SendAndLogAsync(HttpRequestMessage request, CancellationToken cancellationToken, String msgId, String localMsgId)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.LoggingMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.MessagesIdentificationMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.TaskExtensions.RunTaskFunc[T](Func1 taskFunc, CancellationTokenSource cts, TaskCompletionSource1 tcs, Clock clock, ILogSource log, String taskDescription)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.AsyncRetryMessageHandler.<>c__DisplayClass7_0.<<SendAsync>b__0>d.MoveNext() --- End of stack trace from previous location ---
at Microsoft.Rtc.MPService.Internal.ExternalLibs.RetryUtil.RunWithRetries[T](IAsyncRetryPolicy1 policy, Func1 retryableMethod, Clock clock, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Microsoft.Rtc.MPService.Internal.ExternalLibs.RetryUtil.RunWithRetries[T](IAsyncRetryPolicy1 policy, Func1 retryableMethod, Clock clock, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.ContextForwardingMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationTokenoriginalCancellationToken)
at Microsoft.Rtc.Internal.MediaProcessor.WebAPI.MtlsHttpRequestSender.InternalSendAsync[T](T message, Uri requestUri, HttpMethod httpMethod, Uri implicitCallbackLink, String contentType, String authToken)
at Microsoft.Rtc.Internal.MediaProcessor.WebAPI.MtlsHttpRequestSender.SendAsync[T](T message, Uri requestUri, String authToken, HttpMethod httpMethod)
at Microsoft.Rtc.Internal.MediaProcessor.WebAPI.EventManager.SendNotificationAsync[T](T notificationMessage, Uri requestUri, HttpMethod httpMethod, Boolean sendNotificationUsingCompositeTransport, Uri implicitCallbackLink, String authToken) ---> (Inner Exception #1) System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.. ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
at System.Net.Security.SslStream.<FillHandshakeBufferAsync>g__InternalFillHandshakeBufferAsync|189_0[TIOAdapter](TIOAdapter adap, ValueTask1 task, Int32 minSize) at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter) at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request) at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.LoggingMessageHandler.SendAndLogAsync(HttpRequestMessage request, CancellationToken cancellationToken, String msgId, String localMsgId)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.LoggingMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.MessagesIdentificationMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.TaskExtensions.RunTaskFunc[T](Func1 taskFunc, CancellationTokenSource cts, TaskCompletionSource1 tcs, Clock clock, ILogSource log, String taskDescription)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.AsyncRetryMessageHandler.<>c__DisplayClass7_0.<<SendAsync>b__0>d.MoveNext() --- End of stack trace from previous location ---
at Microsoft.Rtc.MPService.Internal.ExternalLibs.RetryUtil.RunWithRetries[T](IAsyncRetryPolicy1 policy, Func1 retryableMethod, Clock clock, CancellationToken cancellationToken)<--- ---> (Inner Exception #2) System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.. ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
at System.Net.Security.SslStream.<FillHandshakeBufferAsync>g__InternalFillHandshakeBufferAsync|189_0[TIOAdapter](TIOAdapter adap, ValueTask1 task, Int32 minSize) at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter) at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request) at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.LoggingMessageHandler.SendAndLogAsync(HttpRequestMessage request, CancellationToken cancellationToken, String msgId, String localMsgId)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.LoggingMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.MessagesIdentificationMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.TaskExtensions.RunTaskFunc[T](Func1 taskFunc, CancellationTokenSource cts, TaskCompletionSource1 tcs, Clock clock, ILogSource log, String taskDescription)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.AsyncRetryMessageHandler.<>c__DisplayClass7_0.<<SendAsync>b__0>d.MoveNext() --- End of stack trace from previous location ---
at Microsoft.Rtc.MPService.Internal.ExternalLibs.RetryUtil.RunWithRetries[T](IAsyncRetryPolicy1 policy, Func1 retryableMethod, Clock clock, CancellationToken cancellationToken)<---

genemgh avatar Dec 05 '24 01:12 genemgh

@genemgh a few questions:

  1. What nuget version of Microsoft.Skype.Bots.Media are you running?
  2. What target framework (is it net framework or net6.0?)
  3. What version of TLS / SSL are you configured with on your machine?

adityaramgopal avatar Dec 05 '24 17:12 adityaramgopal

Hi @adityaramgopal ! Answering you questions:

  1. Microsoft.Skype.Bots.Media version: 1.27.0.2.alpha
  2. It’s Net 6.0.
  3. TLS 1.2 and TLS 1.3 are enabled on my machine (Windows 11) Thank you!

genemgh avatar Dec 05 '24 18:12 genemgh

@genemgh do you have any firewall rules on outbound traffic?

adityaramgopal avatar Dec 05 '24 18:12 adityaramgopal

@adityaramgopal In fact, I do. Our IT team manages them. Which port should be open to communicate with the MS media server? The media we need from a Teams meeting comes through to our local bot, so it looks like it does not happen every time... Firewall would block all the media. Is that correct?

genemgh avatar Dec 05 '24 19:12 genemgh

@genemgh can you temporarily disable all outbound firewall rules and see if the exception is thrown? Just to clarify we're asking about disabling outbound firewall rules (not inbound)

adityaramgopal avatar Dec 05 '24 19:12 adityaramgopal

I'm not supposed to do it. And it's a security risk.

genemgh avatar Dec 05 '24 20:12 genemgh

without eliminating that as the reason it's going to be hard to figure out the root cause.

adityaramgopal avatar Dec 05 '24 20:12 adityaramgopal

It's an HTTPS request which goes through port 443, which is always open. So, I don't think it's firewall. And usually there are better ways to debug than disabling your firewall :-) WireShark? Thank you anyway.

genemgh avatar Dec 05 '24 21:12 genemgh

Are you saying that you allow all outbound traffic to port 443 and don't constrain them to specific IP addresses?

adityaramgopal avatar Dec 05 '24 21:12 adityaramgopal

The media we need from a Teams meeting comes through to our local bot, so it looks like it does not happen every time...

Also, can you clarify what you mean by this comment. This issue reproduces sometimes but not always? Or does it reproduce in a specific machine (say local) but not in another machine?

adityaramgopal avatar Dec 05 '24 21:12 adityaramgopal

What I mean is that I'm not sure how often that http request fails (on one single machine). And I'm not sure how this request is related to the media stream. But we are getting the media stream, so it means that the request is either not critical or does not fail every time. Do you know what that request does and how it's related to the media stream?

genemgh avatar Dec 05 '24 21:12 genemgh

To my understanding, if an outbound firewall rule blocks an outgoing request, the request won't be sent. In our case, it looks like it's reaching the remote http server, and the remote host closes the connection. So, it does not look like a firewall issue.

genemgh avatar Dec 05 '24 21:12 genemgh

@genemgh can you share a sample call id of a successful call vs a failing call? Might help us debug further. Maybe a wireshark trace of a failing call might also help

adityaramgopal avatar Dec 05 '24 22:12 adityaramgopal

In order to reproduce it, we tried to hit the same URL directly from Curl in the verbose mode (both from Windows and WLS): Windows> curl -vv -d '{}' -H "Content-Type: application/json" https://002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701/in_7/media/v2/mpproxy/1/mpcontext/9c869ec7-5161-4b0c-9ed5-67e79cf96eeb/mpEventPostback

  • Host 002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701 was resolved.
  • IPv6: (none)
  • IPv4: 172.171.152.231
  • Trying 172.171.152.231:10701...
  • Connected to 002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com (172.171.152.231) port 10701
  • schannel: disabled automatic use of client certificate
  • ALPN: curl offers http/1.1
  • Recv failure: Connection was reset
  • schannel: failed to receive handshake, SSL/TLS connection failed
  • closing connection #0 curl: (35) Recv failure: Connection was reset

WSL Ubuntu:~$ curl -vv -d '{}' -H "Content-Type: application/json" https://002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701/in_7/media/v2/mpproxy/1/mpcontext/9c869ec7-5161-4b0c-9ed5-67e79cf96eeb/mpEventPostback

  • Trying 172.171.152.231:10701...
  • Connected to 002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com (172.171.152.231) port 10701 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • CAfile: /etc/ssl/certs/ca-certificates.crt
  • CApath: /etc/ssl/certs
  • TLSv1.0 (OUT), TLS header, Certificate Status (22):
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • OpenSSL SSL_connect: Connection reset by peer in connection to 002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701
  • Closing connection 0
  • TLSv1.0 (OUT), TLS header, Unknown (21):
  • TLSv1.3 (OUT), TLS alert, decode error (562): curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to 002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701

WSL Ubuntu:~$ curl -k -vv -d '{}' -H "Content-Type: application/json" https://002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701/in_7/media/v2/mpproxy/1/mpcontext/9c869ec7-5161-4b0c-9ed5-67e79cf96eeb/mpEventPostback

  • Trying 172.171.152.231:10701...
  • Connected to 002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com (172.171.152.231) port 10701 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • TLSv1.0 (OUT), TLS header, Certificate Status (22):
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • OpenSSL SSL_connect: Connection reset by peer in connection to 002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701
  • Closing connection 0
  • TLSv1.0 (OUT), TLS header, Unknown (21):
  • TLSv1.3 (OUT), TLS alert, decode error (562): curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to 002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701

WSL Ubuntu:~$ telnet 172.171.152.231 10701

  • Trying 172.171.152.231...
  • Connected to 172.171.152.231.
  • Escape character is '^]'.
  • Connection closed by foreign host.

genemgh avatar Dec 23 '24 14:12 genemgh

It was a VPN issue, fixed now.

genemgh avatar Jan 15 '25 15:01 genemgh

We still have a couple of roadblocks, and we tried to submit a paid ticket to Microsoft Developer Support CS via our MS VS subscription. We went through 3 different MS CS agents, and it all ended with a link Contact Us - Developer Support. The link allows only an email connection. We've sent a question to them, but no reply so far.

Is there any way to submit such a ticket?

Thank you!

genemgh avatar Jan 15 '25 15:01 genemgh

It looks like the MS developer support is not in a hurry. I'd like to post the same message here.

genemgh avatar Jan 16 '25 16:01 genemgh

Hello,

We are working on a Microsoft Teams bot which is supposed to obtain a real-time audio stream from a Teams meeting and redirect it to our platform. In addition to the audio stream, our platform also requires some meeting information. Our platform performs a real-time analysis, and the result of the analysis is supposed be displayed in the meeting. Currently, we use a Developer-365 sandbox based on our Visual Studio Professional subscription.

The implementation consists of the front-end and the back-end service. The front-end is a Teams plug-in application which contains both a tab and a bot. It's implemented in Visual Studio Code + Teams Toolkit in Type Script + React, and it's based on the botbuilder 4.22.2 library. The tab part of the Teams application is responsible for displaying our platform analysis results. The bot part catches the real-time meeting events and also sends the join call request containing the meeting JoinUrl to the back-end.

The back-end is running on our platform server, it’s implemented in Visual Studio in C#, Net 6.0 and uses the Microsoft Media Platform. The back-end service sends the audio packets via a web socket to our platform. It also sends the following events: call joined, call ended, participant added, participant removed.

We encountered the following 3 issues which prevent us from moving forward:

================================================ 1.

In order to get the JoinUrl (and some other info), we call await TeamsInfo.getMeetingInfo (context, meetingId) in the front-end application bot, where meetingId = context.activity.channelData.meeting.id. That function call throws exception "Error in Microsoft Teams botbuilder TeamsInfo.getMeetingInfo: bot not allowed to get meeting details".

It looks like a permission issue. We tried to resolve it. The following code is added to the local application manifest: "authorization": { "permissions": { "resourceSpecific": [ { "name": "OnlineMeeting.ReadBasic.Chat", "type": "Application" }, { "name": "OnlineMeeting.ReadBasic.Group", "type": "Application" }, { "name": "OnlineMeetingParticipant.Read.Chat", "type": "Application" } ] } },

Also, we registered the application in Microsoft Entra for our sandbox tenant. The application id is the same as the bot id in the application env.local file: BOT_ID=cbc9003c-1b5a-4c71-9c64-add26aa604d6. The following permissions are added and the admin consent for them is granted for that application registration:

OnlineMeeting.Read

OnlineMeeting.Read.All

CallEvents.Read

CallEvents.Read.All

User.Read

User.Read.All

...

QUESTION: What else are we supposed to do to fix the exception in TeamsInfo.getMeetingInfo call?

================================================ 2.

For the back-end bot service, we created another application registration and a separate bot registration in the Azure Portal (based on that application). We currently use Postman to send the join call request to the bot service. The back-end bot service successfully joins the meeting and receives the audio stream. It also receives the call and participant update notifications. There are 3 different participant ids available in the back-end bot service in IParticipant participant:

participant.id. It's a UUID string like 30845b4c-fee6-4e53-b0bb-18673f18d398

participant.Resource.Info.ParticipantId. It's also a UUID string, but not equal to participant.id

the participant id from the audio stream. It's a simple integer like 201

QUESTION: Which of those ids are correct? How can we establish the correspondence between the participant id from the audio stream and another (correct) one?

================================================ 3.

We are able to get the participant ids and their display names in the participant update notification callbacks. But it’s possible that some participants joined the meeting before the bot service joined it. In such a case, those participants won’t appear in the callbacks.

QUESTION: Is it possible to get the list of participants (with their ids and display names) who already joined the meeting via the API in the back-end service bot?

Thank you for your help.

genemgh avatar Jan 16 '25 16:01 genemgh

Hi any luck on this? @genemgh your use case is exactly like ours.

joelAbulencia avatar Jul 17 '25 04:07 joelAbulencia

Hey @genemgh To answer your 2nd question, the id ( activeSpeakerId ) you get from the AudioSocket.AudioMediaReceived is simply the mediasourceid. (It is not participant id) To map the participant with the media source id, you have to get the media source id when the participant onUpdated event triggers. participant?.Resource?.MediaStreams -> which gives you the list of media source id. You can then use this id to map the dominant speaker id, in this way you'll be able to know which audio belongs to which participant.

Butcher-vj avatar Jul 18 '25 10:07 Butcher-vj