Creating too many connections to the Exchange Server

[viktdm]

I am implementing a .NET application with the EWS Managed API and have decided to port it to .NET Core. I've learned that the official Microsoft EWS Managed API library doesn't currently support .NET Core and found this fork, which does.

Unfortunately, while testing my application with Microsoft.Exchange.WebServices.NETStandard.dll, I noticed that it would start failing with "Connection with the server could not be established" errors after running for less than few mins. (There are no problems with the official Microsoft.Exchange.WebServices.dll under .NET Framework, though). Further investigation revealed that the application was creating a lot of network connections and leaving many of them in TIME_WAIT state. Using .NET Core 2.0 or 2.1 as the target framework didn't make a difference.

Looking at EWS Managed API source code and reading various StackOverflow posts, it turns out that this is actually .NET Core to blame. Microsoft.Exchange.WebServices.NETStandard.dll is still using old HttpWebRequest and HttpWebResponse classes, which are NOT reusing network connections in .NET Core as they would do in .NET Framework. It is sad to see that Microsoft doesn't plan to fix this issue (https://github.com/dotnet/corefx/issues/26373) and recommends to move to HttpClient.

Because of this problem, Microsoft.Exchange.WebServices.NETStandard.dll is not really suitable for applications working with lots of user mailboxes and/or items. It will practically cause DDoS attack against Exchange Server.

It seems that Microsoft.Exchange.WebServices.NETStandard.dll is already using HttpClient in Autodiscovery, but the core logic is still implemented with HttpWebRequest and HttpWebResponse classes. How difficult would it be to scrape HttpWebRequest/HttpWebResponse off entirely and switch to HttpClient?

[sherlock1982]

This is related to #25. I think I had a branch cleanup_netrequests which was targeting this issue. But there are too much things inside to check. I will work on it when I have time - will try next week.

[viktdm]

Thanks! I'll take a look at that branch and try to contribute with fixing and testing this issue.

[sherlock1982]

Switched to HttpClient internally.

[viktdm]

Thanks! I am going to give it a try.

[sherlock1982]

If this doesn't help than there's one more change. Currently HttpClient created/disposed per request and that's how EWS uses it. Theoreticaly I can try to create one HttpClient per service for example. Than it will be much better but internally EWS is not that designed for it. So it might be a bigger breaking change.

[viktdm]

Unfortunately, there are no significant improvements with the new version. When HttpClient is created/disposed, a new connection is created/closed. So, it results same outcome--when an app makes a lot of bind requests, there are many connections left in TIME_WAIT state. With the original Microsoft EWS API library and .NET Framework, HTTP connections are properly reused and this is apparently done by/inside ServicePointManager.

Instead of creating one HttpClient per Exchange service, I'd rather consider using HttpClientFactory (see https://docs.microsoft.com/en-us/dotnet/standard/microservices-architecture/implement-resilient-applications/use-httpclientfactory-to-implement-resilient-http-requests). It may require to bring DI, though.

[sherlock1982]

That's a very intresting case. I never thought of it in this way. Though I don't think DI is needed here. What can help is to reuse HttpClient per service for example. We will need to check usage of Cookies and other params and probably do it. Than the whole code will become easier.

I don't really understand how a connection can be reused. What one will need to do is to keep it alive. How reusing HttpClient can improve situation I don't see so far. I need to read more about it.

[viktdm]

BTW, below is the sample code demonstrating the problem.

When you build and run it with the orginal Microsoft.Exchange.WebServices.dll and .NET Framework, the number of established connections to the server (checked with netstat) remains 1 all the time.

When you build and run it with Microsoft.Exchange.WebServices.NETStandard.dll and .NET Core (EWS_MANAGED_API_NET_STANDARD is defined), then the number of established connections to the server is still 1, but the number of connections in TIME_WAIT is increasing and gets above thousands even before the test completes.

This is of course just a simple test against one user mailbox. You can imagine how bad things may go when your app/service needs to handle hundred, if not thousand, mailboxes.

using System;
using System.Net;
using Microsoft.Exchange.WebServices.Data;

namespace ews_stress
{
    class Program
    {
        private static int _maxNumTests = 100;
        private static int _maxPageSize = 100;

        static void Main(string[] args)
        {
            ExchangeService service = new ExchangeService(ExchangeVersion.Exchange2013_SP1);
            service.Credentials = new NetworkCredential("<user_email_address>", "<user_password>");
            service.Url = new Uri("https://<exchange_server_adress>/EWS/Exchange.asmx");

            for (int test = 0; test < _maxNumTests; test++)
            {
                ItemView itemView = new ItemView(_maxPageSize);
                itemView.PropertySet = BasePropertySet.IdOnly;
                itemView.Traversal = ItemTraversal.Shallow;

#if EWS_MANAGED_API_NET_STANDARD
                FindItemsResults<Item> findResults = service.FindItems(WellKnownFolderName.Inbox, itemView).Result;
#else
                FindItemsResults<Item> findResults = service.FindItems(WellKnownFolderName.Inbox, itemView);
#endif          
                foreach (Item item in findResults)
                {
#if EWS_MANAGED_API_NET_STANDARD
                    Item itemObject = Item.Bind(service, item.Id, BasePropertySet.FirstClassProperties).Result;

#else
                    Item itemObject = Item.Bind(service, item.Id, BasePropertySet.FirstClassProperties);
#endif
                    Console.WriteLine("Item subject: {0}", itemObject.Subject);
                }
            }
        }
    }
}

[tjmoore]

Noticed this myself looking at the code and the way it's using HttpClient now.

The .NET Core application I'm working on has a similar issue using HttpClient itself but it also uses EWS and would ideally be using the same IHttpClientFactory, e.g. https://docs.microsoft.com/en-us/aspnet/core/fundamentals/http-requests?view=aspnetcore-5.0

Maybe there needs to be an option to pass in the factory or HttpClient. Don't need DI for IHttpClientFactory as where I'm instantiating the EWS library is not in the scope of DI but I can use that in my app's service layer and pass something down to EWS.

The app I'm working on has regular polls to EWS and looking at TcpView, there are a lot of TIME_WAIT sockets to the Exchange server.

[harry-flw]

I have multithreading application which synchronizes MS SharePoint data about organizations with business contacts in user's folders of MS Exchange.

It works fine when I use 1 or 2 threads (1 or 2 user's folders). If number of threads more than 2 I constantly have an error "Too many concurrent connections opened."

I do POST requests duly completed as SOAP and use IHttpClientFactory to create a HttpClient.

I always dispose every HttpRequestMessage and HttpClient.

[tjmoore]

An alternative to IHttpClientFactory if you use long-lived HttpClient and don't have DI easily available is to simply use static or singleton instances but need to set PooledConnectionLifetime on them. https://learn.microsoft.com/en-us/dotnet/fundamentals/networking/http/httpclient-guidelines

Meanwhile, is anyone looking at this? I haven't got around to it myself, but currently looking at HttpClient use throughout the product I'm working on so may need to have a look, but I'm not that familiar with the EWS library code.