How can I implement service-to-service authentication with Office365 using the OAuth2 SASL mechanism?

[DGrahamFC]

Hi,

I am trying to send email from a server application via Office 365 using OAUTH2 using MailKit with the client credentials flow. I have tried to resolve for several days so as not to waste your time - but am now admitting defeat. Here's the code I have so far:

        SmtpClient result = new SmtpClient(new ProtocolLogger(Console.OpenStandardOutput()));
        result.SslProtocols = SslProtocols.Tls12;
        result.CheckCertificateRevocation = false;

        result.Connect(smtpSettings.ServerAddress, smtpSettings.Port, SecureSocketOptions.StartTls);

        result.AuthenticationMechanisms.Clear();
        result.AuthenticationMechanisms.Add("XOAUTH2");

        var app = ConfidentialClientApplicationBuilder.Create("a client id here")      // client id of application
                                                  .WithClientSecret("a client secret here")  // client secret
                                                  .WithAuthority(new Uri("https://login.microsoftonline.com/{tenant id here}/"))   // tenant id is guid at the end
                                                  .Build();
        var scopes = new List<string>();
        scopes.Add("https://graph.microsoft.com/.default");

        var threadResult = app.AcquireTokenForClient(scopes).ExecuteAsync();
        var token = threadResult.Result;

        var oauth2 = new SaslMechanismOAuth2("[email protected]", token.AccessToken);
        result.Authenticate(oauth2);    // error thrown here

The protocol log looks like this and shows the error at the end

Connected to smtp://smtp.office365.com:587/?starttls=always S: 220 LO4P123CA0060.outlook.office365.com Microsoft ESMTP MAIL Service ready at Mon, 4 Jan 2021 16:29:08 +0000 C: EHLO [127.0.0.1] S: 250-LO4P123CA0060.outlook.office365.com Hello [176.248.34.216] S: 250-SIZE 157286400 S: 250-PIPELINING S: 250-DSN S: 250-ENHANCEDSTATUSCODES S: 250-STARTTLS S: 250-8BITMIME S: 250 SMTPUTF8 C: STARTTLS S: 220 2.0.0 SMTP server ready C: EHLO [127.0.0.1] S: 250-LO4P123CA0060.outlook.office365.com Hello [176.248.34.216] S: 250-SIZE 157286400 S: 250-PIPELINING S: 250-DSN S: 250-ENHANCEDSTATUSCODES S: 250-AUTH LOGIN XOAUTH2 S: 250-8BITMIME S: 250 SMTPUTF8 C: AUTH XOAUTH2 dXNlcj1kYW5ueS5ncmFoYW1AZmFzdGNsb3NlLnVrAWF1dGg9QmVhcmVyIGV5SjBlWE.....etc.....0hxdUhYdwEB S: 451 4.7.0 Temporary server error. Please try again later. PRX4 [LO4P123CA0060.GBRP123.PROD.OUTLOOK.COM]

The application permissions are set up in Azure as below

I'm probably doing (or more likely not doing) something stupid - but I just can't figure it - thanks in advance for your time.

[jstedfast]

You could try:

var scopes = new string[] {
    "email",
    "offline_access",
    "https://outlook.office.com/SMTP.Send"
};

[DGrahamFC]

Hi there - thanks for coming back.

I now get this: One or more errors occurred. (AADSTS70011: The provided request must include a 'scope' input parameter. The provided value for the input parameter 'scope' is not valid. The scope email offline_access https://outlook.office.com/SMTP.Send is not valid. Trace ID: c0e45afa-9207-4b32-b0e1-7f2cbd881401 Correlation ID: 47d32171-7c4a-4f73-9122-2737936f44a3 Timestamp: 2021-01-05 17:12:34Z)

[jstedfast]

What about just "https://outlook.office.com/SMTP.Send" ?

[DGrahamFC]

Very similar error:

One or more errors occurred. (AADSTS70011: The provided request must include a 'scope' input parameter. The provided value for the input parameter 'scope' is not valid. The scope https://outlook.office.com/SMTP.Send is not valid. Trace ID: bc2c8838-07a6-4593-b3b7-5ea12efc5d00 Correlation ID: 823de67a-fe8d-46c8-8ac6-eaf71907b68c Timestamp: 2021-01-05 17:38:01Z)

One of the things that I think is unusual is that I've been trying to setup the permissions using application permissions rather than delegated permissions, like this: on the assumption that the application would have the rights to send an email from whatever "from" address is passed, which is what the description on the "Mail.Send" permissions implies.

[jstedfast]

Oh, I see... I've never looked into this particular OAuth2 type.

[DGrahamFC]

Ok. Thanks for your time with this Jeffrey, I really appreciate it.

So taking a step back, this is the use case:

We have a server product that needs the ability to periodically send emails in an unattended fashion when certain conditions arise. In that scenario it would want to send an email from the classic "[email protected]" address to some random set of users and we have to cope with the scenario that the outbound mailserver is provided by Office 365.

Since Microsoft have now retired basic auth on 365 we have to support OAUTH to do this.

Am I using the correct flow? Or are you aware of an alternate that I could use to meet this need.

Thanks for your help.

[jstedfast]

What you are trying to do makes perfect sense to me. I'm just not sure if it's the correct flow or not. I've only figured this out for use from a "mail client" perspective.

[DGrahamFC]

Thanks Jeff - good to know I'm not going mad!

I understand this is a side project so time isn't always easy to come by. That said, do you think this is something you would look to investigate ie: falls into the intended scope of the library? And if so - any idea as to a timeline?

I fully expect to be told I've messed up some code or configuration - but I think even that would be really useful documentation for people coming after.

Many thanks.

[jstedfast]

It's definitely something that is within the intended scope.

[punthoofd07]

Hi,

I've been using MailKit for a long time now and it's great. So first of all, a big thanks!

Recently having issues with configuring new Office 365 Tenants to use app passwords. I always get it working, but I'm tired of setting the app passwords, so looking for a complete OAuth2 setup inside my application. Working with delegate permissions works, but I prefer settings and forgetting with application permissions.

As mentioned by the OP, I also tried using the confidential client (with app Id, Tenant Id and a secret). I can acquire a token, and use this token: var oauth2 = new SaslMechanismOAuth2("email address", authToken.AccessToken);

But then at: await client.AuthenticateAsync(oauth2);

It fails saying authentication unsuccessful.

Time, time, time! We need more of it :-)

[bojanmisic]

I'm having the exact same issues, has any of you had luck of setting this up successfully?

[Jedrzej94]

Seems like I was referring to this issue in another one. Let's roll into this one.

My latest reply: https://github.com/jstedfast/MailKit/issues/989#issuecomment-759693524

[Neustradamus]

I follow this ticket...

[Jedrzej94]

From my side I'd like to post an update that I've decided to abandon Mailkit and move onto Full-Graph-Client library with Graph Auth.

[crumpsbrother]

I seem to have the same issue. I'm following the method described here... https://docs.microsoft.com/en-us/graph/auth-v2-service I am creating the SaslMechanism like this... await client.AuthenticateAsync(new SaslMechanismOAuth2("email address", access_token)); and I always receive a 535: 5.7.3 Authentication unsuccessful response. Since the token is generated on the application and not the user I am just using any account that is in the AD. I don't know if that is correct though.

log is as follows...

Connected to smtp://smtp.office365.com:587/?starttls=always S: 220 MN2PR08CA0019.outlook.office365.com Microsoft ESMTP MAIL Service ready at Wed, 3 Feb 2021 20:58:54 +0000 C: EHLO [192.168.39.154] S: 250-MN2PR08CA0019.outlook.office365.com Hello [72.137.218.162] S: 250-SIZE 157286400 S: 250-PIPELINING S: 250-DSN S: 250-ENHANCEDSTATUSCODES S: 250-STARTTLS S: 250-8BITMIME S: 250-BINARYMIME S: 250-CHUNKING S: 250 SMTPUTF8 C: STARTTLS S: 220 2.0.0 SMTP server ready C: EHLO [192.168.39.154] S: 250-MN2PR08CA0019.outlook.office365.com Hello [72.137.218.162] S: 250-SIZE 157286400 S: 250-PIPELINING S: 250-DSN S: 250-ENHANCEDSTATUSCODES S: 250-AUTH LOGIN XOAUTH2 S: 250-8BITMIME S: 250-BINARYMIME S: 250-CHUNKING S: 250 SMTPUTF8 C: AUTH XOAUTH2 dXNlcj1kZXZAcG9j etc...

edit (sorry just noticed the error didn't post)

S: 535 5.7.3 Authentication unsuccessful [MN2PR08CA0019.namprd08.prod.outlook.com] Exception thrown: 'MailKit.Security.AuthenticationException' in mscorlib.dll 535: 5.7.3 Authentication unsuccessful [MN2PR08CA0019.namprd08.prod.outlook.com]

any insight would be helpful. thanks

[mantas-amberlo]

@crumpsbrother your error is different.. but for "451 4.7.0 Temporary server error. Please try again later. PRX4"

I have an answer: if you are trying to connect to Office365 Enterprise account with MFA enabled - forget about it, it will not work unless you disable a lot of security related stuff.

See NOTE in Option 1: https://docs.microsoft.com/en-us/exchange/mail-flow-best-practices/how-to-set-up-a-multifunction-device-or-application-to-send-email-using-microsoft-365-or-office-365

[fszewczy]

I'm receiving same error while trying to authenticate using IMAP or POP3.

It seems that scopes like Mail.ReadWrite, Mail.Sand etc. are restricted for use with the Graph API. Permissions for IMAP, SMTP, and POP3 are only visible in Delegated permissions, no such options in Application permissions.

Then the only way is to receive and send messages as a daemon is using the Graph API.

[mconca-kube]

I'm also trying to download mail from a background process (POP/IMAP). I followed this issue: https://github.com/jstedfast/MailKit/issues/989 I came up with this code:

using MailKit;
using MailKit.Net.Imap;
using MailKit.Security;
using Microsoft.Identity.Client;
using System;
using System.Threading.Tasks;

namespace CAppHarvestEmail
{
	class Program3
    {
        static async Task Main(string[] args)
        {
            var scopes = new string[] {
                //"email",
                //"offline_access",
                //"https://outlook.office.com/IMAP.AccessAsUser.All", // Only needed for IMAP
				//"https://outlook.office.com/POP.AccessAsUser.All",  // Only needed for POP
				//"https://outlook.office.com/SMTP.Send", // Only needed for SMTP
                "https://graph.microsoft.com/.default",
            };

            var confidentialClientApplication = ConfidentialClientApplicationBuilder
                    .Create(MySecrets.ClientID)
                    .WithClientSecret(MySecrets.ClientSecret)
                    .WithAuthority(new Uri("https://login.microsoftonline.com/" + MySecrets.TenantID + "/"))
                    .Build();

            var authenticationResult = await confidentialClientApplication.AcquireTokenForClient(scopes).ExecuteAsync();

            var authToken = authenticationResult;
	    var oauth2 = new SaslMechanismOAuth2(MySecrets.UserName, authToken.AccessToken);

	    using (var client = new ImapClient(new ProtocolLogger("imapLog.txt")))
            {
                client.Connect("outlook.office365.com", 993, SecureSocketOptions.SslOnConnect);
                //client.AuthenticationMechanisms.Remove("XOAUTH2");
                client.Authenticate(oauth2);
                var inbox = client.Inbox;
                inbox.Open(MailKit.FolderAccess.ReadOnly);
                Console.WriteLine("Total messages: {0}", inbox.Count);
                Console.WriteLine("Recent messages: {0}", inbox.Recent);
                client.Disconnect(true);
            }
        }
    }
}

This is the log:

S: * OK The Microsoft Exchange IMAP4 service is ready. [....==]
C: A00000000 CAPABILITY
S: * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+
S: A00000000 OK CAPABILITY completed.
C: A00000001 AUTHENTICATE XOAUTH2 dX.....==
S: A00000001 NO AUTHENTICATE failed.

Here are app permissions:

Authentication fails, but I really don't understand why...

[fszewczy]

@mconca-kube, If I understand correctly, in the above code you generate a token that allows you to connect to the mail via Graph API, and then you use it for authorization for IMAP. I did the same.

Summarizing, it seems to me that:

• We need to use Graph API to send/receive e-mails as an application (daemon) using 'Application permissions'. (https://docs.microsoft.com/en-us/graph/api/resources/mail-api-overview?view=graph-rest-1.0)
• If we want to use IMAP or SMTP we need to authenticate as user, using 'Delegated permissions' as described here: https://github.com/jstedfast/MailKit/blob/master/ExchangeOAuth2.md.

Can anyone confirm my assumptions?

[Jedrzej94]

Just out of curiosity... Is there any reason why you can't get your Graph authentication that way and then carry on with e-mail message management?

private GraphServiceClient GetGraphClient(Dictionary<string, object> systemParams)
		{
			string clientId = systemParams[Consts.AttrDef.InterfaceSystems.graph_interface_client_id].ToString();
			string tenantId = systemParams[Consts.AttrDef.InterfaceSystems.graph_interface_tenant_id].ToString();
			string clientSecret = Utils.Decrypt(systemParams[Consts.AttrDef.InterfaceSystems.graph_interface_clientsecret].ToString());

			IConfidentialClientApplication confidentialClientApplication = ConfidentialClientApplicationBuilder
				.Create(clientId)
				.WithTenantId(tenantId)
				.WithClientSecret(clientSecret)
			.Build();

			IAuthenticationProvider authProvider = new ClientCredentialProvider(confidentialClientApplication);

			return new GraphServiceClient(authProvider);
		}

[mconca-kube]

@fszewczy I'm trying to understand, but I feel like I'm very far off... I think you should be able to send/receive emails using one of these two ways.

1. PublicClient
2. ConfidentialClient

PublicClient is described here: https://github.com/jstedfast/MailKit/blob/master/ExchangeOAuth2.md This method requires you to manually enter your credentials when you run the app. These scopes/permissions (Delegated) must be defined:

• email
• offline_access
• IMAP.AccessAsUser.All

ConfidentialClient is not described in MailKit's documentation, and I tried to document it in my post above (failing...). This method requires you to create a client secret on your Azure App and use these data to authenticate. This method can be used in a daemon app and does not require user intervention. These scope/permission (Application) must be defined:

• https://graph.microsoft.com/.default I couldn't get it to work with any other scope. With my code I can generate the Oauth2 token, but then I cannot authenticate (I tried both .Net5 and .Net Core 3.1).

[mconca-kube]

@Jedrzej94 I tried, but got this error when requesting messages:

ServiceException: Code: ErrorAccessDenied
Message: Access is denied. Check credentials and try again.

[Jedrzej94]

@mconca-kube these are my AzureAD permissions (you don't need all of them - but for test - add these below)

Also, I hope you are aware of this website which helps you get your Azure AD permissions right, right? https://developer.microsoft.com/en-us/graph/graph-explorer

[fszewczy]

@Jedrzej94, that's what I did. We support many methods of managing e-mails, using known protocols is just more convenient, than implementing the usage of another one.

@mconca-kube, using SMTP, IMAP or POP3 is just not supported for ConfidentialClient. The error you get from the Graph client means you are referring to a mailbox that you do not have permission to - probably you need to specify a mailbox from which you want to retrieve messages. (https://docs.microsoft.com/en-us/graph/api/resources/mail-api-overview?view=graph-rest-1.0)

[mconca-kube]

@Jedrzej94 My permissions are the same. I didn't know about the graph explorer, thanks.

@fszewczy Specifying a mailbox worked, I thought "Me" was enough. Is ConfidentialClient Auth's support something that is missing on MailKit side or am I/we doing something wrong?

[crumpsbrother]

@Jedrzej94 I think the microsoft graph library is the way i am going to have to go. The only thing holding me back is the that we are currently on .net 4.5.2 and need to upgrade to 4.6.1 and that will probably take one of our release cycles to upgrade with testing etc. so I am in a holding pattern for now.

Thanks for your updates.

[jstedfast]

Seems someone else had this issue and asked about it on a Microsoft help forum: https://answers.microsoft.com/en-us/msoffice/forum/msoffice_o365admin-mso_other-mso_o365b/graph-url-scopes-not-supported-for-smtp-and-imap/4bb49947-f9af-4044-949b-0ab35b285069

They were redirected to https://docs.microsoft.com/en-us/answers/search.html?type=question&q=microsoft-graph-* to ask the question again.

That site is apparently a StackOverflow-like forum which is kinda cool. I tried searching, but it seems the question was not asked again there.

May be worth asking on that forum to see if anyone with knowledge of the Microsoft Graph APIs can answer.

[fabmoll]

Hello I'm trying too to authenticate without interactive mode and with my clientsecret clientId,... But on my side I've too the authentication failed message 😢 Do we need to move to Graph only code without MailKit?

[jstedfast]

@fabmoll sadly, it seems that might be the case.

If I find a way to do this with MailKit, I'll make a note here about it and add additional documentation to the ExchangeOAuth2.md documentation.

[fabmoll]

@jstedfast thank you for the information.