How to handle OData::ServerError: 503 ConcurrentItemSave ?

[giangth94]

Hello everyone! In my app, I use this line to create a new calendar event: @graph.service.post("me/calendars/#{my_calendar_id}/events", data.to_json) which works most of the time. But sometimes, the error tracking app (I use Sentry) send me the exception, and I am not able to re-produce this 503 error.

OData::ServerError: 503 ConcurrentItemSave: "Conflicts occurred during saving data to store (saveresult: IrresolvableConflict, properties: ). Please retry the request." from "https://graph.microsoft.com/v1.0/me/calendars/AAMkADg1YmMzNjE2LTM5Y2MtNGI2NC05NjJjLWRiY2RhZTZkNDRkMABGAAAAAADZc5aq6QclSqy_4oRMVv_EBwB4ZnW6S6xLTZDSXIsgJfprAAAAAAEGAAB4ZnW6S6xLTZDSXIsgJfprAAAGEcjmAAA=/events"

Every time this occurs, 2 same events are created. From HTTP status codes, I can see what the generic cause of the issue is, but I am having trouble debugging and handling this exception.

I would appreciate if anyone here could help me to debug the issue, thanks in advance!

I have also posted this on Stackoverflow, and haven't got the solution yet.

Update: When I look into the duplicate events using Graph Explorer, I found that they were created at very close timestamp, for example: "createdDateTime": "2019-02-18T04:55:07.9453147Z" "createdDateTime": "2019-02-18T04:55:24.9173342Z"

[jasonjoh]

Could you (or anyone else hitting this issue) provide details on the event you're creating? Are you adding open extensions to it by chance?

[giangth94]

@jasonjoh I create event with open extensions for our business need. The issue is still happening until now. Can you tell me what's matter with open extensions please?

This is how I create an event:

# The time vars are formatted and returned form a method
start_time, end_time = build_start_end 

data = {
  subject: "Event subject",
  body: { content: "Event body"  },
  start: { dateTime: start_time, timeZone: "Australia/Melbourne" },
  end: { dateTime: end_time, timeZone: "Australia/Melbourne" },
  attendees: attendees, # array of attendees here
  extensions: [{
    "@odata.type": "microsoft.graph.openTypeExtension",
    "extensionName": "MY_EXTENSION",
    "studentNumbers": 16
  }]
}

@graph.service.post("me/calendars/#{my_calendar_id}/events", data.to_json)

The approach works well in development environment, the issue is happening again and again on production. I keep receiving server error about concurrency, however, it seems like there's always an event created along with the error (my guess from what I've found so far, because I cannot reproduce the error)

[alexmaie]

Hey,

We are currently using the old Office 365 api to create events in the outlook calendar and also encountered this issue. Yes, we are using open extensions. Is this the appropriate location for this issue? We are not using the ruby sdk

Thanks Alex

[masterjs]

We also have this same issue (amongs many other). Since these are never really answered by the MS group (or on StackOverflow), I:

• Track the failing 503 with this specific error code
• Try to find an event with the same start/end/subject
• Then delete it
• Retry the post.

hope this helps until we get an answer! JS

[alexmaie]

@jasonjoh

This issues intensified over the last couple of days. Any news from your side?

[jasonjoh]

Yes, sorry for the delay. We believe we know what is causing this on our side. Basically, when you create an event with an open extension, the way that gets processed by the server is a two-step process: create the event, then update the event with the open extension. We've found an issue there where in some cases that two-step process is causing the conflict state.

While we're investigating this on our side, you can mitigate the problem by breaking your code into two separate POSTs. First create the event without the open extension, then POST your extension to the event.

Here's a simplified example. Let's say you're currently creating an event like this:

POST /me/events

{
  "subject": "My event",
  "start": {
    "dateTime": "2019-05-24T20:53:43.487Z",
    "timeZone": "UTC"
  },
  "end": {
    "dateTime": "2019-05-31T20:53:43.487Z",
    "timeZone": "UTC"
  },
  "extensions": [
    {
      "@odata.type":"microsoft.graph.openTypeExtension",
      "extensionName":"com.contoso.simple",
      "foo":"bar"
    }
  ]
}

Instead, create the event without the extension:

POST /me/events

{
  "subject": "My event",
  "start": {
    "dateTime": "2019-05-24T20:53:43.487Z",
    "timeZone": "UTC"
  },
  "end": {
    "dateTime": "2019-05-31T20:53:43.487Z",
    "timeZone": "UTC"
  }
}

Then using the id of the new event, POST your extension:

POST /me/events/{event-id}/extensions

{
  "@odata.type":"microsoft.graph.openTypeExtension",
  "extensionName":"com.contoso.simple",
  "foo":"bar"
}

I know that's not ideal (doing two POSTs instead of one) but it should avoid this issue while we investigate.

[giangth94]

@jasonjoh Thanks very much for the update. At least now we know how to avoid the issue. Hopefully we all can get notified from your side after the issue is fixed.

[alexmaie]

@jasonjoh We implemented the provided workaround, but the request to the extensions endpoint is also generating the error. It's through that at least we are creating the events correctly, but we can't really on the information in the extensions

Response message {"error":{"code":"ConcurrentItemSave","message":"Conflicts occurred during saving data to store (saveresult: IrresolvableConflict, properties: ). Please retry the request."}}

[jasonjoh]

@alexmaie thanks for the report. I'll forward to engineering.

[jasonjoh]

@alexmaie (or anyone hitting this with two separate requests): can you provide the full response headers? This will help our engineering folks correlate with their own logs to help the investigation.

Also, I'm wondering what happens here if you use the eTag of the event (returned when it's created) in an If-Match header on your second request?

[alexmaie]

Hey Jason,

We would need to deploy some code in production in order to log more information. This is difficult in the moment since the customer isn't that happy with "adhoc" publishes. If an opportunity appears to make some amendments, I will try to make the changes. Also, we can't reproduce this issue :(

[adamconway]

Hi @jasonjoh

We are also experiencing this issue when trying to use open extensions on calendar events. (using the dotnet sdk though).

Separating them into two separate post requests didn't seem to solve the problem. It happens about once in every 20-30 events created.

Headers:

{
Cache-Control: private
Date: Tue, 09 Jul 2019 01:48:06 GMT
Transfer-Encoding: chunked
Retry-After: 0
request-id: 243629e8-1bcb-44ed-aaec-584aac28f7e7
client-request-id: 243629e8-1bcb-44ed-aaec-584aac28f7e7
x-ms-ags-diagnostic: {"ServerInfo":{"DataCenter":"Australia Southeast","Slice":"SliceC","Ring":"5","ScaleUnit":"002","RoleInstance":"AGSFE_IN_1","ADSiteName":"ASE"}}
Duration: 905.1279
Strict-Transport-Security: max-age=31536000
}

Another:

{
Cache-Control: private
Date: Tue, 09 Jul 2019 03:53:15 GMT
Transfer-Encoding: chunked
Retry-After: 0
request-id: ef5d7ab9-883d-4bf3-8a6d-08563a55fd3f
client-request-id: ef5d7ab9-883d-4bf3-8a6d-08563a55fd3f
x-ms-ags-diagnostic: {"ServerInfo":{"DataCenter":"Southeast Asia","Slice":"SliceC","Ring":"2","ScaleUnit":"000","RoleInstance":"AGSFE_IN_8","ADSiteName":"SEA"}}
Duration: 180.5712
Strict-Transport-Security: max-age=31536000
}

~Edit: While not an ideal solution, adding a 1-2 second delay between the two requests seems to have fixed it~ - Started erroring again

[jasonjoh]

@adamconway sent this along to engineering. Did you use the etag in your second request?

[adamconway]

@jasonjoh no I didn't. Will test that now and update comment after.

Edit: Still errored when using eTag:

First request when creating the event successfully:

request-id: "0d99f34e-b054-450d-97f1-780c754f3b6a"

Failed request to add an open extension to previously created event (using eTag as well):

{
Cache-Control: private
Date: Tue, 09 Jul 2019 23:59:22 GMT
Transfer-Encoding: chunked
Retry-After: 0
request-id: 9bd5932e-1cc6-46f3-965a-3f3fbefd49c4
client-request-id: 9bd5932e-1cc6-46f3-965a-3f3fbefd49c4
x-ms-ags-diagnostic: {"ServerInfo":{"DataCenter":"Southeast Asia","Slice":"SliceC","Ring":"3","ScaleUnit":"002","RoleInstance":"AGSFE_IN_5","ADSiteName":"SEA"}}
Duration: 1103.3763
Strict-Transport-Security: max-age=31536000
}

[stpeteparadox]

We too have been encountering a seemingly inconsistent/random issue with calendar event creations the past several months, but only in the past few days have had time to investigate it in more detail. The error code led us here to this open issue.

We tried the suggested workarounds from @jasonjoh , but they didn't resolve the issue.

Ultimately we did as @masterjs suggests (catch 409 code, find event with matching subject, delete event, retry creation), and that works reliably thus far. We limit the retry attempts to three, but our logs show that the first retry attempt has thus far always succeeded.

Update: just updating this comment because I now realize that our http error status code was 409 not 503, though the resulting error message the same. Screenshot below for reference.

[MikeN123]

@jasonjoh any update on this? We are seeing hundreds of these errors a day, and as the event is created without extensions it means we are unable to properly and reliably track our 'own' events.

[jasonjoh]

No sorry. I passed this along to engineering. I'll see if there's been any developments. In the meantime as a workaround folks are reporting good results with catching that specific error and retrying.

[DerGuteWolf]

@jasonjoh any update on this? We also get this ConcurrentItemSave as 409 at a customer using direct REST calls to the Graph API. Do mean with retrying simple post the exact same request for event creation again?

[MikeN123]

@DerGuteWolf no you want to create the event without extended properties first, and then retry a couple of times to create the extended properties. It usually succeeds on the second try. If you create the event with extended properties in one go, you may end up with a failure and an event that has been partly created.

I do not really understand why such an obvious transactionality failure does not get higher priority at MSFT, reporting a failure and leaving a mess in people's calendars is not really something I would expect Exchange/O365 to do, but I guess we have to try to live with it.

[pintosack]

As @MikeN123 stated, the second try seems to always work and that's what I've done in my code. It's a quick fix/work around but I really don't like doing this. Would be nice to have this issue fixed.

[DerGuteWolf]

@pintosack and @MikeN123 : Yes, doing it in this way also works for us at first or second try. MS support demanded lots of things, single replication was not enough, so this was not perused further.

[GiffenGood]

I am experiencing this also, just making the raw http requests (No SDK). It's a pity this has remained an issue for so long. The workarounds themselves are error prone. If adding an Event and an Extension in the same request are not guaranteed to be transactional, it should be marked as such in the api.

[anttipajunen]

@jasonjoh I've been playing with open extensions with the event resource. I'm seeing the error described in this thread when doing POST operations in Power Automate to the event resource. I'm adding an open extension in the POST request as I can't afford for a separate update to fire off.

Is this still a known issue in open extensions?

Conflicts occurred during saving data to store (saveresult: IrresolvableConflict, properties: ). Please retry the request.

[jasonjoh]

Hello everyone - sorry for the slow communication here. This isn't really a place to report issues with the Graph service, so this is mainly me pinging the calendar team and getting suggestions. I would strongly encourage anyone hitting this error to open a case with support both to get more direct assistance and to help get more reports on this particular error.

The suggestion now is to use the new transactionId property on the event resource to mitigate this issue. The idea would be that your app would do something like:

1. Create the event with the open extension and transactionId (set to a unique value you generate) like so:

   {
     "transactionId": "f50366cf-1fd1-412c-b2be-c67f53ab4ffa",
     "subject": "Event subject",
     "start": {
       "dateTime": "2021-02-02T09:00:00",
       "timeZone": "Pacific Standard Time"
     },
     "end": {
       "dateTime": "2021-02-02T09:30:00",
       "timeZone": "Pacific Standard Time"
     },
     "extensions": [
       {
         "@odata.type": "microsoft.graph.openTypeExtension",
         "extensionName": "MY_EXTENSION",
         "studentNumbers": 16
       }
     ]
   }
   

2. If the POST succeeds, discard the transaction ID.

3. If the POST fails, retry the same POST with the same transaction ID.

[MikeN123]

@jasonjoh if this error is known and occurring thousands of times a day, I'm not sure why a support case is necessary? It's not like something new is to be discovered here.

Are we 100% sure setting a transactionId will work in this case, and will overwrite the existing event with the correct attributes? I'm not sure I want to experiment with this if the current approach (create event + create extended property in two steps) works.

[jasonjoh]

@MikeN123 it's not necessary, I'm just saying if you want more direct assistance, that's the way to go. There may be more to discover on this issue, but I'm not directly involved with engineering.

I'm not willing to say I'm 100% sure of this approach because I have not reproduced this particular error, so I haven't tested the approach against a reliable repro. It is the recommended approach from the calendar team. Their belief is that the issue will go away with this approach. If it doesn't, it's more direct to the folks that can fix it if the issue is reported via support. I will continue to forward information to them from here, but I can't promise any specific outcome.

[anttipajunen]

@jasonjoh thanks for circling back with a suggestion. I'll give it a try.

[anttipajunen]

Updating this thread as it's still the best source of info to be found for this error. I've tried transactionId but that doesn't solve the issue. Sometimes a POST operation still fails in my flow's HTTP connector. Funny enough an event is nevertheless created!

[michelcarlo]

I am having the same issue from a C# Azure function, calling graph API using the REST endpoints.. randomly when events are created or updated I get the irresolvableConflict error. I'm also using open extensions...any update or advice on this other than retrying the operation? TransactionID seems not to solve...