Events removed when URL returns 5xx

[jerrywoo96]

The problem

When the URL returns HTTP Error 5xx, Events are removed from the calendar.

Is it possible to do nothing to the calendar if requests are not successful?

Version of GAS-ICS-Sync

5.7

Additional information & file uploads

var howFrequent = 360; var onlyFutureEvents = false; var addEventsToCalendar = true; var modifyExistingEvents = true; var removeEventsFromCalendar = true; var addAlerts = "no"; var addOrganizerToTitle = false; var descriptionAsTitles = false; var addCalToTitle = false; var addAttendees = false; var defaultAllDayReminder = -1; var overrideVisibility = "public"; var addTasks = false;

var emailSummary = true;

[derekantrican]

Sounds like this may be solved by #235 - however, this has become stale.

[willjgit]

Don't think this has been fixed. I had a http 500 error when pulling a .ics file from google drive, and the script seems to think it pulled 0 events and proceeded to delete all the events on my google calendar. Is there a way to handle the error so that when you get a http 5xx error, the script just stops and hopefully the next run will refresh the ics?

[Lonestarjeepin]

I just ran across this same issue last week. I have an ical that is returning "[ERROR] Incorrect ics/ical URL..." and then all of the events are deleted. I looked at #235 , but I hesitated to implement that code you gave Jonas because it looks like there are some conflicts with newer code in the script (colorID, etc.). I also didn't really see where it was so different than what was in the Master in terms of handling the error (but you know my coding prowess is weak-sauce).

[jerrywoo96]

Here's my easy and simple to understand fix: On this line is a for loop. https://github.com/derekantrican/GAS-ICS-Sync/blob/e5cd8e02417ee38a04eadbd795201822dbac2e2d/Code.gs#L147 So, on this line it fetches your ics file and reads the number of calendars. https://github.com/derekantrican/GAS-ICS-Sync/blob/e5cd8e02417ee38a04eadbd795201822dbac2e2d/Code.gs#L161 In theory, if the script fails to fetch the calendars, it will return 0 calendars and there will be nothing to sync. So, what I did is I added a length == 0 check and tell the script to continue to the next url, skipping the rest of the import or delete code. The code will look like this:

    //------------------------ Fetch URL items ------------------------
    var responses = fetchSourceCalendars(sourceCalendarURLs);
    if (responses.length == 0){
      Logger.log("Error Syncing " + targetCalendarName + ". Skipping...");
      continue;
    }
    Logger.log("Syncing " + responses.length + " calendars to " + targetCalendarName);

the term continue means go to the next url in the for-loop that was on this line: https://github.com/derekantrican/GAS-ICS-Sync/blob/e5cd8e02417ee38a04eadbd795201822dbac2e2d/Code.gs#L147

[jerrywoo96]

Meanwhile this is my quick and easy fix until the owners of this project releases a new version. I scheduled it to sync every 6 hours so there's always a next retry and it won't put so much load on my server. If every request returns 500, then it might be something to do with the source url.

[octogonz]

I encountered the same problem.

Due to spurious service I see something like this:

Timestamp	Type	Description
Dec 8, 2023, 4:21:33 PM	Info	Error: Encountered HTTP error 403 when accessing https://example.com/api/ext/1104/e966ac263/icalendar/feed/feed.ics
Dec 8, 2023, 4:21:33 PM	Info	Syncing 0 calendars to Calendar
Dec 8, 2023, 4:21:33 PM	Info	Working on calendar: [email protected]
Dec 8, 2023, 4:21:40 PM	Info	Fetched 212 existing events from Calendar
Dec 8, 2023, 4:21:40 PM	Info	Parsed 0 events from ical sources
Dec 8, 2023, 4:21:40 PM	Info	Processing 0 events
Dec 8, 2023, 4:21:41 PM	Info	Done processing events
Dec 8, 2023, 4:21:41 PM	Info	Checking 212 events for removal
Dec 8, 2023, 4:21:41 PM	Info	Deleting old event d9e89a69-adb5-471c-bc5f-9a4f192edc3f
Dec 8, 2023, 4:21:41 PM	Info	Deleting old event d4898d10-c74a-4814-8a0b-bb401319477f
Dec 8, 2023, 4:21:42 PM	Info	Deleting old event 7c87b02e-d7e4-416a-8538-dcc4f2c9a629
. . .

The entire calendar is deleted!

Then 10 minutes later on the next run:

Timestamp	Type	Description
Dec 8, 2023, 4:31:33 PM	Info	Syncing 1 calendar to Calendar
Dec 8, 2023, 4:31:34 PM	Info	Working on calendar: [email protected]
Dec 8, 2023, 4:31:40 PM	Info	Fetched 0 existing events from Calendar
Dec 8, 2023, 4:31:40 PM	Info	Parsed 213 events from ical sources
Dec 8, 2023, 4:31:40 PM	Info	Processing 213 events
Dec 8, 2023, 4:31:41 PM	Info	Adjusted RRule/RDate to exclude past instances
Dec 8, 2023, 4:31:41 PM	Info	Adding new event d9e89a69-adb5-471c-bc5f-9a4f192edc3f
Dec 8, 2023, 4:31:41 PM	Info	Adjusted RRule/RDate to exclude past instances
Dec 8, 2023, 4:31:41 PM	Info	Adding new event d4898d10-c74a-4814-8a0b-bb401319477f
Dec 8, 2023, 4:31:42 PM	Info	Adding new event 7c87b02e-d7e4-416a-8538-dcc4f2c9a629
. . .

...the calendar is suddenly restored.

Two suggestions:

1. For a potentially intermittent error such as 403, it should abort syncing.
2. But could it communicate the problem somehow? For example send an email alert once per day?

[jerrywoo96]

@octogonz Have you tried my solution had worked if 403 is encountered?

[yermulnik]

Not sure which solution fixed missing Outlook calendar events in my Google calendar since I copied over both: this and https://github.com/derekantrican/GAS-ICS-Sync/pull/386 — not matter which, I'd like to thank both contributors! Thank you!

[octogonz]

Have you tried my solution had worked if 403 is encountered?

@jerrywoo96 if I understand correctly, your fix essentially ignores network failures, temporarily skipping the sync for that calendar.

In my case, the calendar contains important events that are used to manage my week. Comparing...

1. Having all the synced events suddenly disappear from my calendar (possibly reappearing an hour later); versus...
2. Having the calendar look completely normal, except the information is wrong because syncing has stopped, possibly for several days before I notice

...then the first one actually seems safer. At least I will immediately notice that something is broken. The second approach could lead to embarrassment of scheduling conflicting meetings or not showing up for events, until I finally realize that the calendar is outdated.

This is why I was asking about a notification mechanism: When the syncing fails, the script should send an email alert, or maybe insert a fake "error" event on the calendar, etc.

[jerrywoo96]

Have you tried my solution had worked if 403 is encountered?

@jerrywoo96 if I understand correctly, your fix essentially ignores network failures, temporarily skipping the sync for that calendar.

In my case, the calendar contains important events that are used to manage my week. Comparing...

1. Having all the synced events suddenly disappear from my calendar (possibly reappearing an hour later); versus...
2. Having the calendar look completely normal, except the information is wrong because syncing has stopped, possibly for several days before I notice

...then the first one actually seems safer. At least I will immediately notice that something is broken. The second approach could lead to embarrassment of scheduling conflicting meetings or not showing up for events, until I finally realize that the calendar is outdated.

This is why I was asking about a notification mechanism: When the syncing fails, the script should send an email alert, or maybe insert a fake "error" event on the calendar, etc.

    //------------------------ Fetch URL items ------------------------
    var responses = fetchSourceCalendars(sourceCalendarURLs);
    if (responses.length == 0){
      Logger.log("Error Syncing " + targetCalendarName + ". Skipping...");

      var message = {
        to: email,
        subject: "Error Syncing Calendar",
        htmlBody: "Failed to sync " + targetCalendarName + ".",
        name: "GAS-ICS-Sync"
      };

      MailApp.sendEmail(message);

      continue;
    }
    Logger.log("Syncing " + responses.length + " calendars to " + targetCalendarName);

I have no idea if it works or not. Update the thread if it works. Based on the code from https://github.com/derekantrican/GAS-ICS-Sync/blob/71bb0a678982178325cc69f44b8b81af86e72a55/Helpers.gs#L1039

@octogonz @Lonestarjeepin

[octogonz]

Suppose we wanted to send at most one notification email per day? Does the Google Apps Script have any way to remember state between runs?

Software engineering is hard. 😄

[octogonz]

From a little research, it seems like we should be able to do something like:

    var userProperties = PropertiesService.getUserProperties();
    userProperties.setProperty('lastEmailSent', new Date().toISOString())

I'll try this when I get some time.

[octogonz]

I'm now experiencing this error very frequently. It is probably not Google Calendar's fault, but rather decreased reliability of the company that provides my calendar service.

Looking more closely at the code, the script implements a retry/backoff for each network request:

var defaultMaxRetries = 10; // Maximum number of retries for api functions (with exponential backoff)

/**
 * Runs the specified function with exponential backoff and returns the result.
 * Will return null if the function did not succeed afterall.
 *
 * @param {function} func - The function that should be executed
 * @param {Number} maxRetries - How many times the function should try if it fails
 * @return {?Calendar.Event} The Calendar.Event that was added in the calendar, null if func did not complete successfully
 */
var backoffRecoverableErrors = [
  "service invoked too many times in a short time",
  "rate limit exceeded",
  "internal error"];
function callWithBackoff(func, maxRetries) {
  var tries = 0;
  var result;
  while ( tries <= maxRetries ) {
    tries++;
    try{
      result = func();
      return result;
    }
    catch(err){
      err = err.message  || err;
      if ( err.includes("HTTP error") ) {
        Logger.log(err);
        return null;
      } else if ( err.includes("is not a function")  || !backoffRecoverableErrors.some(function(e){
              return err.toLowerCase().includes(e);
            }) ) {
        throw err;
      } else if ( tries > maxRetries) {
        Logger.log(`Error, giving up after trying ${maxRetries} times [${err}]`);
        return null;
      } else {
        Logger.log( "Error, Retrying... [" + err  +"]");
        Utilities.sleep (Math.pow(2,tries)*100) + 
                            (Math.round(Math.random() * 100));
      }
    }
  }
  return null;
}

There are 10 reattempts at exponential intervals (200ms, 400ms, 800ms...) producing an overall delay (200ms, 200+400=600ms, 200+400+800=1400ms, ...) which I calculated should delay about ~3.5 minutes for all 10 failures. This should be pretty robust.

HOWEVER, this retry logic is ONLY applied for recognized error messages ("service invoked too many times in a short time", etc). If the errors are coming from a third-party calendar, the strings will be different, so no retries will occur at all.

Maybe callWithBackoff() can actually solve this problem? If so, then we just need to make it more configurable by end users:

• Move the backoffRecoverableErrors array to be an official configuration setting
• When an error occurs, print the message in the logs, so the user can easily identify the string to be added

I'll experiment with this idea and follow up.

[octogonz]

I also noticed that fetchSourceCalendars() has this note:

https://github.com/derekantrican/GAS-ICS-Sync/blob/17cde1a768ad4c6d5ede0e7401e20606fff0be2a/Helpers.gs#L162-L164

However in the snippet above, the HTTP error substring causes callWithBackoff() to return immediately without retrying:

https://github.com/derekantrican/GAS-ICS-Sync/blob/17cde1a768ad4c6d5ede0e7401e20606fff0be2a/Helpers.gs#L1098-L1103

Apparently this inconsistency was introduced by @jonas0b1011001 in https://github.com/derekantrican/GAS-ICS-Sync/pull/245 which was trying to improve error handling with multiple calendars: If the first calendar reports an error, processing should continue for the other calendars.

But it seems like a mistake to completely disable retries for all HTTP operations. 🤔

[octogonz]

Update: Enabling retries for the HTTP requests definitely seems to resolve the intermittent connection failures that I had encountered. 👍

I also found that if important an error occurs (the total failure to sync a calendar after retries), the script can throw an uncaught top-level exception which will cause the run to appear as "failed" in the "Executions" log. This makes it easier to identify how frequently such failures are occurring.

I'm going to try this solution for a while. If it performs well, then I'll contribute a PR with these changes.

[octogonz]

Here's my PR: https://github.com/derekantrican/GAS-ICS-Sync/pull/403