Subscriber server sometimes doesn't receive the content update event within the configured time.

[RyuLindow]

Which Umbraco version are you using? (Please write the exact version, example: 10.1.0)

10.1.0

Bug summary

Content on the subscriber servers sometimes doesn't update within the configured time.

Specifics

I've created a load-balanced setup on Azure 1 Redis cache 1 SQL server 1 SQL DB 1 app service plan with 2 following web apps: 1 master 1 secondary scaled up to 2 instances

The very first content update worked without any problems and the content got updated on the master and both instances of the secondary web app.

I've then continued to update the content 3-4 times every hour or so to see if everything works properly and the last content update only reached one of the instances. I updated the content and republished the page. That content version didn't reach any of the secondary instances.

This resulted in me having:

Master: up to date 1st secondary: 1 publish event behind 2nd secondary: 2 publish events behind

After than, I left it for that day and checked back on the next. The content was up to date on both secondary instances.

I was able to get the same result one more time on a copy of the exact same load balanced set up but the resources are separate, as it's meant to be used for a dev env and the first one is prod.

I haven't added these app settings, however, the default setting is 5seconds when looking at the code

This has also happened on 9.5.x

The that the secondary web app has more than 1 instance doesn't affect this in any way, as I was able to reproduce it after I scaled it down to 1.

Steps to reproduce

Create a load balanced setup on Azure which I've described above.

Update and publish some content a few times.

Expected result / actual result

Content updates within the specified time period.

---

This item has been added to our backlog AB#22588

[simonhunter1]

We are experiencing the same issue on our site too.

2 service plans, one for publisher and one for subscriber 1 or 2 scaled subscriber app services (Linux) 1 publisher app service (Linux) We don't have a Redis cache

When the app service is restarted, the changes are all updated. Some users reported seeing updated content earlier than other users when we had 2 scaled subscriber apps. Changes seem to be updated quickly within about 5 minutes of a restart, then they are no longer updated for many hours. So we've set up a job to automatically restart the subscribers every night in the meantime.

We plan on changing the app services to Windows to hopefully mitigate another bug we are seeing which seems to be related to this: https://github.com/dotnet/SqlClient/issues/647 where we see timeout errors intermittently. I'm just including that here in case it's relevant but maybe it's not.

[kjac]

Hi there @RyuLindow and @simonhunter1 👋

I'm afraid I can't reproduce this. I have created a test setup (see details below), where I have continuously been able to publish content updates from the publisher to the subscriber.

Over the course of eight hours yesterday I performed five consecutive updates every one to two hours. Each update appeared on the subscriber within a few seconds.

I resumed the same pattern of testing this morning with the same result.

Before we dig any deeper into this, could you please verify that your setups are in line with the setup described below?

Test setup

The test setup runs v10.1 on Azure and is provisioned as follows:

• A publisher (admin) app service with its own app service plan (S2, single instance, Windows)
• A subscriber (delivery) app service with its own app service plan (S2, single instance, Windows)
• A shared SQL server

The app services are configured as described on Our and are using explicit server roles.

Both app services are configured for 64 bit with "Always on" enabled and "AAR affinity" disabled. Neither of these configurations should really matter for this issue, but are mentioned for the sake of completion:

Publisher configuration and setup

Server role accessor implementation:

using Umbraco.Cms.Core.Sync;

namespace Issue12986.Publisher.Compose;

public class SchedulingPublisherServerRoleAccessor : IServerRoleAccessor
{
    public ServerRole CurrentServerRole => ServerRole.SchedulingPublisher;
}

Server role accessor composer implementation:

using Umbraco.Cms.Core.Composing;
using Umbraco.Cms.Infrastructure.DependencyInjection;

namespace Issue12986.Publisher.Compose;

public class SchedulingPublisherServerRoleComposer : IComposer
{
    public void Compose(IUmbracoBuilder builder)
        => builder.SetServerRegistrar<SchedulingPublisherServerRoleAccessor>();
}

Server configuration:

{
  "$schema": "./appsettings-schema.json",
  "Serilog": {
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft": "Warning",
        "Microsoft.Hosting.Lifetime": "Information",
        "System": "Warning"
      }
    }
  },
  "Umbraco": {
    "CMS": {
      "Global": {
        "Id": "333ffc01-6d0e-4042-9d12-9b9556b35fce",
        "SanitizeTinyMce": true,
        "MainDomLock" : "FileSystemMainDomLock"
      },
      "Content": {
        "AllowEditInvariantFromNonDefault": true,
        "ContentVersionCleanupPolicy": {
          "EnableCleanup": true
        }
      },
      "ModelsBuilder": {
        "ModelsMode": "Nothing"
      },
      "Hosting": {
        "LocalTempStorageLocation" : "EnvironmentTemp"
      },
      "Examine": {
        "LuceneDirectoryFactory" : "SyncedTempFileSystemDirectoryFactory"
      }
    }
  },
  "ConnectionStrings": {
    "umbracoDbDSN": "****",
    "umbracoDbDSN_ProviderName": "Microsoft.Data.SqlClient"
  }
}

Subscriber configuration and setup

Server role accessor implementation:

using Umbraco.Cms.Core.Sync;

namespace Issue12986.Subscriber.Compose;

public class SubscriberServerRoleAccessor : IServerRoleAccessor
{
    public ServerRole CurrentServerRole => ServerRole.Subscriber;
}

Server role accessor composer implementation:

using Umbraco.Cms.Core.Composing;
using Umbraco.Cms.Infrastructure.DependencyInjection;

namespace Issue12986.Subscriber.Compose;

public class SubscriberServerRoleComposer : IComposer
{
    public void Compose(IUmbracoBuilder builder)
        => builder.SetServerRegistrar<SubscriberServerRoleAccessor>();
}

Server configuration:

{
  "$schema": "./appsettings-schema.json",
  "Serilog": {
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft": "Warning",
        "Microsoft.Hosting.Lifetime": "Information",
        "System": "Warning"
      }
    }
  },
  "Umbraco": {
    "CMS": {
      "Global": {
        "Id": "333ffc01-6d0e-4042-9d12-9b9556b35fce",
        "SanitizeTinyMce": true,
        "MainDomLock" : "FileSystemMainDomLock"
      },
      "Content": {
        "AllowEditInvariantFromNonDefault": true,
        "ContentVersionCleanupPolicy": {
          "EnableCleanup": true
        }
      },
      "ModelsBuilder": {
        "ModelsMode": "Nothing"
      },
      "Hosting": {
        "LocalTempStorageLocation" : "EnvironmentTemp"
      },
      "Examine": {
        "LuceneDirectoryFactory" : "TempFileSystemDirectoryFactory"
      }
    }
  },
  "ConnectionStrings": {
    "umbracoDbDSN": "****",
    "umbracoDbDSN_ProviderName": "Microsoft.Data.SqlClient"
  }
}

[simonhunter1]

Thanks for this very thorough response, @kjac! 🚀

I noticed one difference in our configuration, it seems like our "MainDomLock" was set to SqlMainDomLock. I've changed it to FileSystemMainDomLock on both of the servers, and on initial testing in our staging environments this seems to have resolved the issue. We'll keep testing and try pushing that to production next week. Thanks!

As a side question - is it always required that the Publisher and Subscriber are each on their own app service plans?

[kjac]

@simonhunter1 that is great news 🎉 here's hoping it works in production! I'll be looking forward to an update 😄

As for separate app service plans the answer is a definite "yes". The Publisher role cannot run on a scaled out app service plan - it'll cause all kinds of trouble like cache inconsistency, file locking, concurrency issues etc. etc. ... And since load balancing very much means scaling out the Subscriber, they must each have their own app service plan, where the Publisher plan must be a single instance plan.

[kjac]

@simonhunter1 just following up here; did you get it to production?

@RyuLindow did you get your issues resolved?

[simonhunter1]

@kjac yes we went into production yesterday. I've been checking yesterday and today at various times. So far, publishing has been happening within a few seconds or so. So I think this has solved our issue. Thanks again!

[kjac]

@simonhunter1 that is really great news 🎉 I'll close this issue soon then - do make sure to report back if you experience any further problems in this regard 😄

[RyuLindow]

I've set up a new solution on v10, again using Redis cache, and experienced the same issue on the subscriber web app which is scaled up to 3 instances.

This time the admin and subscriber each have their own app service plans even though, I don't believe that it's not necessary for the sake of testing.

After updating and republishing the node, the content was updated on the scaled subscriber web app, which is the same behavior I reported earlier.

I'll get rid of Redis and see if the issue persists :)

Perhaps the issue is with the redis cache.

[RyuLindow]

Hmm, I can't reproduce this issue any more after removing the Redis cache conn string.

I'll try again tomorrow.

[kjac]

This time the admin and subscriber each have their own app service plans even though, I don't believe that it's not necessary for the sake of testing.

@RyuLindow it is indeed necessary to have the admin instance and the subscriber instances on separate app service plans, because the admin instance must not be scaled out.

[RyuLindow]

Ah, you're right.

I forgot that the whole app service plan is scaled out and not just the web app. My mistake.

I made a PR where I specified this in the docs on Our.

I've also just tested my setup again just a moment ago and it works perfectly fine without Redis.

This is how my setup looks like

[kjac]

@RyuLindow that's fantastic news 🎉 your setup looks just right. And thanks for the docs PR, I'll have a look at it 😄

Cross referencing PR here for future discoverability: https://github.com/umbraco/UmbracoDocs/pull/4438

[kjac]

I'm going to go ahead and close this issue now. Thank you to everyone for testing out solutions, I'm happy we got it all resolved.

Please don't hesitate to open a new issue if you run into problems later on.

Thanks again 👏