Maximum number of running instances reached

[iliyanpetrov-sag]

Hi everyone,

This is more like a question than an issue. We are experiencing the following warning in our logs:

WARNING:apscheduler.scheduler:Execution of job "sync_all_teams (trigger: cron[month='*', day='*', day_of_week='*', hour='*', minute='0'], next run at: 2022-02-08 06:00:00 UTC)" skipped: maximum number of running instances reached (1)
WARNING:apscheduler.scheduler:Execution of job "sync_all_teams (trigger: cron[month='*', day='*', day_of_week='*', hour='*', minute='0'], next run at: 2022-02-08 06:00:00 UTC)" skipped: maximum number of running instances reached (1)
WARNING:apscheduler.scheduler:Execution of job "sync_all_teams (trigger: cron[month='*', day='*', day_of_week='*', hour='*', minute='0'], next run at: 2022-02-08 07:00:00 UTC)" skipped: maximum number of running instances reached (1)
WARNING:apscheduler.scheduler:Execution of job "sync_all_teams (trigger: cron[month='*', day='*', day_of_week='*', hour='*', minute='0'], next run at: 2022-02-08 07:00:00 UTC)" skipped: maximum number of running instances reached (1)
WARNING:apscheduler.scheduler:Execution of job "sync_all_teams (trigger: cron[month='*', day='*', day_of_week='*', hour='*', minute='0'], next run at: 2022-02-08 08:00:00 UTC)" skipped: maximum number of running instances reached (1)

Could you help us to resolve the warning? Maybe some configuration needs to be applied? Also there is another question. Is it possible this skipping of cron job task to result in not updating user if there is new change in the LDAP group?

Best Regards, Iliyan

[iliyanpetrov-sag]

After some monitoring, we think this warning is connected with the lack of new updates coming from changed LDAP groups. Would you need some other details around it?

[primetheus]

@iliyanpetrov-sag apologies for the delayed response! If you could provide more details about the behaviors, I can try to reproduce it. Here are a couple questions I have off the top of my head:

1. Do you have a large LDAP environment, that this might be running longer than 1h to sync all teams/users?
2. Has this worked at any point, or is this error preventing all functionality?

Is it possible this skipping of cron job task to result in not updating user if there is new change in the LDAP group?

I don't think it's likely. The cron should grab all users in LDAP groups and in the GitHub team and sync the delta. If nothing changed then it'll just move to the next group/team.

After some monitoring, we think this warning is connected with the lack of new updates coming from changed LDAP groups. Would you need some other details around it?

It's possible that this could be an event-based run conflicting with a cron-based run, but would need to know a bit more to debug it

[primetheus]

@iliyanpetrov-sag if you still need help with this please feel free to reach back out