Round-Sync icon indicating copy to clipboard operation
Round-Sync copied to clipboard

Scheduled Triggers not running

Open prairietree opened this issue 2 years ago • 17 comments

I got some task set up and they work fine when ran manually with the "Start Task" option. I also set up some triggers that are scheduled to run every hour or so. But they never seem to run.

App Version 2.1.4 Rclone Version 1.63.1-extract Android 13

prairietree avatar Sep 04 '23 16:09 prairietree

Have you disabled battery optimizations? Those are ususally problematic.

I assume they also dont show up in the "Logs"-Tab?

newhinton avatar Sep 06 '23 13:09 newhinton

Thanks for the reply. I went into settings and apps and set App battery usage to Unrestricted. I have not seen it run the timed syncs yet, so I am not sure that helped.

Yes, they do not show up in the Logs tab. I am not sure if the automatic runs would show up there. Then I run them manually they do.

prairietree avatar Sep 07 '23 00:09 prairietree

Yes, automated syncs will show up in the logs. That also means the definitively didnt run, so its not a notification issue either.

Does your android 13 has a specific permission for alarms? Eg. GrapheneOS has an "Alarms and Reminder" setting. If that were to be turned off, that would also explain the issue.

newhinton avatar Sep 07 '23 14:09 newhinton

I have the same problem. I have 10 tasks set to run at the same time of day and in the logs I can see that only 2-3 of them are executed each day. Yesterday RoundSync synchronized some files that were about 7 days old, which means that this task was skipped for at least a week.. that got my attention. I'm sure not all of the scheduled tasks are starting. At least if many of them are triggered to run at the same time.

nvllz avatar Sep 19 '23 09:09 nvllz

Ah yeah that might be a known problem: #135

At the moment the sync service handles incoming intents directly. There is not actually a queue that would allow me to sequentially handle multiple start intents, or by running them in parallel. It seems, at some point "unprocessed" intents are discarded, with the result that some task are "lost".

This is very likely a hard problem, and a solution non trivial. For the moment i would advise you to not schedule tasks while other tasks might be running.

newhinton avatar Sep 19 '23 09:09 newhinton

I have been working on allowing parallel runs for syncs, but it will take a while to make everything work.

newhinton avatar Sep 19 '23 16:09 newhinton

Oh, great! I'm sorry I can't help with coding, but I'm all yours for testing :D

nvllz avatar Sep 19 '23 17:09 nvllz

In the meantime i have both migrated the sync service to workmanager, and added a permission fragment that will show you if you granted all the required permissions. Can you check if that solved your issues?

newhinton avatar Nov 08 '23 13:11 newhinton

For me it works a bit better now (2.4.2 beta) as I can see more trigger starts in the logs. But there were also some drops, so I need to test it further.

I have several tasks with the same name and sometimes it doesn't show every start. Can the same name be the case of dropping the latter tasks? Especially when logging "there was nothing to do" when sometimes I get only one "x task started" log related to a trigger which is named as 5 others (so there should be 6 different logs for these triggers with the same name).

nvllz avatar Nov 12 '23 12:11 nvllz

With 2.4.x, i changed the sync-component away from IntentService to the newer WorkManager. Especially .2 and .3 wont work properly, since by default android worker's are limited to exactly 10 minutes. After that, they are terminated and restarted at a later time. Each time a new log entry/notification is created.

I have since fixed the 10 minute limit, but not yet released it.

newhinton avatar Nov 12 '23 13:11 newhinton

I made a spreadsheet to show you my logs from today in a more readable way, and in the same order as they were (or should be) logged. I also included expected executions that were never logged, as well as sync failures that were marked as successful and later logged as failed. Something is broken here.

Context: I have 10 triggers scheduled to start 10 different tasks. These are usually simple one-two file copy tasks that are scheduled to run at:

  • 09:40 AM - task 0 > remote A
  • 09:42 AM - tasks 1, 2, 3 > remote A
  • 09:43 AM - tasks 4, 5, 6 > remote B
  • 09:44 AM - tasks 7, 8, 9 > remote C

It looks like tasks 6 and 9 were never started because they are not logged for today and I have no idea why. Both belong to different remotes, both run at different times. Sometimes there are more dropped tasks without any log. I turned off my wifi and mobile data at 10:07 (~25 mins after the last scheduled task) and Round Sync logged all 8 successful entries as failed. So it looks like despite the success log, these were either not really executed or still in progress? Idk... the whole module needs an investigation.

image

Legend:

  • TASK ID - distinguish unique tasks, each task is identified in the order of its scheduled run time
  • LOG TITLE, LOG MESSAGE - log details
  • LOG TIME - time of a log (HH:MM:SS)
  • SCHEDULED RUN TIME - expected start time of each trigger (HH:MM)

Colors:

  • green: successfully executed
  • red: never logged, so I can't be sure it ran at all
  • yellow: failed... but contains only eight tasks previously marked as successful

I'm wondering what the problem is with this. I just want to execute 10 tasks in the pattern I've described above. Tried my best to draw the detailed problem, but idk if it's really helpful. I hope there's a way to make this module reliable.

nvllz avatar Nov 26 '23 12:11 nvllz

Thanks for the detailed observation!

Are you still running 2.4.2?

Round Sync logged all 8 successful entries as failed

So the schedules ran, the sync completed without errors, and then the log showed them as failed?

Another question for clarity, take the first occurence of tasks 1,2 and 3 from the bottom, are they scheduled at the same time? (9:42)

yellow: failed... but contains only eight tasks previously marked as successful

Besides that task 9 is missing, this is expected. If a task cannot be executed due to constraints (in this case, missing wifi), they are logged as failed. The intention here is that the user is notified about a task that has not been executed due to constraints, and therefore an expected sync has not occured. (Since the goal is to execute the sync, we failed the goal, therefore the failure notification)

newhinton avatar Nov 26 '23 15:11 newhinton

Yeah, I'm on 2.4.2, because the latest beta appear to be a separate app called Round Sync Debug.    

So the schedules ran, the sync completed without errors, and then the log showed them as failed?

As you can see on the image (read it in reverse order like RS logs), tasks with the same IDs are first executed correctly (green) and after I cut off my network it got logged (for a second time each) as 'Sync error'. So like it sent a log with a success message but still was in a queue, because it should never be alive for 20 minutes, while I had a stable connection.

   

Another question for clarity, take the first occurence of tasks 1,2 and 3 from the bottom, are they scheduled at the same time? (9:42)

Yeah, three are scheduled to run at the same time. As I mentioned above:

09:40 AM - task 0 > remote A 09:42 AM - tasks 1, 2, 3 > remote A 09:43 AM - tasks 4, 5, 6 > remote B 09:44 AM - tasks 7, 8, 9 > remote C

So at 09:42 three triggers are set to start three tasks, associated with IDs 1, 2, 3. All the tasks work with remote A. When I set all the tasks to the same run time, the app dropped more tasks (never executed some random tasks, only like 3-4).

   

Besides that task 9 is missing, this is expected. If a task cannot be executed due to constraints (in this case, missing wifi), they are logged as failed.

Tasks 6 and 9 are never logged. I turned off wifi like 25 minutes later, so it can't be a case. The same appeared when I got off wifi range 9 hours after the scheduled task time - it logged the Sync error for tasks which should be executed 9h earlier.

nvllz avatar Nov 26 '23 18:11 nvllz

Generally speaking, 2.4.2 and 2.4.3 have been very unstable (They are after all prereleases). There have been some major changes, which created some serious bugs. 2.4.4 should be way more stable. I have not yet released a prerelease of .4.

Nightly-builds and prereleases will all be debug builds going forward. Release builds will only be stable versions.

newhinton avatar Nov 26 '23 18:11 newhinton

@nvllz Can you check if this still persists with 2.5.3?

newhinton avatar Apr 28 '24 18:04 newhinton

Well... (Un)fortunately I have completely reworked my sync strategy, so it is no longer possible for me to try this setup in the latest release. Even if something was wrong, I'm sure it was also a problem with my overcomplicated setup. Since I improved it a few months ago, Round Sync no longer seems to miss any of the scheduled tasks.

Gotta set all of my current triggers to run at the same time and come back with a report soon. Thanks for your interest!

nvllz avatar Apr 28 '24 19:04 nvllz

@newhinton According to logs, having all of my tasks set to run at the same time results in running only a few (1-2) of the 5 scheduled. However, making them run separately in a frame of 5 minutes works fine.

nvllz avatar Apr 30 '24 21:04 nvllz