Uploaded files are not pushed (groupfolders)

[iljur]

After installling notify-push I did some tests with uploading/deleting some files and experienced that files uploaded via the web interface (Firefox, W10) are not being pushed to other clients. Takes a looong time (minutes) until they appear in other clients assumingly through the standard sync method.

Deleting files in the webinterface is instantly pushed. Uploading/deleting files in Windows explorer is both instantly pushed through the Windows desktop client.

Any ideas on this?

[iljur]

This is part of the log output which explains quite good what is (not) happening: push_log.txt

You can see that if files are deleted, the push service is sending a "notify_file" to the clients to push the change.

If a file is added (doesn't matter if added in Explorer/Desktop client or web-UI) there is nothing sent but keepalive pings. Resulting in added files not showing up at other clients for a long time. If there is interaction on the other client (deleting/adding) the new files show up.

Why does the service not send "notify_files" when files are added? Seems like a bug to me...

[icewind1991]

I'm guessing Test_Ordner/homework.docx is the newly uploaded file here, is this within a share, groupfolder or external storage?

[iljur]

Yes, all files mentioned in the log reside in a groupfolder (Test_Ordner). First 2 have been deleted. Last one (homework.docx) has been added. Please let me know if further information or testing is needed.

[damiankaczkowski]

Same problem. NC 22.1.1 Windows Client: 3.3.2 Linux Client: 3.2.3 notify_push: 0.2.2 groupfolders: 10.0.0-beta1

notify_push works when creating/deleting file/dir in user folder. notify_push does not work when creating/deleting file/dir in groupfolder.

Logs shows that notify_push smmes to be aware of dir creation/deletion in groupfolder and even seems to sends push notification but for some reason they do not work. Client ignores them?

[2021-08-30 02:39:18.507984 +02:00] DEBUG [notify_push] src/lib.rs:403: Received storage update notification for storage 16 and path
[2021-08-30 02:39:18.508020 +02:00] DEBUG [notify_push::storage_mapping] src/storage_mapping.rs:96: querying storage mapping for 16
[2021-08-30 02:39:18.508188 +02:00] INFO [sqlx::query] <unnamed>:0: /* SQLx ping */; rows: 0, elapsed: 145.564µs
[2021-08-30 02:39:18.509340 +02:00] INFO [sqlx::query] <unnamed>:0: SELECT user_id, path FROM …; rows: 111, elapsed: 1.035ms

SELECT
  user_id,
  path
FROM
  nc_mounts
  INNER JOIN nc_filecache ON root_id = fileid
WHERE
  storage_id = 16

[2021-08-30 02:39:18.509440 +02:00] INFO [sqlx::query] <unnamed>:0: /* SQLx ping */; rows: 0, elapsed: 51.345µs
[2021-08-30 02:39:18.512333 +02:00] DEBUG [notify_push] src/lib.rs:403: Received storage update notification for storage 16 and path
[2021-08-30 02:39:18.529192 +02:00] DEBUG [notify_push] src/lib.rs:403: Received storage update notification for storage 16 and path Test/_003
[2021-08-30 02:39:18.531814 +02:00] DEBUG [notify_push] src/lib.rs:403: Received storage update notification for storage 16 and path Test/_003
[2021-08-30 02:39:18.533876 +02:00] DEBUG [notify_push] src/lib.rs:403: Received storage update notification for storage 16 and path Test
[2021-08-30 02:39:18.744284 +02:00] DEBUG [notify_push] src/lib.rs:403: Received storage update notification for storage 16 and path
[2021-08-30 02:39:18.747268 +02:00] DEBUG [notify_push] src/lib.rs:403: Received storage update notification for storage 16 and path
[2021-08-30 02:39:19.117675 +02:00] DEBUG [notify_push] src/lib.rs:403: Received storage update notification for storage 16 and path
[2021-08-30 02:39:19.120883 +02:00] DEBUG [notify_push] src/lib.rs:403: Received storage update notification for storage 16 and path
[2021-08-30 02:39:19.847798 +02:00] DEBUG [notify_push] src/lib.rs:403: Received storage update notification for storage 16 and path
[2021-08-30 02:39:19.851020 +02:00] DEBUG [notify_push] src/lib.rs:403: Received storage update notification for storage 16 and path
[2021-08-30 02:39:38.804922 +02:00] DEBUG [notify_push::connection] src/connection.rs:106: Sending ping to foo.bar
[2021-08-30 02:39:38.805068 +02:00] DEBUG [notify_push::connection] src/connection.rs:106: Sending ping to foo.bar
[2021-08-30 02:39:38.805210 +02:00] DEBUG [notify_push::connection] src/connection.rs:106: Sending ping to foo.bar
[2021-08-30 02:39:38.805292 +02:00] DEBUG [notify_push::connection] src/connection.rs:106: Sending ping to foo.bar

[damiankaczkowski]

Seems that creation of "Level 1" dir on groupfolder root triggers client push sync properly. Only >= "Level 2" dir creation on groupfolder does not trigger client push sync. Just as shown on logs creating "_003" dir in "Test/_003" does not work. Creating for example "_003" dir on groupfolder root works.

[nrgeen]

Hello, we are also experiencing the problem. A few days ago we switched from network drives to completely Nextcloud, including group folders, desktop clients and HBP. Here we quickly noticed that the syncronization in the first folder level works very well. In the second level, however, syncing does not take place even after a few days. How can I support you in finding and solving the problem?

@iljur In our case this problem seems to be specifically related to the group folders. Does it make sense to add the issue title accordingly?

[iljur]

The problem is diffuse and we don't have it all the time, but every now and then. Therefore hard to describe and report. It is probably related specifically to group folders and external storage. Even more likely, however, it is related to the client software (Windows). For example, we have already had to completely reinstall a client (Windows) 3 times (incl. complete cleaning of the settings, database) after a client was not used for some time (vacation). In each case a different client. But the same pattern.

I really wish the dev team would spend some time focusing on hardening and debugging file syncing as THE key feature. Instead of constantly releasing new niche functions that then only come along half-baked anyway.

[bob4os]

I'm having the same issue. Using "nextcloud.exe --logwindow" I don't see any "notify_file" notification being received for changes in a subfolder of a groupfolder. Also unsharing a subfolder of a groupfolder did not trigger a "notify_file" to my testing client.

[bob4os]

Is it possible to rename this issue, since this is specific to groupfolders (probably external storage) and the missing "notify_file" notification?

The Nextcloud client does show notifications ("notify_notification") that a folder has been shared/unshared - is there any hard-coded filtering criteria? (I have no experience with Rust yet.)

[nrgeen]

Hey, Is there any news about this issue?

Unfortunately, I have the impression that there is currently no interest on the part of the developers. I have already made a suggestion to include group folders in the title above. Maybe we need to open a similar issue in the group folder repository.

Internally, we are now unfortunately encountering this problem more frequently and are currently asking ourselves how we are going to deal with it in the long term. Are there any ideas or suggestions?

[iljur]

@nrgeen: Yes. Me too! As I already said above, I would really appreciate if the devs would focus on hardening and debugging the main file sharing functions including groupfolders, external storage and the clients.

Groupfolders have always been buggy (file permissions...) and the last version of the windows client (3.4.0) had an ugly showstopper bug resetting the timestamp of files. All this reminds me more of a public alpha/beta test than mature software for production environments. It is advisable to update nextcloud software only >6month after release to not run into ever new bugs.

And yes: You get the idea of looking for a more stable solution as a daily driver.

[bob4os]

I‘m trying to read through the code - my guess is, that the event is filtered out. Probably in the following places: https://github.com/nextcloud/notify_push/blob/4b75698ee4f5c2e3f43a236cf438a211b3e62221/lib/Listener.php#L53 - but I doubt that groupfolders match IHomeStorage https://github.com/nextcloud/notify_push/blob/4b75698ee4f5c2e3f43a236cf438a211b3e62221/src/storage_mapping.rs#L84 - I don‘t think that groupfolders match this condition and therefore no notification is queued (if understand this correctly)

Something else I noticed - Nextcloud groupfolders all have the storage_id (1 in my instance) and differ in the groupfolder id in the path, which seems to be cut in the debug log, therefore missing the complete path like __groupfolders/1/…

[bob4os]

@nrgeen you may change the Nextcloud clients „forceSyncInterval“ (in https://docs.nextcloud.com/desktop/3.4/advancedusage.html) to 30000 until this is fixed.

[nrgeen]

@nrgeen you may change the Nextcloud clients „forceSyncInterval“ (in https://docs.nextcloud.com/desktop/3.4/advancedusage.html) to 30000 until this is fixed.

Oh, very cool. Thank you. I'll try that out and let you know how it went.

[olewales]

I am experiencing this issue too. We are using group folders extensively and I had to disable notify_push for now as a workaround. I cannot expect everyone using desktop client to make configuration changes locally because of server bug.

[seijikun]

Where does this bug reside, exactly? Is this a bug in notify_push, the groupfolders extension, or is this a design flaw in the server?

[nrgeen]

Hello, is there any news about the described problem yet? I perceive that notify_push is used for more and more programs, like deck or also files_lock. Currently we have notify_push disabled, because unfortunately the files_lock feature does not work at all in group folders. Now I'm asking myself if I should change the file structure so that we don't have to rely on group folders anymore. Before I start the project, I am interested here if there is a plan to bind the groupfolders in again properly from notify_push? If not, does it make more sense from a programmer's perspective to just drop groupfolders?

I would be very happy about a feedback!

[icewind1991]

https://github.com/nextcloud/notify_push/pull/244 should fix updates not being pushed for groupfolders

[nrgeen]

Great news! Thanks a lot! 🥳

[q-wertz]

@iljur @nrgeen do you know/already test if the new release v0.6.1 fixed it?

[joshtrichards]

Released in v0.6.1 and the only feedback received was positive:

https://github.com/nextcloud/notify_push/pull/244#pullrequestreview-1350658453

Closing since this was released 6+ months ago and no further feedback has been received here either way.

Can always reopen if it's not fixed after you update.