cells-sync
cells-sync copied to clipboard
pydio
Sync directory with many small files
I have a folder with about 300k small files (like *.php or so). I added sync task this folder to my Personal Files. After string with "Capturing first snapshots now" appeared in log - my PC almost freezes for 30-40 seconds. But then it drops an error and sync task were gone.
Here my log:
2019-10-28T17:34:52.683+0300 INFO sync-task Analyzed 9129 nodes (86%)
2019-10-28T17:34:52.711+0300 INFO sync-task Analyzed 9393 nodes (91%)
2019-10-28T17:34:52.743+0300 INFO sync-task Analyzed 9699 nodes (96%)
2019-10-28T17:34:52.753+0300 INFO sync-task Diff Stats {"s": {"EndpointLeft":"https://cells.example.com:8002/common-files/test","EndpointRight":"fs:///home/vallefor/Cells","conflicts":0,"missingLeft":0,"missingRight":0}}
2019-10-28T17:34:52.754+0300 INFO sync-task Finished analyzing nodes {"i": 15048}
2019-10-28T17:34:52.754+0300 INFO sync-task Filtering TreePatch took {"time": "3.752µs", "source": "https://cells.example.com:8002/common-files/test", "target": "fs:///home/vallefor/Cells"}
2019-10-28T17:34:52.754+0300 INFO sync-task Filtering TreePatch took {"time": "2.508µs", "source": "fs:///home/vallefor/Cells", "target": "https://cells.example.com:8002/common-files/test"}
2019-10-28T17:34:52.754+0300 INFO sync-task Merged Patch {"stats": {"Source":"https://cells.example.com:8002/common-files/test","Target":"fs:///home/vallefor/Cells","Type":"TreePatch"}}
2019-10-28T17:34:52.755+0300 INFO sync-task Empty Patch : nothing to do
2019-10-28T17:34:56.497+0300 INFO sync-task Analyzed 286571 nodes (92%)
2019-10-28T17:34:58.285+0300 INFO sync-task Analyzed 287376 nodes (97%)
2019-10-28T17:34:59.515+0300 INFO sync-task Diff Stats {"s": {"EndpointLeft":"https://cells.example.com:8002/personal-files","EndpointRight":"fs:///home/vallefor/Sync-d","conflicts":0,"missingLeft":230072,"missingRight":747}}
2019-10-28T17:34:59.515+0300 INFO sync-task Finished analyzing nodes {"i": 289972}
2019-10-28T17:35:00.097+0300 INFO sync-task Filtering TreePatch took {"time": "216.223µs", "source": "https://cells.example.com:8002/personal-files", "target": "fs:///home/vallefor/Sync-d"}
2019-10-28T17:35:25.499+0300 INFO oidc Additional safe check for token https://[email protected]:8002
2019-10-28T17:36:33.632+0300 INFO sync-task Filtering TreePatch took {"time": "144.058325ms", "source": "fs:///home/vallefor/Sync-d", "target": "https://cells.example.com:8002/personal-files"}
2019-10-28T17:36:36.590+0300 INFO sync-task Merged Patch {"stats": {"Pending":{"CreateFile":207464,"CreateFolder":23355,"Total":230819},"Source":"https://cells.example.com:8002/personal-files","Target":"fs:///home/vallefor/Sync-d","Type":"TreePatch"}}
2019-10-28T17:36:39.009+0300 INFO sync-task Capturing first snapshots now
2019-10-28T17:37:25.499+0300 INFO oidc Additional safe check for token https://[email protected]:8002
2019-10-28T17:37:39.170+0300 ERROR endpoint.cells.remote Stopping watcher on errorrpc error: code = DeadlineExceeded desc = context deadline exceeded
2019-10-28T17:37:39.192+0300 INFO endpoint.cells.remote Connection finished rpc error: code = DeadlineExceeded desc = context deadline exceeded
2019-10-28T17:37:39.192+0300 INFO sync-task https://cells.example.com:8002/common-files/test is currently disconnected
2019-10-28T17:37:44.192+0300 INFO endpoint.cells.remote Restarting events watcher after 5s
2019-10-28T17:37:44.221+0300 INFO sync-task https://cells.example.com:8002/common-files/test is now connected
hey - we tested until 50k, but I guess it's not yet optimized enough for 300k files ...
Looking forward for optimization! :) I user Seafile for now - it can handle many small files easyly. But it have some other minuses so I time to time looking for new alternatives. And your project looks very promising!
I got the same error once or twice. I just restarted the sync client and it continued to sync.
ERROR endpoint.cells.remote Stopping watcher on errorrpc error: code = DeadlineExceeded desc = context deadline exceeded
