seadroid icon indicating copy to clipboard operation
seadroid copied to clipboard
verified publisher icon haiwen

Seadroid v3.0.9 re-uploads all the photos if upload history is cleared

Open foxel opened this issue 10 months ago • 11 comments

Trying to mitigate #1062 I cleared the seafile application data and it started uploading photos. The problem is that it started uploading all the already uploaded photos from the device. This never happened to me when I cleared the 3.0.7 version (and older) data.

To make sure the issue is caused by a recent update, I've done a review of changes, and I think I've found the root cause in #1056. Please see the review comments added to that PR

The application also starts all the uploads again if I click "Clear All Tasks" on the "Transfer list" screen. Which makes this issue even worse.

Related: #503

foxel avatar Mar 07 '25 15:03 foxel

Maybe this would be useful:

Subject: [PATCH] Fix duplicate uploads
---
Index: app/src/main/java/com/seafile/seadroid2/framework/data/db/entities/FileTransferEntity.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/app/src/main/java/com/seafile/seadroid2/framework/data/db/entities/FileTransferEntity.java b/app/src/main/java/com/seafile/seadroid2/framework/data/db/entities/FileTransferEntity.java
--- a/app/src/main/java/com/seafile/seadroid2/framework/data/db/entities/FileTransferEntity.java	(revision a3e046c30de68ffb33914762b055e062747f92e1)
+++ b/app/src/main/java/com/seafile/seadroid2/framework/data/db/entities/FileTransferEntity.java	(date 1741369900482)
@@ -508,6 +508,7 @@
 
 
         if (remoteDirent != null) {
+            entity.file_id = remoteDirent.id;
             entity.transfer_action = TransferAction.UPLOAD;
             entity.result = TransferResult.TRANSMITTED.name();
             entity.transfer_status = TransferStatus.SUCCEEDED;
Index: app/src/main/java/com/seafile/seadroid2/framework/worker/upload/MediaBackupScannerWorker.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/app/src/main/java/com/seafile/seadroid2/framework/worker/upload/MediaBackupScannerWorker.java b/app/src/main/java/com/seafile/seadroid2/framework/worker/upload/MediaBackupScannerWorker.java
--- a/app/src/main/java/com/seafile/seadroid2/framework/worker/upload/MediaBackupScannerWorker.java	(revision a3e046c30de68ffb33914762b055e062747f92e1)
+++ b/app/src/main/java/com/seafile/seadroid2/framework/worker/upload/MediaBackupScannerWorker.java	(date 1741414177337)
@@ -23,6 +23,7 @@
 import com.seafile.seadroid2.account.Account;
 import com.seafile.seadroid2.account.SupportAccountManager;
 import com.seafile.seadroid2.enums.TransferResult;
+import com.seafile.seadroid2.enums.TransferStatus;
 import com.seafile.seadroid2.framework.data.db.AppDatabase;
 import com.seafile.seadroid2.framework.data.db.entities.DirentModel;
 import com.seafile.seadroid2.framework.data.db.entities.FileTransferEntity;
@@ -55,6 +56,7 @@
 import java.util.Set;
 import java.util.UUID;
 import java.util.concurrent.TimeUnit;
+import java.util.regex.Pattern;
 
 import okhttp3.RequestBody;
 import retrofit2.Call;
@@ -388,8 +390,8 @@
 
             FileTransferEntity transferEntity;
             if (!CollectionUtils.isEmpty(transferEntityList)) {
-                SLogs.d("skip file -> dateAddedString" + dateAddedString + ", localPath " + localPath + ", because we have uploaded it in the past.");
-                transferEntity = transferEntityList.get(0);
+                SLogs.d("skip file -> dateAddedString " + dateAddedString + ", localPath " + localPath + ", because we have uploaded it in the past.");
+                continue;
             } else {
                 transferEntity = convertLocalFilePathToEntity(bucketName, localPath);
                 SLogs.d("new file -> dateAddedString " + dateAddedString + ", localPath " + localPath);
@@ -486,21 +488,33 @@
                 continue;
             }
 
-            //check whether the file in the parent directory contains the bucket name
+            // search for existing remote file matching
+            String filename = transferEntity.getFileName();
+            String prefix = filename.substring(0, filename.lastIndexOf("."));
+            String suffix = filename.substring(filename.lastIndexOf("."));
+            Pattern pattern = Pattern.compile(Pattern.quote(prefix) + "( \\(\\d+\\))?" + Pattern.quote(suffix));
             DirentModel exitsDirent = null;
-            Optional<DirentModel> firstOp = remoteList.stream().filter(f -> f.name.equals(transferEntity.file_name)).findFirst();
+            /*
+             * It would be cool if the API2 offered a way to query the hash of a remote file.
+             * Currently, comparing the file size is the best we can do.
+             */
+            Optional<DirentModel> firstOp = remoteList.stream()
+                    .filter(f -> pattern.matcher(f.name).matches() && f.size == transferEntity.file_size)
+                    .findFirst();
             if (firstOp.isPresent()) {
                 exitsDirent = firstOp.get();
             }
 
-            if (exitsDirent == null) {
-                transferList.add(transferEntity);
-                continue;
+            if (exitsDirent != null) {
+                SLogs.d("mark file uploaded -> filename " + filename + ", because we found existing file " + exitsDirent.id);
+                transferEntity.file_id = exitsDirent.id;
+                transferEntity.repo_id = repoConfig.getRepoId();
+                transferEntity.repo_name = repoConfig.getRepoName();
+                transferEntity.result = TransferResult.TRANSMITTED.name();
+                transferEntity.transfer_status = TransferStatus.SUCCEEDED;
             }
 
-            if (!TextUtils.equals(transferEntity.file_id, exitsDirent.id)) {
-                transferList.add(transferEntity);
-            }
+            transferList.add(transferEntity);
         }
 
         if (!CollectionUtils.isEmpty(transferList)) {

foxel avatar Mar 07 '25 17:03 foxel

I can confirm this issue. I was having issues where Seadroid was refusing to sync, so I toggled Camera Upload on and off. I guess that might've reset something, because afterwards it decided to reupload all 12 000 photos and videos I have on my phone. It took for ever and created a "(1)" suffixed duplicate of each one. I'm cleaning it up now. This can't possibly be intended behaviour? It shouldn't upload the file a second time if the same file of the same size already exists, I would argue.

bertvandepoel avatar Mar 09 '25 11:03 bertvandepoel

We have refactored the album upload logic structure, released in v3.0.10.

zhwanng avatar Apr 22 '25 02:04 zhwanng

@zhwanng Tested on an emulator. It looks like working fine in terms of not uploading existing files. And it seems like the background upload worked now.

The only thing that caught my eye is that uploaded files are sometimes shown twice in the uploads list:

Image

foxel avatar Apr 23 '25 10:04 foxel

Yes, the "uploading list" shows the data from the memory of the app. Once the app restarts, it will not show again. The "upload finished list" shows the data from the db.no matter whether the app restarts or not, it will only always show some of the latest uploaded data.

zhwanng avatar Apr 23 '25 14:04 zhwanng

And today 3.0.10 did it again for no particular reason. O just noticed long upload and found 600+ existing files were uploaded second time

Image Image

foxel avatar Apr 30 '25 19:04 foxel

Can you provide the log file in the "/sdcard/android/data/com.seafile.seadroid2" folder, or provide the following steps, or other useful information?

zhwanng avatar May 07 '25 09:05 zhwanng

Hi, @zhwanng

I was unable to retrieve the suggested file from non-rooted phone file system. If I experience this once again and will able to deduct steps to reproduce I'll post here.

I've updated to 3.0.11 BTW. The changes show some fix for downloaded files re-upload. Can this issue be the cause of my case?

foxel avatar May 09 '25 10:05 foxel

@foxel Thank you. however, that fix will have no impact on your case.

zhwanng avatar May 09 '25 11:05 zhwanng

We have released the latest release, adding the log export feature on the settings page, if you encounter this problem again next time, you can share the log content if possible.

zhwanng avatar May 30 '25 10:05 zhwanng

We have released the latest release, adding the log export feature on the settings page, if you encounter this problem again next time, you can share the log content if possible.

@zhwanng this is really cool. The app works. And I did not catch any duplicate uploads so far. And now I can finally give some info on background sync:

Versions from 3.x.x were struggling to do background upload at all on my physical Pixel 9.

Version 3.0.14 does trigger something but the file ends up with cancelled upload in UI and I can't retry it besides force-closing the app and opening it again.

Here is the log from this:

2025-05-30 15:34:23.728 E/Seafile-SLog: media store：newVersion -> 1506:f32e5823-d1e8-40cc-98dd-defe04a775b6, lastVersion -> 1506:f32e5823-d1e8-40cc-98dd-defe04a775b6
2025-05-30 15:34:23.728 E/Seafile-SLog: A new file is detected and the Media task begins
2025-05-30 15:34:23.731 D/Seafile-SLog: CameraUploadManager, performSyncByStatus()
2025-05-30 15:34:23.751 D/Seafile-SLog: CameraSyncService, onCreate
2025-05-30 15:34:23.758 D/Seafile-SLog: AlbumBackupAdapter, onPerformSync(),  isForce = false
2025-05-30 15:34:23.760 D/Seafile-SLog: BackgroundJobManagerImpl, startMediaBackupChain(), isForce:false
2025-05-30 15:34:23.765 D/Seafile-SLog: BackgroundJobManagerImpl, oneTimeRequestBuilder(), Creating WorkRequest for: MediaBackupScanWorker
2025-05-30 15:34:23.767 D/Seafile-SLog: BackgroundJobManagerImpl, oneTimeRequestBuilder(), Creating WorkRequest for: MediaBackupUploadWorker
2025-05-30 15:34:23.837 D/Seafile-SLog: MediaBackupScanWorker, doWork(), started execution
2025-05-30 15:34:23.849 D/Seafile-SLog: MediaBackupScanWorker, doWork(), start scan
2025-05-30 15:34:23.850 D/Seafile-SLog: repo fragment, event: scanning, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:34:23.850 D/Seafile-SLog: TabSettings2Fragment, doBusWork(), on event: scanning, dataSource: ALBUM_BACKUP
2025-05-30 15:34:23.851 D/Seafile-SLog: DownloadListFragment, on event: scanning, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:34:23.851 D/Seafile-SLog: UploadListFragment, on event: scanning, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:34:23.884 D/Seafile-SLog: MediaBackupScanWorker, loadImages(), images query count : 1
2025-05-30 15:34:23.913 D/Seafile-SLog: MediaBackupScanWorker, iterateCursor(), new file -> [wait for check] /storage/emulated/0/DCIM/Camera/PXL_20250530_113413628.jpg
2025-05-30 15:34:23.922 D/Seafile-SLog: MediaBackupScanWorker, loadVideos(), video query count : 0
2025-05-30 15:34:23.924 D/Seafile-SLog: MediaBackupScanWorker, doWork(), album backup scan time：73.91 ms
2025-05-30 15:34:25.100 D/Seafile-SLog: MediaBackupScanWorker, checkAndInsert(), skip file -> [remote exists] PXL_20250530_113224939.jpg, because we have uploaded it in the past.
2025-05-30 15:34:25.111 D/Seafile-SLog: MediaBackupScanWorker, checkAndInsert(), skip file -> [remote exists] PXL_20250530_113235079.jpg, because we have uploaded it in the past.
2025-05-30 15:34:25.112 D/Seafile-SLog: repo fragment, event: scan_end, dataSource: ALBUM_BACKUP, count: 1
2025-05-30 15:34:25.112 D/Seafile-SLog: TabSettings2Fragment, doBusWork(), on event: scan_end, dataSource: ALBUM_BACKUP
2025-05-30 15:34:25.112 D/Seafile-SLog: DownloadListFragment, on event: scan_end, dataSource: ALBUM_BACKUP, count: 1
2025-05-30 15:34:25.112 D/Seafile-SLog: UploadListFragment, on event: scan_end, dataSource: ALBUM_BACKUP, count: 1
2025-05-30 15:34:25.221 D/Seafile-SLog: MediaBackupUploadWorker, doWork(), started execution
2025-05-30 15:34:25.228 D/Seafile-SLog: BaseUploadWorker, transfer start, model:
2025-05-30 15:34:25.228 D/Seafile-SLog: BaseUploadWorker, TransferModel{id='3B95A6A054F19E9566F92D830B2D87E4', repo_id='9849654b-c084-4b34-bd9c-0baeac628250', file_name='PXL_20250530_113413628.jpg', data_source=ALBUM_BACKUP, save_to=DB, full_path='/storage/emulated/0/DCIM/Camera/PXL_20250530_113413628.jpg'}
2025-05-30 15:34:25.228 D/Seafile-SLog: BaseUploadWorker, transferFile(), start transfer, local file path: /storage/emulated/0/DCIM/Camera/PXL_20250530_113413628.jpg
2025-05-30 15:34:25.234 D/Seafile-SLog: repo fragment, event: file_in_transfer, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:34:25.234 D/Seafile-SLog: TabSettings2Fragment, doBusWork(), on event: file_in_transfer, dataSource: ALBUM_BACKUP
2025-05-30 15:34:25.235 D/Seafile-SLog: DownloadListFragment, on event: file_in_transfer, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:34:25.235 D/Seafile-SLog: UploadListFragment, on event: file_in_transfer, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:34:25.235 D/Seafile-SLog: BaseUploadWorker, transferFile(), start transfer, remote path: /My Photos/Camera/PXL_20250530_113413628.jpg
2025-05-30 15:34:25.982 E/Seafile-SLog: media store：newVersion -> 1506:f32e5823-d1e8-40cc-98dd-defe04a775b6, lastVersion -> 1506:f32e5823-d1e8-40cc-98dd-defe04a775b6
2025-05-30 15:34:25.982 E/Seafile-SLog: A new file is detected and the Media task begins
2025-05-30 15:34:25.982 D/Seafile-SLog: CameraUploadManager, performSyncByStatus()
2025-05-30 15:34:25.991 E/Seafile-SLog: media store：newVersion -> 1506:f32e5823-d1e8-40cc-98dd-defe04a775b6, lastVersion -> 1506:f32e5823-d1e8-40cc-98dd-defe04a775b6
2025-05-30 15:34:25.991 E/Seafile-SLog: A new file is detected and the Media task begins
2025-05-30 15:34:25.991 D/Seafile-SLog: CameraUploadManager, performSyncByStatus()
2025-05-30 15:34:25.996 E/Seafile-SLog: media store：newVersion -> 1506:f32e5823-d1e8-40cc-98dd-defe04a775b6, lastVersion -> 1506:f32e5823-d1e8-40cc-98dd-defe04a775b6
2025-05-30 15:34:25.996 E/Seafile-SLog: A new file is detected and the Media task begins
2025-05-30 15:34:25.996 D/Seafile-SLog: CameraUploadManager, performSyncByStatus()
2025-05-30 15:34:26.004 D/Seafile-SLog: CameraSyncService, onCreate
2025-05-30 15:34:26.007 D/Seafile-SLog: AlbumBackupAdapter, onPerformSync(),  isForce = false
2025-05-30 15:34:26.009 D/Seafile-SLog: BackgroundJobManagerImpl, startMediaBackupChain(), isForce:false
2025-05-30 15:34:26.009 D/Seafile-SLog: BackgroundJobManagerImpl, oneTimeRequestBuilder(), Creating WorkRequest for: MediaBackupScanWorker
2025-05-30 15:34:26.010 D/Seafile-SLog: BackgroundJobManagerImpl, oneTimeRequestBuilder(), Creating WorkRequest for: MediaBackupUploadWorker
2025-05-30 15:34:26.016 D/Seafile-SLog: BaseUploadWorker, onStopped()
2025-05-30 15:34:26.020 E/Seafile-SLog: java.io.IOException: Canceled
2025-05-30 15:34:26.023 E/Seafile-SLog: An exception occurred and the next transfer will continue
2025-05-30 15:34:26.025 D/Seafile-SLog: MediaBackupUploadWorker, doWork(), complete
2025-05-30 15:34:26.029 D/Seafile-SLog: repo fragment, event: transfer_finish, dataSource: ALBUM_BACKUP, count: 1
2025-05-30 15:34:26.033 D/Seafile-SLog: TabSettings2Fragment, doBusWork(), on event: transfer_finish, dataSource: ALBUM_BACKUP
2025-05-30 15:34:26.033 D/Seafile-SLog: DownloadListFragment, on event: transfer_finish, dataSource: ALBUM_BACKUP, count: 1
2025-05-30 15:34:26.033 D/Seafile-SLog: UploadListFragment, on event: transfer_finish, dataSource: ALBUM_BACKUP, count: 1
2025-05-30 15:34:26.106 D/Seafile-SLog: MediaBackupScanWorker, doWork(), started execution
2025-05-30 15:34:26.112 D/Seafile-SLog: MediaBackupScanWorker, doWork(), start scan
2025-05-30 15:34:26.113 D/Seafile-SLog: repo fragment, event: scanning, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:34:26.113 D/Seafile-SLog: TabSettings2Fragment, doBusWork(), on event: scanning, dataSource: ALBUM_BACKUP
2025-05-30 15:34:26.113 D/Seafile-SLog: DownloadListFragment, on event: scanning, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:34:26.113 D/Seafile-SLog: UploadListFragment, on event: scanning, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:34:26.133 D/Seafile-SLog: MediaBackupScanWorker, loadImages(), images query count : 0
2025-05-30 15:34:26.137 D/Seafile-SLog: MediaBackupScanWorker, loadVideos(), video query count : 0
2025-05-30 15:34:26.139 D/Seafile-SLog: MediaBackupScanWorker, doWork(), album backup scan time：25.07 ms
2025-05-30 15:34:27.129 D/Seafile-SLog: repo fragment, event: scan_end, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:34:27.129 D/Seafile-SLog: TabSettings2Fragment, doBusWork(), on event: scan_end, dataSource: ALBUM_BACKUP
2025-05-30 15:34:27.129 D/Seafile-SLog: DownloadListFragment, on event: scan_end, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:34:27.129 D/Seafile-SLog: UploadListFragment, on event: scan_end, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:34:27.712 D/Seafile-SLog: FileSyncService, onStart()
2025-05-30 15:34:27.748 D/Seafile-SLog: FileSyncService, onStop()
2025-05-30 15:34:32.749 D/Seafile-SLog: FileSyncService, onStart()
2025-05-30 15:34:32.805 D/Seafile-SLog: FileSyncService, onStop()
2025-05-30 15:34:37.806 D/Seafile-SLog: FileSyncService, onStart()
2025-05-30 15:34:37.859 D/Seafile-SLog: FileSyncService, onStop()
2025-05-30 15:34:42.859 D/Seafile-SLog: FileSyncService, onStart()
2025-05-30 15:34:42.907 D/Seafile-SLog: FileSyncService, onStop()
2025-05-30 15:34:47.908 D/Seafile-SLog: FileSyncService, onStart()
2025-05-30 15:34:47.923 D/Seafile-SLog: FileSyncService, onStop()
2025-05-30 15:34:49.150 D/Seafile-SLog: onCleared
2025-05-30 15:34:49.151 D/Seafile-SLog: CompositeDisposable dispose
2025-05-30 15:34:49.163 D/Seafile-SLog: onCleared
2025-05-30 15:34:49.163 D/Seafile-SLog: CompositeDisposable dispose
2025-05-30 15:34:49.164 D/Seafile-SLog: onCleared
2025-05-30 15:34:49.164 D/Seafile-SLog: CompositeDisposable dispose
2025-05-30 15:34:49.165 D/Seafile-SLog: CompositeDisposable clear all
2025-05-30 15:34:49.171 D/Seafile-SLog: onCleared
2025-05-30 15:34:49.171 D/Seafile-SLog: CompositeDisposable dispose
2025-05-30 15:34:49.185 D/Seafile-SLog: onCleared
2025-05-30 15:34:49.185 D/Seafile-SLog: CompositeDisposable dispose
2025-05-30 15:34:49.187 D/Seafile-SLog: onCleared
2025-05-30 15:34:49.187 D/Seafile-SLog: CompositeDisposable dispose
2025-05-30 15:34:49.198 D/Seafile-SLog: onCleared
2025-05-30 15:34:49.198 D/Seafile-SLog: CompositeDisposable dispose
2025-05-30 15:34:49.199 D/Seafile-SLog: onCleared
2025-05-30 15:34:49.199 D/Seafile-SLog: CompositeDisposable dispose

The messages looking particularly interesting are these ones (BTW the app is allowed to access all the files):

2025-05-30 15:34:26.020 E/Seafile-SLog: java.io.IOException: Canceled
2025-05-30 15:34:26.023 E/Seafile-SLog: An exception occurred and the next transfer will continue

Image

When I force-closed and re-open the app it uploaded the file. The log part is here:

2025-05-30 15:35:18.879 D/Seafile-SLog: App Env Info
2025-05-30 15:35:18.879 D/Seafile-SLog: SDK: 35
2025-05-30 15:35:18.879 D/Seafile-SLog: Brand: google
2025-05-30 15:35:18.879 D/Seafile-SLog: Model: Pixel 9
2025-05-30 15:35:18.879 D/Seafile-SLog: Release: 15
2025-05-30 15:35:18.879 D/Seafile-SLog: App Version: 3.0.14
2025-05-30 15:35:19.585 D/Seafile-SLog: CameraUploadManager, performSync()
2025-05-30 15:35:19.586 D/Seafile-SLog: BackgroundJobManagerImpl, startDownloadChain()
2025-05-30 15:35:19.586 D/Seafile-SLog: BackgroundJobManagerImpl, oneTimeRequestBuilder(), Creating WorkRequest for: DownloadFileScannerWorker
2025-05-30 15:35:19.586 D/Seafile-SLog: BackgroundJobManagerImpl, oneTimeRequestBuilder(), Creating WorkRequest for: DownloadWorker
2025-05-30 15:35:19.586 D/Seafile-SLog: BackgroundJobManagerImpl, startFolderBackupChain(), isForce:false
2025-05-30 15:35:19.586 D/Seafile-SLog: BackgroundJobManagerImpl, cancelFolderBackupWorker()
2025-05-30 15:35:19.587 D/Seafile-SLog: BackgroundJobManagerImpl, oneTimeRequestBuilder(), Creating WorkRequest for: FolderBackupScanWorker
2025-05-30 15:35:19.587 D/Seafile-SLog: BackgroundJobManagerImpl, oneTimeRequestBuilder(), Creating WorkRequest for: FolderBackupUploadWorker
2025-05-30 15:35:19.587 D/Seafile-SLog: BackgroundJobManagerImpl, startFileUploadWorker()
2025-05-30 15:35:19.587 D/Seafile-SLog: BackgroundJobManagerImpl, oneTimeRequestBuilder(), Creating WorkRequest for: FileUploadWorker
2025-05-30 15:35:19.589 D/Seafile-SLog: FileSyncService, scanLocalCacheFile: cacheList size is 8
2025-05-30 15:35:19.604 D/Seafile-SLog: MainActivity, bond FileSyncService
2025-05-30 15:35:19.604 D/Seafile-SLog: CameraSyncService, onCreate
2025-05-30 15:35:19.605 D/Seafile-SLog: FileSyncService, scanLocalCacheFile, skip: local file md5 is same, path: /My Photos/Camera/PXL_20250501_071442310.jpg
2025-05-30 15:35:19.605 D/Seafile-SLog: AlbumBackupAdapter, onPerformSync(),  isForce = true
2025-05-30 15:35:19.606 D/Seafile-SLog: BackgroundJobManagerImpl, startMediaBackupChain(), isForce:true
2025-05-30 15:35:19.606 D/Seafile-SLog: BackgroundJobManagerImpl, oneTimeRequestBuilder(), Creating WorkRequest for: MediaBackupScanWorker
2025-05-30 15:35:19.607 D/Seafile-SLog: BackgroundJobManagerImpl, oneTimeRequestBuilder(), Creating WorkRequest for: MediaBackupUploadWorker
2025-05-30 15:35:19.615 D/Seafile-SLog: FileSyncService, scanLocalCacheFile, skip: local file md5 is same, path: /My Photos/Camera/PXL_20250425_142332223.jpg
2025-05-30 15:35:19.636 D/Seafile-SLog: FileSyncService, scanLocalCacheFile, skip: local file md5 is same, path: /My Photos/Camera/PXL_20250501_071516300.jpg
2025-05-30 15:35:19.642 D/Seafile-SLog: FolderBackupScanWorker, doWork(), started execution
2025-05-30 15:35:19.642 D/Seafile-SLog: FileUploadWorker, start(), started execution
2025-05-30 15:35:19.642 D/Seafile-SLog: DownloadFileScannerWorker, doWork(), started execution
2025-05-30 15:35:19.642 D/Seafile-SLog: MediaBackupScanWorker, doWork(), started execution
2025-05-30 15:35:19.643 D/Seafile-SLog: FolderBackupScanWorker, doWork(), The folder scan task was not started, because the switch is off
2025-05-30 15:35:19.645 D/Seafile-SLog: MediaBackupScanWorker, doWork(), start scan
2025-05-30 15:35:19.647 D/Seafile-SLog: FileSyncService, scanLocalCacheFile, skip: local file md5 is same, path: /My Photos/Camera/PXL_20250501_071445428.jpg
2025-05-30 15:35:19.656 D/Seafile-SLog: FileSyncService, scanLocalCacheFile, skip: local file md5 is same, path: /My Photos/Camera/PXL_20250425_163451669.jpg
2025-05-30 15:35:19.657 D/Seafile-SLog: repo fragment, event: transfer_finish, dataSource: FILE_BACKUP, count: 0
2025-05-30 15:35:19.657 D/Seafile-SLog: repo fragment, event: scan_end, dataSource: FOLDER_BACKUP, count: 0
2025-05-30 15:35:19.657 D/Seafile-SLog: repo fragment, event: scan_end, dataSource: DOWNLOAD, count: 0
2025-05-30 15:35:19.657 D/Seafile-SLog: repo fragment, event: scanning, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:35:19.659 D/Seafile-SLog: MediaBackupScanWorker, loadImages(), images query count : 1557
2025-05-30 15:35:19.664 D/Seafile-SLog: MediaBackupScanWorker, iterateCursor(), new file -> [wait for check] /storage/emulated/0/DCIM/Camera/PXL_20250530_113452560.jpg
2025-05-30 15:35:19.665 D/Seafile-SLog: MediaBackupScanWorker, iterateCursor(), new file -> [wait for check] /storage/emulated/0/DCIM/Camera/PXL_20250530_113413628.jpg
2025-05-30 15:35:19.666 D/Seafile-SLog: MediaBackupScanWorker, iterateCursor(), skip file -> [local exists] /storage/emulated/0/DCIM/Camera/PXL_20250530_113235079.jpg, because we have uploaded it in the past.
...
2025-05-30 15:35:19.852 D/Seafile-SLog: MediaBackupScanWorker, iterateCursor(), new file -> [wait for check] /storage/emulated/0/DCIM/Camera/PXL_20250515_165058849.jpg
...
2025-05-30 15:35:20.489 D/Seafile-SLog: MediaBackupScanWorker, iterateCursor(), new file -> [wait for check] /storage/emulated/0/DCIM/Camera/PXL_20250329_111124167.jpg
2025-05-30 15:35:20.491 D/Seafile-SLog: MediaBackupScanWorker, iterateCursor(), new file -> [wait for check] /storage/emulated/0/DCIM/Camera/PXL_20250328_100326791.jpg
...
2025-05-30 15:35:21.706 D/Seafile-SLog: MediaBackupScanWorker, loadVideos(), video query count : 45
2025-05-30 15:35:21.716 D/Seafile-SLog: MediaBackupScanWorker, iterateCursor(), skip file -> [local exists] /storage/emulated/0/DCIM/Camera/PXL_20250530_054417942.mp4, because we have uploaded it in the past.
2025-05-30 15:35:21.719 D/Seafile-SLog: MediaBackupScanWorker, iterateCursor(), skip file -> [local exists] /storage/emulated/0/DCIM/Camera/PXL_20250530_054402160.mp4, because we have uploaded it in the past.
...
2025-05-30 15:35:21.975 D/Seafile-SLog: MediaBackupScanWorker, doWork(), album backup scan time：2.329 s
2025-05-30 15:35:23.625 D/Seafile-SLog: MediaBackupScanWorker, checkAndInsert(), skip file -> [remote exists] PXL_20250305_154715107.jpg, because we have uploaded it in the past.
2025-05-30 15:35:23.627 D/Seafile-SLog: MediaBackupScanWorker, checkAndInsert(), skip file -> [remote exists] PXL_20250220_073828147.mp4, because we have uploaded it in the past.
...
2025-05-30 15:35:24.294 D/Seafile-SLog: repo fragment, event: scan_end, dataSource: ALBUM_BACKUP, count: 2
2025-05-30 15:35:24.294 D/Seafile-SLog: TabSettings2Fragment, doBusWork(), on event: scan_end, dataSource: ALBUM_BACKUP
2025-05-30 15:35:24.322 D/Seafile-SLog: MediaBackupUploadWorker, doWork(), started execution
2025-05-30 15:35:24.325 D/Seafile-SLog: BaseUploadWorker, transfer start, model:
2025-05-30 15:35:24.325 D/Seafile-SLog: BaseUploadWorker, TransferModel{id='06D9DC495E19B7A90E01972275ED2F32', repo_id='9849654b-c084-4b34-bd9c-0baeac628250', file_name='PXL_20250530_113452560.jpg', data_source=ALBUM_BACKUP, save_to=DB, full_path='/storage/emulated/0/DCIM/Camera/PXL_20250530_113452560.jpg'}
2025-05-30 15:35:24.325 D/Seafile-SLog: BaseUploadWorker, transferFile(), start transfer, local file path: /storage/emulated/0/DCIM/Camera/PXL_20250530_113452560.jpg
2025-05-30 15:35:24.328 D/Seafile-SLog: repo fragment, event: file_in_transfer, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:35:24.328 D/Seafile-SLog: TabSettings2Fragment, doBusWork(), on event: file_in_transfer, dataSource: ALBUM_BACKUP
2025-05-30 15:35:24.329 D/Seafile-SLog: BaseUploadWorker, transferFile(), start transfer, remote path: /My Photos/Camera/PXL_20250530_113452560.jpg
2025-05-30 15:35:24.742 D/Seafile-SLog: FileSyncService, onStart()
2025-05-30 15:35:24.766 D/Seafile-SLog: FileSyncService, onStop()
2025-05-30 15:35:25.679 D/Seafile-SLog: BaseUploadWorker, onProgressNotify(), UPLOAD: PXL_20250530_113452560.jpg -> progress：34
2025-05-30 15:35:25.728 D/Seafile-SLog: repo fragment, event: file_in_transfer, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:35:25.728 D/Seafile-SLog: TabSettings2Fragment, doBusWork(), on event: file_in_transfer, dataSource: ALBUM_BACKUP
2025-05-30 15:35:25.729 D/Seafile-SLog: DownloadListFragment, on event: file_in_transfer, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:35:25.729 D/Seafile-SLog: UploadListFragment, on event: file_in_transfer, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:35:27.665 D/Seafile-SLog: BaseUploadWorker, transferFile(), result，file ID：8791fbe466abfe7cfd67a25a1c5ffe0e3265fa0b
2025-05-30 15:35:27.678 D/Seafile-SLog: BaseUploadWorker, transfer start, model:
2025-05-30 15:35:27.678 D/Seafile-SLog: BaseUploadWorker, TransferModel{id='3B95A6A054F19E9566F92D830B2D87E4', repo_id='9849654b-c084-4b34-bd9c-0baeac628250', file_name='PXL_20250530_113413628.jpg', data_source=ALBUM_BACKUP, save_to=DB, full_path='/storage/emulated/0/DCIM/Camera/PXL_20250530_113413628.jpg'}
2025-05-30 15:35:27.678 D/Seafile-SLog: BaseUploadWorker, transferFile(), start transfer, local file path: /storage/emulated/0/DCIM/Camera/PXL_20250530_113413628.jpg
2025-05-30 15:35:27.681 D/Seafile-SLog: repo fragment, event: file_transfer_success, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:35:27.681 D/Seafile-SLog: TabSettings2Fragment, doBusWork(), on event: file_transfer_success, dataSource: ALBUM_BACKUP
2025-05-30 15:35:27.681 D/Seafile-SLog: DownloadListFragment, on event: file_transfer_success, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:35:27.681 D/Seafile-SLog: UploadListFragment, on event: file_transfer_success, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:35:27.681 D/Seafile-SLog: BaseUploadWorker, transferFile(), start transfer, remote path: /My Photos/Camera/PXL_20250530_113413628.jpg
2025-05-30 15:35:27.682 D/Seafile-SLog: repo fragment, event: file_in_transfer, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:35:27.682 D/Seafile-SLog: TabSettings2Fragment, doBusWork(), on event: file_in_transfer, dataSource: ALBUM_BACKUP
2025-05-30 15:35:27.682 D/Seafile-SLog: DownloadListFragment, on event: file_in_transfer, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:35:27.682 D/Seafile-SLog: UploadListFragment, on event: file_in_transfer, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:35:27.810 D/Seafile-SLog: BaseUploadWorker, transferFile(), newCall has executed(), cancel it
2025-05-30 15:35:28.765 D/Seafile-SLog: BaseUploadWorker, onProgressNotify(), UPLOAD: PXL_20250530_113413628.jpg -> progress：100
2025-05-30 15:35:28.769 D/Seafile-SLog: repo fragment, event: file_in_transfer, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:35:28.769 D/Seafile-SLog: TabSettings2Fragment, doBusWork(), on event: file_in_transfer, dataSource: ALBUM_BACKUP
2025-05-30 15:35:28.770 D/Seafile-SLog: DownloadListFragment, on event: file_in_transfer, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:35:28.770 D/Seafile-SLog: UploadListFragment, on event: file_in_transfer, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:35:29.768 D/Seafile-SLog: FileSyncService, onStart()
2025-05-30 15:35:29.817 D/Seafile-SLog: BaseUploadWorker, transferFile(), result，file ID：bbf239797aa190e0a49765a6608a48afd044df1c
2025-05-30 15:35:29.827 D/Seafile-SLog: FileSyncService, onStop()
2025-05-30 15:35:29.827 D/Seafile-SLog: repo fragment, event: file_transfer_success, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:35:29.828 D/Seafile-SLog: TabSettings2Fragment, doBusWork(), on event: file_transfer_success, dataSource: ALBUM_BACKUP
2025-05-30 15:35:29.828 D/Seafile-SLog: MediaBackupUploadWorker, doWork(), complete
2025-05-30 15:35:29.828 D/Seafile-SLog: DownloadListFragment, on event: file_transfer_success, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:35:29.828 D/Seafile-SLog: UploadListFragment, on event: file_transfer_success, dataSource: ALBUM_BACKUP, count: 0
2025-05-30 15:35:29.836 D/Seafile-SLog: repo fragment, event: transfer_finish, dataSource: ALBUM_BACKUP, count: 2
2025-05-30 15:35:29.840 D/Seafile-SLog: TabSettings2Fragment, doBusWork(), on event: transfer_finish, dataSource: ALBUM_BACKUP
2025-05-30 15:35:29.840 D/Seafile-SLog: DownloadListFragment, on event: transfer_finish, dataSource: ALBUM_BACKUP, count: 2
2025-05-30 15:35:29.840 D/Seafile-SLog: UploadListFragment, on event: transfer_finish, dataSource: ALBUM_BACKUP, count: 2
2025-05-30 15:35:34.829 D/Seafile-SLog: FileSyncService, onStart()
2025-05-30 15:35:34.894 D/Seafile-SLog: FileSyncService, onStop()
2025-05-30 15:35:39.895 D/Seafile-SLog: FileSyncService, onStart()
2025-05-30 15:35:39.950 D/Seafile-SLog: FileSyncService, onStop()
2025-05-30 15:35:40.895 D/Seafile-SLog: MainActivity, onRestart
2025-05-30 15:35:41.467 D/Seafile-SLog: onCleared
2025-05-30 15:35:41.467 D/Seafile-SLog: CompositeDisposable dispose
2025-05-30 15:35:41.467 D/Seafile-SLog: onCleared
2025-05-30 15:35:41.468 D/Seafile-SLog: CompositeDisposable dispose
2025-05-30 15:35:44.952 D/Seafile-SLog: FileSyncService, onStart()
2025-05-30 15:35:45.026 D/Seafile-SLog: FileSyncService, onStop()
2025-05-30 15:35:50.027 D/Seafile-SLog: FileSyncService, onStart()
2025-05-30 15:35:50.087 D/Seafile-SLog: FileSyncService, onStop()

foxel avatar May 30 '25 11:05 foxel

Hi @zhwanng, as of 3.0.16 beta, the app appears to be uploading in the background. And no reuploads as of now 👍

foxel avatar Jul 08 '25 07:07 foxel

@zhwanng please reopen. The 3.0.16 just re-uploaded all the photos. I've cleared the app data some days ago and today it uploaded a copy of every photo, but with different time stamp.

Also it looks like it started the uploads when the server was restarted. Looking thru the AlbumScanHelper.java code I would suspect it's related of checkAndInsert is allowed to be running when the is no connection to the server.

foxel avatar Jul 15 '25 00:07 foxel

@foxel Can you provide us with the log files?

Starting from v3.0.15, a create time field was added when uploading files, and the value is the create time/last modification time of the file. But this shouldn't affect the duplication of server files.

If your server is down, it will throw an exception and skip uploading this file.

zhwanng avatar Jul 15 '25 02:07 zhwanng

Hi @zhwanng .

I've captured the logs right after noticing the uploads. I was analyzing it. Let's see what I have for one particular file (PXL_20250115_165506824). Here are some parts I see interesting:

2025-07-15 00:06:23.457 D/SLog: AlbumBackupPeriodicScanStarter, data plan is not allowed, current network type: , NETWORK_WIFI
2025-07-15 00:06:23.457 D/SLog: AlbumBackupPeriodicScanStarter, doWork(), start scan
2025-07-15 00:06:23.457 D/SLog: AlbumScanHelper, readMediaResult(), no last scan time
2025-07-15 00:06:23.555 D/SLog: AlbumScanHelper, loadImages(), images query count : 2153, read result: true
2025-07-15 00:06:23.555 D/SLog: AlbumScanHelper, iterateCursorResult(), start iterate
...
2025-07-15 00:06:32.196 D/SLog: AlbumScanHelper, iterateCursor(), new file -> [wait for check] /storage/emulated/0/DCIM/Camera/PXL_20250115_165506824.jpg
...
2025-07-15 00:06:41.719 D/SLog: AlbumScanHelper, checkAndInsert(), skip file -> [remote exists] PXL_20250115_165506824.jpg, because we have uploaded it in the past.
...
2025-07-15 00:24:23.497 D/SLog: AlbumScanHelper, iterateCursor(), new file -> [wait for check] /storage/emulated/0/DCIM/Camera/PXL_20250115_165506824.jpg
...
2025-07-15 00:24:42.934 D/SLog: AlbumScanHelper, checkAndInsert(), skip file -> [remote exists] PXL_20250115_165506824.jpg, because we have uploaded it in the past.
...
2025-07-15 00:44:01.407 D/SLog: AlbumScanHelper, iterateCursor(), new file -> [wait for check] /storage/emulated/0/DCIM/Camera/PXL_20250115_165506824.jpg
...
2025-07-15 00:44:18.806 D/SLog: AlbumScanHelper, checkAndInsert(), skip file -> [remote exists] PXL_20250115_165506824.jpg, because we have uploaded it in the past.
...
2025-07-15 00:59:07.416 D/SLog: AlbumScanHelper, iterateCursor(), new file -> [wait for check] /storage/emulated/0/DCIM/Camera/PXL_20250115_165506824.jpg
...
2025-07-15 00:59:23.465 D/SLog: AlbumScanHelper, checkAndInsert(), skip file -> [remote exists] PXL_20250115_165506824.jpg, because we have uploaded it in the past.
...
2025-07-15 01:19:19.760 D/SLog: AlbumScanHelper, iterateCursor(), new file -> [wait for check] /storage/emulated/0/DCIM/Camera/PXL_20250115_165506824.jpg
...
2025-07-15 01:19:26.449 D/SLog: AlbumScanHelper, checkAndInsert(), skip file -> [remote exists] PXL_20250115_165506824.jpg, because we have uploaded it in the past.
...
2025-07-15 01:34:29.606 D/SLog: AlbumScanHelper, iterateCursor(), new file -> [wait for check] /storage/emulated/0/DCIM/Camera/PXL_20250115_165506824.jpg
...
2025-07-15 01:34:40.104 D/SLog: AlbumScanHelper, checkAndInsert(), skip file -> [remote exists] PXL_20250115_165506824.jpg, because we have uploaded it in the past.
...
2025-07-15 01:54:40.796 D/SLog: AlbumScanHelper, iterateCursor(), new file -> [wait for check] /storage/emulated/0/DCIM/Camera/PXL_20250115_165506824.jpg
...
2025-07-15 01:54:52.135 D/SLog: AlbumScanHelper, checkAndInsert(), skip file -> [remote exists] PXL_20250115_165506824.jpg, because we have uploaded it in the past.
...
2025-07-15 02:20:07.529 D/SLog: AlbumScanHelper, iterateCursor(), new file -> [wait for check] /storage/emulated/0/DCIM/Camera/PXL_20250115_165506824.jpg
...
2025-07-15 02:20:51.911 D/SLog: AlbumScanHelper, getDirentWrapper(), request dirents failed, Unable to parse TLS packet header
2025-07-15 02:20:51.911 D/SLog: MediaBackupScanWorker, doWork(), loadMedia() failed：Network error
2025-07-15 02:20:52.039 D/SLog: MediaBackupUploadWorker, doWork(), started execution
2025-07-15 02:20:52.043 D/SLog: MediaBackupUploadWorker, data plan is not allowed, current network type, NETWORK_WIFI
2025-07-15 02:20:52.043 E/SLog: MediaBackupUploadWorker, pending count: 2167
...
2025-07-15 04:17:18.104 D/SLog: BaseUploadWorker, transferFile(), start transfer, local file path: /storage/emulated/0/DCIM/Camera/PXL_20250115_165506824.jpg
2025-07-15 04:17:18.104 D/SLog: BaseUploadWorker, transfer start, model:
2025-07-15 04:17:18.104 D/SLog: BaseUploadWorker, TransferModel{id='A7BD81C7B417EE877BF05DE695649923', repo_id='9849654b-c084-4b34-bd9c-0baeac628250', file_name='PXL_20250115_165506824.jpg', data_source=ALBUM_BACKUP, save_to=DB, full_path='/storage/emulated/0/DCIM/Camera/PXL_20250115_165506824.jpg'}
2025-07-15 04:17:18.111 D/SLog: BaseUploadWorker, transferFile(), start transfer, remote path: /My Photos/Camera/PXL_20250115_165506824.jpg
2025-07-15 04:17:18.116 D/SLog: FileUtils, read file create timestamp success., Wed Jan 15 20:55:10 GMT+04:00 2025
2025-07-15 04:17:18.117 D/SLog: BaseUploadWorker, file create timestamp : 2025-01-15T20:55:10+04:00
2025-07-15 04:17:18.249 D/SLog: BaseUploadWorker, transferFile(), newCall has executed(), cancel it
2025-07-15 04:17:18.318 D/SLog: BaseUploadWorker, onProgressNotify(), UPLOAD: PXL_20250115_165506824.jpg -> progress：100
2025-07-15 04:17:20.759 D/SLog: BaseUploadWorker, onRes(), response code: 200, protocol: http/1.1
2025-07-15 04:17:20.759 D/SLog: BaseUploadWorker, checkProtocol(), protocol: http/1.1
2025-07-15 04:17:20.759 D/SLog: BaseUploadWorker, transferFile(), result，file ID：7e8b77a193880684108038f0c31c6e00fe421a43
...

So I see that the critical flaw is that if in AlbumScanHelper::checkAndInsert an error prevented the file from being checked for existence on the host, the file reference stayed in GlobalTransferCacheList.ALBUM_BACKUP_QUEUE and then was uploaded. I believe, in my case, the scanning happened right when I was restarting the server - you can see the error in the logs.

A minor flaw I see here also is that once checkAndInsert does it's job the files on the device are being ignored only for one run, and each run they get re-checked. This means in case the app is reinstalled on a phone where the majority of files are already on the server, the re-checks are checking enormous about of files each time.

Another noticeable change from uploadness detection existed before 3.x.x, that I personally feel bad about, is that now only the names matching left and the size checking is removed. IMO, checking not only that a candidate with a matching name is found, but also whatever we can to exclude clearly not the same file, is still needed here:

            /*
             * It would be cool if the API2 offered a way to query the hash of a remote file.
             * Currently, comparing the file size is the best we can do.
             */
            Optional<DirentModel> firstOp = remoteList.stream()
                    .filter(f -> pattern.matcher(f.name).matches())
                    .findFirst();

            if (firstOp.isPresent()) {
                SafeLogs.d(TAG, "checkAndInsert()", "skip file -> [remote exists] " + filename + ", because we have uploaded it in the past.");

                GlobalTransferCacheList.ALBUM_BACKUP_QUEUE.remove(bucketName, transferModel);
            }

P.S. I can send you entire 9+ MB log in some PM if needed

P.P.S. The fact that now the camera upload sets the time of the file on the server to match the modification time on the phone makes removing the duplicates a much bigger pain. The original 3.0.9 issue was easier to clean up after the disaster. IMHO, this is not really a beneficial change.

foxel avatar Jul 15 '25 06:07 foxel

The changes I suggest to fix the issue at this point would be:

Making sure the files don't get pushed to ALBUM_BACKUP_QUEUE before the check is made, or making the ALBUM_BACKUP_QUEUE members have an explicit marker for whether the file was successfully checked and is approved for uploading.

And I would restore the size checking for the firstOp filter routine

foxel avatar Jul 15 '25 06:07 foxel

Hi, @foxel

Thank you for your analysis.

So I see that the critical flaw is that if in AlbumScanHelper::checkAndInsert an error prevented the file from being checked for existence on the host, the file reference stayed in GlobalTransferCacheList.ALBUM_BACKUP_QUEUE and then was uploaded. I believe, in my case, the scanning happened right when I was restarting the server - you can see the error in the logs.

I found out what the problem is.

A minor flaw I see here also is that once checkAndInsert does it's job the files on the device are being ignored only for one run, and each run they get re-checked. This means in case the app is reinstalled on a phone where the majority of files are already on the server, the re-checks are checking enormous about of files each time.

Since there is no synchronization mechanism, we have no way of knowing whether local files have been uploaded to the server. Therefore, to ensure that local files do not duplicate those on the server, we need to compare the document list data between local files and those on the server. If there are many files on the server side, it will indeed consume a lot of time.

This is also why we only compare file names. Apps usually don't know about file changes on the server, nor do they know whose file content is newer between the app and the server. Therefore, local files can only be backed up once, and there can't be any file duplication issues with the server (unless there are bugs).

zhwanng avatar Jul 15 '25 09:07 zhwanng

Since there is no synchronization mechanism, we have no way of knowing whether local files have been uploaded to the server. Therefore, to ensure that local files do not duplicate those on the server, we need to compare the document list data between local files and those on the server. If there are many files on the server side, it will indeed consume a lot of time.

Yes. And so IMO, once the file is considered existing on the server, a record in fileTransferDAO should be done so that the next time the file is excluded early and not searched in the dirent list on the server every time scan happens

foxel avatar Jul 15 '25 09:07 foxel

Regarding the size check. Comparing integers is considered faster than running a regexp. So a change like below will only make the checkAndInsert faster (note the size is compared first):

            Optional<DirentModel> firstOp = remoteList.stream()
                    .filter(f -> transferModel.file_size == f.size && pattern.matcher(f.name).matches())
                    .findFirst();

foxel avatar Jul 15 '25 09:07 foxel

  1. Regarding fileTransferDAO, in fact, when the first backup is successful, the data has already been added to the local database. The next time scan, it will first check whether this data exists in the database. But if there are new files, still need to check whether they exist on the server. If there are no new files, won't check the server.
  2. Regarding size check. Since can't fully guarantee that the local files and the server files are exactly the same, they may be modified in some cases. Therefore, once the file size is inconsistent, it may be re-uploaded, thus overwriting the server files.

zhwanng avatar Jul 15 '25 10:07 zhwanng

@zhwanng it's for the case when the upload is actually happening. As you can see from my logs, if the file is already on the server (uploaded before the database cleared), it never ends up in the fileTransferDAO. It's put into ALBUM_BACKUP_QUEUE and removed from it again and again on every scan. And every time it's tested for existence on the server.

Regarding the size check:

Since can't fully guarantee that the local files and the server files are exactly the same, they may be modified in some cases.

Yup that's exactly what we ideally want to check, whether the photo was already uploaded or not. We should be as sure that the file is already in the cloud as we can, but still not making duplicates. If you look back into the code before 3.x.x you'll see that the code finds a file with the same or suffixed name and wants to check whether it's actually the same file or not. But since there is no hash check available to the code it just checks the size, so that if the size differs, it's not the same file and the new file is uploaded. Although it's not replacing the file on the server, but is uploaded in addition to that.

Actually, the comment regarding the size check as a quick check in the absence of hash information is still in the code: https://github.com/haiwen/seadroid/blob/ca6b6599a619560faabf4ab55a9781afbbb127cb/app/src/main/java/com/seafile/seadroid2/framework/service/scan/AlbumScanHelper.java#L480-L483

foxel avatar Jul 15 '25 20:07 foxel

@foxel 1、Regarding the issue of repeated file uploads: You can update to the latest version and try again.

2、Regarding the comment: It is old and the content is not updated in time.

3、Regarding fileTransferDAO: For now, it may be relatively correct not to upload a file compared to having it overwritten by error. this feature will be updated in the future and we will need more conditions to check if it is the same file.

zhwanng avatar Jul 28 '25 03:07 zhwanng