gosync icon indicating copy to clipboard operation
gosync copied to clipboard

Published 20 hours ago • verified publisher icon hschauhan

Files over 4GB are not correctly synced

Open Fabian42 opened this issue 4 years ago • 20 comments

I have a folder with two copies of a 4.4GB (4669979776B) file. One of those copies got downloaded by GoSync as a 1000MB (1048576010B=1024²×1000B+10B) file, one as a 2.5GB (2726297626B=???) file. The copy in the cloud was not damaged.

Fabian42 avatar May 20 '20 22:05 Fabian42

This is because the download sometimes fails. The idea is to retry. Alain is working on adding exponential backoff algorithm for retries. Should be fixed soon.

hschauhan avatar May 21 '20 05:05 hschauhan

The download failed after exactly 1000MB+10B? That seems to be too exact of a number to be a concidence.

Fabian42 avatar May 21 '20 08:05 Fabian42

Hello,

a fix was added to GoSync to handle large file download.

Can you update GoSync with the current version and share your comments ?

Regards

arentoine avatar May 26 '20 14:05 arentoine

Not worth its own report, but: "Debuging"→"Debugging"

Fabian42 avatar May 26 '20 18:05 Fabian42

Hello,

does it work ? :-)

If not, do you have log ?

Thanks

arentoine avatar May 26 '20 18:05 arentoine

Not worth its own report, but: "Debuging"→"Debugging"

Ouch!

hschauhan avatar May 26 '20 18:05 hschauhan

I was testing it and doing other stuff at the same time, so I didn't notice that I had to click "resume sync". Now I did and it instantly ends again:

2020-05-26 20:44:05,701 - GoSync - DEBUG - SyncThread - run - Trying to acquire lock.
2020-05-26 20:44:05,701 - GoSync - DEBUG - SyncThread - run - Lock acquired.
2020-05-26 20:44:05,701 - GoSync - DEBUG - SyncThread - run - Validating sync settings
2020-05-26 20:44:05,702 - GoSync - INFO - SyncThread - run - validated
2020-05-26 20:44:05,702 - GoSync - INFO - SyncThread - run - Staring the sync now
2020-05-26 20:44:05,702 - GoSync - DEBUG - ###############################################
2020-05-26 20:44:05,702 - GoSync - DEBUG - Start - Syncing remote directory
2020-05-26 20:44:05,702 - GoSync - DEBUG - ###############################################
2020-05-26 20:44:05,702 - GoSync - DEBUG - ###############################################
2020-05-26 20:44:05,702 - GoSync - DEBUG - End - Syncing remote directory
2020-05-26 20:44:05,702 - GoSync - DEBUG - ###############################################

2020-05-26 20:44:05,702 - GoSync - DEBUG - ###############################################
2020-05-26 20:44:05,702 - GoSync - DEBUG - Start - Syncing local directory
2020-05-26 20:44:05,702 - GoSync - DEBUG - ###############################################
2020-05-26 20:44:05,702 - GoSync - DEBUG - ### SyncLocalDirectory: - Sync Started
2020-05-26 20:44:05,702 - GoSync - ERROR - SyncThread - run - Unknown exception
2020-05-26 20:44:05,702 - GoSync - INFO - SyncThread - run - Sync done

10/10 error message right there.

Fabian42 avatar May 26 '20 18:05 Fabian42

Hello,

issue is in the upload section. That definitely need some fixing.

Did you test large file download ? From GoogleDrive to Local ?

Thanks for you comments.

arentoine avatar May 26 '20 18:05 arentoine

I just starting syncing. Since large files didn't work before, that would have been the only thing to sync for it. But it didn't even start syncing properly.

On Tue, 26 May 2020 at 20:50, arentoine [email protected] wrote:

Hello,

issue is in the upload section. That definitely need some fixing.

Did you test large file download ? From GoogleDrive to Local ?

Thanks for you comments.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/hschauhan/gosync/issues/38#issuecomment-634209869, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACP66YXVSBD5E6AFDWTDIY3RTQFOFANCNFSM4NGLNEEQ .

Fabian42 avatar May 26 '20 18:05 Fabian42

Do you know if you have a "half downloaded file" sitting in your local drive ?

arentoine avatar May 26 '20 18:05 arentoine

I do, the big ones.

On Tue, 26 May 2020 at 20:57, arentoine [email protected] wrote:

Do you know if you have a "half downloaded file" sitting in your local drive ?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/hschauhan/gosync/issues/38#issuecomment-634213375, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACP66YWGCIUKYH7YM42BQUTRTQGHVANCNFSM4NGLNEEQ .

Fabian42 avatar May 26 '20 22:05 Fabian42

Can you move the incomplete files in a safe place out of the sync directory. Then if you could update your tree with the most uptodate version. Then tell us if this will help you ?

Regards

arentoine avatar May 27 '20 01:05 arentoine

It seems to have actually uploaded the small, broken version of one of the big files again, duplicating it on Drive. It did not do that for any other one.

I did what you said, moved the broken files somewhere else and tried syncing again. This is what happened last:

2020-05-27 15:22:56,384 - GoSync - DEBUG - Checking: file.mp4

2020-05-27 15:22:56,385 - GoSync - DEBUG - SyncRemoteDirectory: Checking file (file.mp4)
2020-05-27 15:22:56,392 - GoSync - DEBUG - DownloadFileByObject: Download Started - File (/home/fabian/Google Drive/[email protected]/path/to/file.mp4)
2020-05-27 15:22:56,392 - GoSync - DEBUG - DownloadFileByObject: Download Started - File size (1301474114)
2020-05-27 15:22:56,517 - GoSync - DEBUG - Observer: /home/fabian/Google Drive/[email protected]/path/to/file.mp4 created

2020-05-27 15:22:56,522 - GoSync - DEBUG - UploadObservedFile: File /home/fabian/Google Drive/[email protected]/path/to/file.mp4 is created but sync is running. Possibly created by sync. Skipping.
2020-05-27 15:26:24,527 - GoSync - ERROR - SyncRemoteDirectory: Failed to sync directory (file.mp4)
2020-05-27 15:26:24,528 - GoSync - ERROR - SyncRemoteDirectory: Failed to sync directory (to)
2020-05-27 15:26:24,528 - GoSync - ERROR - SyncRemoteDirectory: Failed to sync directory (path)
2020-05-27 15:26:24,529 - GoSync - ERROR - SyncThread - run - Unknown exception
2020-05-27 15:26:24,529 - GoSync - INFO - SyncThread - run - Sync done
2020-05-27 15:26:54,570 - GoSync - DEBUG - SyncThread - run - Waiting for Sync to be enabled
2020-05-27 15:27:14,716 - GoSync - INFO - SyncThread - run - Internet is down. Clearing running.

Now it's stuck at "Network is down" and nothing happens anymore.

Fabian42 avatar May 27 '20 13:05 Fabian42

I will prepare a special python module to gather more information about the issue.

I'll get back to you in a couple of days ...

Greetings

arentoine avatar May 28 '20 01:05 arentoine

Just a quick reminder that I only test this because it's fun and helps you, I don't really need the program. If nobody else has my issues, you don't have to spend all that time.

Fabian42 avatar May 28 '20 08:05 Fabian42

Hi Fabian,

Your testing is quite important to this project. It has uncovered lot of problems and made the project quite robust as compared to previous versions. Please keep testing if you can.

Regards Himanshu

On Thu, 28 May 2020 at 1:43 PM, Fabian Röling [email protected] wrote:

Just a quick reminder that I only test this because it's fun and helps you, I don't really need the program. If nobody else has my issues, you don't have to spend all that time.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/hschauhan/gosync/issues/38#issuecomment-635189552, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABXIY53IDXSB3HKYKQ4IKLRTYMJBANCNFSM4NGLNEEQ .

hschauhan avatar May 28 '20 11:05 hschauhan

Exactly !

Keep having fun testing as much as we do resolving bugs.

And may be, someone will benefit from that !

Thanks again!

arentoine avatar May 28 '20 12:05 arentoine

I found the problem, a silly one. Will commit tomorrow.

On Wed, 27 May 2020 at 12:20 AM, arentoine [email protected] wrote:

Hello,

issue is in the upload section. That definitely need some fixing.

Did you test large file download ? From GoogleDrive to Local ?

Thanks for you comments.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/hschauhan/gosync/issues/38#issuecomment-634209869, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABXIY4AIZBTE4ZNZT7SNN3RTQFOFANCNFSM4NGLNEEQ .

hschauhan avatar May 28 '20 20:05 hschauhan

I have committed the changes and local sync is working fine.

hschauhan avatar May 29 '20 04:05 hschauhan 

2020-05-30 12:12:36,500 - GoSync - DEBUG - Checking: file.iso

2020-05-30 12:12:36,501 - GoSync - DEBUG - SyncRemoteDirectory: Checking file (file.iso)
2020-05-30 12:12:36,506 - GoSync - DEBUG - DownloadFileByObject: Download Started - File (/home/fabian/Google Drive/[email protected]/path/to/file.iso)
2020-05-30 12:12:36,507 - GoSync - DEBUG - DownloadFileByObject: Download Started - File size (4699979776)
2020-05-30 12:12:36,576 - GoSync - DEBUG - Observer: /home/fabian/Google Drive/[email protected]/path/to/file.iso created

2020-05-30 12:12:36,586 - GoSync - DEBUG - UploadObservedFile: File /home/fabian/Google Drive/[email protected]/path/to/file.iso is created but sync is running. Possibly created by sync. Skipping.
2020-05-30 12:14:57,565 - GoSync - ERROR - SyncRemoteDirectory: Failed to sync directory (file.iso)
2020-05-30 12:14:57,565 - GoSync - ERROR - SyncRemoteDirectory: Failed to sync directory (to)
2020-05-30 12:14:57,566 - GoSync - ERROR - SyncRemoteDirectory: Failed to sync directory (path)
2020-05-30 12:14:57,566 - GoSync - ERROR - SyncThread - run - Unknown exception
2020-05-30 12:14:57,566 - GoSync - INFO - SyncThread - run - Sync done
2020-05-30 12:15:27,600 - GoSync - DEBUG - SyncThread - run - Waiting for Sync to be enabled
2020-05-30 12:15:47,744 - GoSync - INFO - SyncThread - run - Internet is down. Clearing running.
2020-05-30 14:41:00,783 - GoSync - INFO - calculateUsage - GoSync is shutting down!

I just tried the "exit" menu option for the first time and after confirming, both the confirmation dialogue and the program froze and needed to be forcibly shut down.

Fabian42 avatar May 30 '20 12:05 Fabian42