getparty

getparty copied to clipboard

Hey Vladimir, before I look closer at the issue I just wanted to check with you if there is any recently known issue/bug which you are aware of that could cause the following behavior?

1. SHA256 hash of 6GB file on the download server is calculated.
2. The 6GB file is downloaded by getparty using 10 part multi-part download
3. File is download successfully, file is reassembled, and file size is verified to be correct (byte count)
4. SHA256 hash of downloaded file is calculated. Most of the time the SHA256 hash matches the hash calculated in step 1....however, rarely the hash does not match.
5. If I see this issue reproduce and change the download parts from 10 to a lower number (e.g. 5) then the file is downloaded and hash is correct (matches hash calculated in step 1).

I am curious what could cause the SHA256 hash of the downloaded file to be incorrect sometimes (even though the downloaded file size is correct and the file is usable + not showing any obvious signs of corruption). Is there anything in the retry logic or reassembly that could cause this to your knowledge?

I am seeing it intermittently, but haven't been able to get a consistent repro for debugging purposes....so I thought I would check here first to see if you've seen anything like this or fixed any issues recently (since version v1.11.1 which is old now, but the version I am on currently).

Thanks!

Hi Nick! I personally use getparty on very unstable connections and never faced such an issue, though I always use a HEAD version.

Is there anything in the retry logic or reassembly that could cause this to your knowledge?

There are no significant changes in retry or reassembly logic I did since very early version. The only change I did in v1.13, which fixes retry logic for dumb servers without 206 Partial Content response support.

If there is a bug in retry logic it will affect retry even in single part mode . So you can attempt to download with single part in order to see if problem is still there. I encourage you to test with head version anyway.

Sorry, I meant to circle back here. This ended up being external (unusual behavior from go crypto pkg where the same exact downloaded file could be passed multiple times to their SHA256 calculation function and each time would be computed as a different SHA256 hash...). This issue is not related to getparty and can be closed.

Hi @vbauerster - it seems that we are still seeing this behavior, although much less frequently.

We have encountered some Windows environments where after downloading (even using only 1 part download) the final file is the correct size in bytes but does not pass a SHA256 hash integrity check (i.e. the file is not the same as on the http server which served the file).

We then noticed this commit which you made a couple of weeks back for a EOF issue:

fee4cf71acdcd84a03469ba89424e113b1846e5b

Is this EOF issue a functional issue which could describe the type of behavior we are seeing? (i.e. a download which seems to be 'successful' and no errors are returned by getparty, but the file on disk is not 100% correct).

Thanks

Is this EOF issue a functional issue which could describe the type of behavior we are seeing? (i.e. a download which seems to be 'successful' and no errors are returned by getparty, but the file on disk is not 100% correct).

No, in that case you would get different size i.e. incomplete download. Make sure there is nothing in the middle like proxy or something. Try reproduce on linux machine.

Thanks @vbauerster

There is definitely a hardware proxy in the middle (majority of our environments are behind proxies). The proxy has inspection disabled and caching was also disabled for these downloads to be tested. Things all appear to look good from the proxy side (at least no differences have been observed in the proxy/dns logs between the 'good' case where a downloaded file is not corrupted and the 'bad' case where the final downloaded file is corrupted and not passing SHA256 hash verification) .

We also confirmed that downloading via other methods (e.g. using curl) is not reproducing the issue, even when the connection goes through the same proxy etc.

I realize this will be tricky to track down. Is there any more verbosity/logging we can enable in getparty layer or lower http client layers to potentially find a difference in the good vs. bad downloads?

Is 'bad' case happens with uninterrupted download or it only happens with retries or worse both? You can enable debug with --debug switch just make sure to redirect standard error output to a file like 2> debut.log.

In the bad case, we are able to reproduce the issue with part count set to 1, and there is no sign of any reconnect occurring in the getparty output (single connection appears to complete without retry and confirms that it has downloaded the correct number of bytes).

Can you please build head master branch and test one more time? I used slightly different io api which might help (0c7dcae91a63430bac011813be6e5b612a88e7df). To build without errors you'll need master version of mpb lib. Clone mpb somewhere then append a line to go.mod file:

replace github.com/vbauerster/mpb/v7 => /path/to/latest/mpb

substitute with real path to just cloned mpb.

If it doesn't help, then my guess it's specific to windows go runtime, unless you can reproduce same behavior on linux machine.