Failed hash check logic passes as a completed download, and a typo
I have been downloading several models but occasionally notice that there are model folders much smaller than they should be. My internet connection is a crappy long distance microwave connection that is error prone. It turns out that, when there is a hash fail, somehow, the program then exits with a download success, like this:
Hash failed for LFS file: Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors
Download of alpindale/goliath-120b completed successfully
I am using the command:
bash <(curl -sSL https://g.bodaay.io/hfd) -m alpindale/goliath-120b
Here is the full output from this particular run, which was actually a re-run because the download must have quietly failed before too:
File already exists. Skipping download.
Model: alpindale/goliath-120b
Branch: main
Storage: Storage
NumberOfConcurrentConnections: 5
Append Filter Names to Folder: false
Skip SHA256 Check: false
Token:
Getting File Download Files List Tree from: https://huggingface.co/api/models/alpindale/goliath-120b/tree/main/
Checking Existsing file: Storage/alpindale_goliath-120b/.gitattributes
file size matched for non LFS file: Storage/alpindale_goliath-120b/.gitattributes
Checking Existsing file: Storage/alpindale_goliath-120b/README.md
file size matched for non LFS file: Storage/alpindale_goliath-120b/README.md
Checking Existsing file: Storage/alpindale_goliath-120b/config.json
file size matched for non LFS file: Storage/alpindale_goliath-120b/config.json
Skipping: Storage/alpindale_goliath-120b/.gitattributes
Skipping: Storage/alpindale_goliath-120b/README.md
Skipping: Storage/alpindale_goliath-120b/config.json
Downloading Storage/alpindale_goliath-120b/model-00001-of-00024.safetensors Speed: 4.70 MB/sec, 100.00%
Merging Storage/alpindale_goliath-120b/model-00001-of-00024.safetensors Chunks
Checking SHA256 Hash for LFS file: Storage/alpindale_goliath-120b/model-00001-of-00024.safetensors
Hash Matched for LFS file: Storage/alpindale_goliath-120b/model-00001-of-00024.safetensors
Downloading Storage/alpindale_goliath-120b/model-00002-of-00024.safetensors Speed: 4.84 MB/sec, 100.00%
Downloading Storage/alpindale_goliath-120b/model-00002-of-00024.safetensors Speed: 4.84 MB/sec, 100.00%
Checking SHA256 Hash for LFS file: Storage/alpindale_goliath-120b/model-00002-of-00024.safetensors
Hash Matched for LFS file: Storage/alpindale_goliath-120b/model-00002-of-00024.safetensors
Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 4.95 MB/sec, 59.40% Error downloading chunk 2: stream error: stream ID 1; INTERNAL_ERROR; received from peer
warning: attempt 1 / 3 failed, error: Error downloading chunk 2: stream error: stream ID 1; INTERNAL_ERROR; received from peer
Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 4.95 MB/sec, 59.62%
Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 4.95 MB/sec, 59.78% ee/main/
Checking Existsing file: Storage/alpindale_goliath-120b/.gitattributes
file size matched for non LFS file: Storage/alpindale_goliath-120b/.gitattributes
Checking Existsing file: Storage/alpindale_goliath-120b/README.md
file size matched for non LFS file: Storage/alpindale_goliath-120b/README.md
Checking Existsing file: Storage/alpindale_goliath-120b/config.json
file size matched for non LFS file: Storage/alpindale_goliath-120b/config.json
Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 4.93 MB/sec, 61.79%
Hash Matched for LFS file: Storage/alpindale_goliath-120b/model-00001-of-00024.safetensors
Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 4.92 MB/sec, 63.59%
Hash Matched for LFS file: Storage/alpindale_goliath-120b/model-00002-of-00024.safetensors
Skipping: Storage/alpindale_goliath-120b/.gitattributes
Skipping: Storage/alpindale_goliath-120b/README.md
Skipping: Storage/alpindale_goliath-120b/config.json
Skipping: Storage/alpindale_goliath-120b/model-00001-of-00024.safetensors
Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 4.91 MB/sec, 63.60%
Found existing incomplete download for the file: model-00003-of-00024.safetensors
Forcing Number of connections to: 5
Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 8.24 MB/sec, 100.00% 0%
Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 3.54 MB/sec, 89.38%
Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 3.54 MB/sec, 89.41%
Hash failed for LFS file: Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors
Download of alpindale/goliath-120b completed successfully
And secondarily, because I pasted the code above and it had spellcheck for the output, and I noticed a typo. I'll put it here, but would you rather me make a new issue just for this?
Checking Existsing file: Storage/alpindale_goliath-120b/.gitattributes
"Existsing" is spelled wrong, it should be, "Existing"