HuggingFaceModelDownloader icon indicating copy to clipboard operation
HuggingFaceModelDownloader copied to clipboard

Failed hash check logic passes as a completed download, and a typo

Open InvalidAdmin opened this issue 2 years ago • 0 comments

I have been downloading several models but occasionally notice that there are model folders much smaller than they should be. My internet connection is a crappy long distance microwave connection that is error prone. It turns out that, when there is a hash fail, somehow, the program then exits with a download success, like this:

Hash failed for LFS file: Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors
Download of alpindale/goliath-120b completed successfully

I am using the command:

bash <(curl -sSL https://g.bodaay.io/hfd) -m alpindale/goliath-120b

Here is the full output from this particular run, which was actually a re-run because the download must have quietly failed before too:

File already exists. Skipping download.
Model: alpindale/goliath-120b
Branch: main
Storage: Storage
NumberOfConcurrentConnections: 5
Append Filter Names to Folder: false
Skip SHA256 Check: false
Token: 

Getting File Download Files List Tree from: https://huggingface.co/api/models/alpindale/goliath-120b/tree/main/
Checking Existsing file: Storage/alpindale_goliath-120b/.gitattributes
file size matched for non LFS file: Storage/alpindale_goliath-120b/.gitattributes
Checking Existsing file: Storage/alpindale_goliath-120b/README.md
file size matched for non LFS file: Storage/alpindale_goliath-120b/README.md
Checking Existsing file: Storage/alpindale_goliath-120b/config.json
file size matched for non LFS file: Storage/alpindale_goliath-120b/config.json
Skipping: Storage/alpindale_goliath-120b/.gitattributes
Skipping: Storage/alpindale_goliath-120b/README.md
Skipping: Storage/alpindale_goliath-120b/config.json

Downloading Storage/alpindale_goliath-120b/model-00001-of-00024.safetensors Speed: 4.70 MB/sec, 100.00% 
Merging Storage/alpindale_goliath-120b/model-00001-of-00024.safetensors Chunks
Checking SHA256 Hash for LFS file: Storage/alpindale_goliath-120b/model-00001-of-00024.safetensors
Hash Matched for LFS file: Storage/alpindale_goliath-120b/model-00001-of-00024.safetensors

Downloading Storage/alpindale_goliath-120b/model-00002-of-00024.safetensors Speed: 4.84 MB/sec, 100.00% 
Downloading Storage/alpindale_goliath-120b/model-00002-of-00024.safetensors Speed: 4.84 MB/sec, 100.00% 
Checking SHA256 Hash for LFS file: Storage/alpindale_goliath-120b/model-00002-of-00024.safetensors
Hash Matched for LFS file: Storage/alpindale_goliath-120b/model-00002-of-00024.safetensors

Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 4.95 MB/sec, 59.40% Error downloading chunk 2: stream error: stream ID 1; INTERNAL_ERROR; received from peer
warning: attempt 1 / 3 failed, error: Error downloading chunk 2: stream error: stream ID 1; INTERNAL_ERROR; received from peer
Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 4.95 MB/sec, 59.62% 
Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 4.95 MB/sec, 59.78% ee/main/
Checking Existsing file: Storage/alpindale_goliath-120b/.gitattributes
file size matched for non LFS file: Storage/alpindale_goliath-120b/.gitattributes
Checking Existsing file: Storage/alpindale_goliath-120b/README.md
file size matched for non LFS file: Storage/alpindale_goliath-120b/README.md
Checking Existsing file: Storage/alpindale_goliath-120b/config.json
file size matched for non LFS file: Storage/alpindale_goliath-120b/config.json
Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 4.93 MB/sec, 61.79% 
Hash Matched for LFS file: Storage/alpindale_goliath-120b/model-00001-of-00024.safetensors
Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 4.92 MB/sec, 63.59% 
Hash Matched for LFS file: Storage/alpindale_goliath-120b/model-00002-of-00024.safetensors
Skipping: Storage/alpindale_goliath-120b/.gitattributes
Skipping: Storage/alpindale_goliath-120b/README.md
Skipping: Storage/alpindale_goliath-120b/config.json
Skipping: Storage/alpindale_goliath-120b/model-00001-of-00024.safetensors
Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 4.91 MB/sec, 63.60% 
Found existing incomplete download for the file: model-00003-of-00024.safetensors
Forcing Number of connections to: 5



Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 8.24 MB/sec, 100.00%    0% 
Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 3.54 MB/sec, 89.38% 
Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 3.54 MB/sec, 89.41% 
Hash failed for LFS file: Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors
Download of alpindale/goliath-120b completed successfully

And secondarily, because I pasted the code above and it had spellcheck for the output, and I noticed a typo. I'll put it here, but would you rather me make a new issue just for this?

Checking Existsing file: Storage/alpindale_goliath-120b/.gitattributes

"Existsing" is spelled wrong, it should be, "Existing"

InvalidAdmin avatar Nov 18 '23 13:11 InvalidAdmin