Llama-3-8b-Instruct / HuggingFace: 401 Unauthorized when downloading non-weight files

[jhostetler]

Before submitting a bug, please make sure the issue hasn't been already addressed by searching through the FAQs and existing/past issues

Describe the bug

I am downloading the Llama-3-8b-Instruct model from HuggingFace using the huggingface_hub.snapshot_download() function. The weight files (*.safetensors) and some additional files (README.md and LICENSE) download successfully, but the other files (e.g., all of the *.json files) raise an HTTP 401 Unauthorized error.

I can view the problematic files on the HuggingFace website without issue.

Minimal reproducible example

from huggingface_hub import snapshot_download
snapshot_download(
        "meta-llama/meta-llama-3-8b-instruct",
        repo_type="model",
        revision="e5e23bbe8e749ef0efcf16cad411a7d23bd23298",
        ignore_patterns="*/*",  # Ignore subdirectories
        token=<my-huggingface-token>,
)

Output

This is a composite of logs from several runs. I manually added each problematic file to ignore_patterns until I had either had success or error on every file:

huggingface_hub: downloading meta-llama/meta-llama-3-8b-instruct
README.md: 100%|██████████| 39.1k/39.1k [00:00<00:00, 3.88MB/s]
LICENSE: 100%|██████████| 7.80k/7.80k [00:00<00:00, 20.3MB/s]
Fetching 12 files:  17%|█▋        | 2/12 [00:00<00:01,  6.24it/s]:00<?, ?B/s]
model-00004-of-00004.safetensors: 100%|██████████| 1.17G/1.17G [00:16<00:00, 71.4MB/s]
model-00003-of-00004.safetensors: 100%|██████████| 4.92G/4.92G [01:48<00:00, 45.2MB/s]
model-00001-of-00004.safetensors: 100%|██████████| 4.98G/4.98G [01:50<00:00, 45.2MB/s]
model-00002-of-00004.safetensors: 100%|██████████| 5.00G/5.00G [01:50<00:00, 45.2MB/s]
Process Process-1:04.safetensors: 100%|█████████▉| 4.98G/5.00G [01:50<00:00, 46.2MB/s]
Traceback (most recent call last):100%|██████████| 5.00G/5.00G [01:50<00:00, 47.1MB/s]

Cannot access gated repo for url https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/resolve/e5e23bbe8e749ef0efcf16cad411a7d23bd23298/.gitattributes.
Access to model meta-llama/Meta-Llama-3-8B-Instruct is restricted. You must be authenticated to access it.

Cannot access gated repo for url https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/resolve/e5e23bbe8e749ef0efcf16cad411a7d23bd23298/USE_POLICY.md.
Access to model meta-llama/Meta-Llama-3-8B-Instruct is restricted. You must be authenticated to access it.

Cannot access gated repo for url https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/resolve/e5e23bbe8e749ef0efcf16cad411a7d23bd23298/config.json.
Access to model meta-llama/Meta-Llama-3-8B-Instruct is restricted. You must be authenticated to access it.

Cannot access gated repo for url https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/resolve/e5e23bbe8e749ef0efcf16cad411a7d23bd23298/generation_config.json.
Access to model meta-llama/Meta-Llama-3-8B-Instruct is restricted. You must be authenticated to access it.

Cannot access gated repo for url https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/resolve/e5e23bbe8e749ef0efcf16cad411a7d23bd23298/model.safetensors.index.json.
Access to model meta-llama/Meta-Llama-3-8B-Instruct is restricted. You must be authenticated to access it.

Cannot access gated repo for url https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/resolve/e5e23bbe8e749ef0efcf16cad411a7d23bd23298/special_tokens_map.json.
Access to model meta-llama/Meta-Llama-3-8B-Instruct is restricted. You must be authenticated to access it.

Cannot access gated repo for url https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/resolve/e5e23bbe8e749ef0efcf16cad411a7d23bd23298/tokenizer.json.
Access to model meta-llama/Meta-Llama-3-8B-Instruct is restricted. You must be authenticated to access it.

Cannot access gated repo for url https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/resolve/e5e23bbe8e749ef0efcf16cad411a7d23bd23298/tokenizer_config.json.
Access to model meta-llama/Meta-Llama-3-8B-Instruct is restricted. You must be authenticated to access it.

Runtime Environment

• Model: meta-llama-3-8b-instruct
• Using via huggingface?: yes
• OS: Linux/Ubuntu
• GPU VRAM: N/A
• Number of GPUs: N/A
• GPU Make: N/A

Additional context Add any other context about the problem or environment here.

[fbnav]

cc @osanseviero and @samuelselvan This looks like an issue with model access on HF. Any suggestions on how they can download all the files/re-submit their request?