huggingface_hub icon indicating copy to clipboard operation
huggingface_hub copied to clipboard
verified publisher icon huggingface

huggingface cli is not honoring http proxy environment variables on version >=1.0.0

Open smothiki opened this issue 4 weeks ago • 8 comments

Describe the bug

hf download command fails while downloading models in a proxy environment Example :

HTTPS_PROXY="http://fooservuce:3128/" HTTP_PROXY="http://rfooservce:3128/" hf download TinyLlama/TinyLlama-1.1B-Chat-v1.0 Always fails and the proxy access log has not log about accessing huggingface and eventually the command fails

Reproduction

Instructions about setting proxy locally using Docker https://gist.github.com/smothiki/0a0176bffaa63954ded4be9aeb135611 HTTPS_PROXY="http://fooservuce:3128" HTTP_PROXY="http://rfooservce:3128" hf download TinyLlama/TinyLlama-1.1B-Chat-v1.0

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
    yield
  File "/usr/lib/python3.10/site-packages/httpx/_transports/default.py", line 250, in handle_request
    resp = self._pool.handle_request(req)
  File "/usr/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
    raise exc from None
  File "/usr/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
    response = connection.handle_request(
  File "/usr/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 101, in handle_request
    raise exc
  File "/usr/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 78, in handle_request
    stream = self._connect(request)
  File "/usr/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 156, in _connect
    stream = stream.start_tls(**kwargs)
  File "/usr/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 154, in start_tls
    with map_exceptions(exc_map):
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: [Errno 104] Connection reset by peer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 240, in snapshot_download
    repo_info = api.repo_info(repo_id=repo_id, repo_type=repo_type, revision=revision)
  File "/usr/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 89, in _inner_fn
    return fn(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 2832, in repo_info
    return method(
  File "/usr/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 89, in _inner_fn
    return fn(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 2625, in model_info
    r = get_session().get(path, headers=headers, timeout=timeout, params=params)
  File "/usr/lib/python3.10/site-packages/httpx/_client.py", line 1053, in get
    return self.request(
  File "/usr/lib/python3.10/site-packages/httpx/_client.py", line 825, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/lib/python3.10/site-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
  File "/usr/lib/python3.10/site-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/usr/lib/python3.10/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/usr/lib/python3.10/site-packages/httpx/_client.py", line 1014, in _send_single_request
    response = transport.handle_request(request)
  File "/usr/lib/python3.10/site-packages/httpx/_transports/default.py", line 249, in handle_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/lib/python3.10/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [Errno 104] Connection reset by peer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/hf", line 7, in <module>
    sys.exit(main())
  File "/usr/lib/python3.10/site-packages/huggingface_hub/cli/hf.py", line 58, in main
    app()
  File "/usr/lib/python3.10/site-packages/typer/main.py", line 338, in __call__
    raise e
  File "/usr/lib/python3.10/site-packages/typer/main.py", line 321, in __call__
    return get_command(self)(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1442, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/typer/core.py", line 728, in main
    return _main(
  File "/usr/lib/python3.10/site-packages/typer/core.py", line 197, in _main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1830, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1226, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 794, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/typer/main.py", line 703, in wrapper
    return callback(**use_params)
  File "/usr/lib/python3.10/site-packages/huggingface_hub/cli/download.py", line 188, in download
    _print_result(run_download())
  File "/usr/lib/python3.10/site-packages/huggingface_hub/cli/download.py", line 149, in run_download
    return snapshot_download(
  File "/usr/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 89, in _inner_fn
    return fn(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 324, in snapshot_download
    raise LocalEntryNotFoundError(
huggingface_hub.errors.LocalEntryNotFoundError: An error happened while trying to locate the files on the Hub and we cannot find the appropriate snapshot folder for the specified revision on the local disk. Please check your internet connection and try again.

Logs

Request cd192256-84e5-45be-975e-2dee663e3f10: GET https://huggingface.co/api/models/TinyLlama/TinyLlama-1.1B-Chat-v1.0/revision/main (authenticated: False)
Send: curl -X GET -H 'accept: */*' -H 'accept-encoding: gzip, deflate' -H 'connection: keep-alive' -H 'host: huggingface.co' -H 'user-agent: huggingface-cli/None; hf_hub/1.1.7; python/3.10.19' -H 'x-amzn-trace-id: cd192256-84e5-45be-975e-2dee663e3f10' -d https://huggingface.co/api/models/TinyLlama/TinyLlama-1.1B-Chat-v1.0/revision/main
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
    yield
  File "/usr/lib/python3.10/site-packages/httpx/_transports/default.py", line 250, in handle_request
    resp = self._pool.handle_request(req)
  File "/usr/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
    raise exc from None
  File "/usr/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
    response = connection.handle_request(
  File "/usr/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 101, in handle_request
    raise exc
  File "/usr/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 78, in handle_request
    stream = self._connect(request)
  File "/usr/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 156, in _connect
    stream = stream.start_tls(**kwargs)
  File "/usr/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 154, in start_tls
    with map_exceptions(exc_map):
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: [Errno 104] Connection reset by peer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 240, in snapshot_download
    repo_info = api.repo_info(repo_id=repo_id, repo_type=repo_type, revision=revision)
  File "/usr/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 89, in _inner_fn
    return fn(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 2758, in repo_info
    return method(
  File "/usr/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 89, in _inner_fn
    return fn(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 2551, in model_info
    r = get_session().get(path, headers=headers, timeout=timeout, params=params)
  File "/usr/lib/python3.10/site-packages/httpx/_client.py", line 1053, in get
    return self.request(
  File "/usr/lib/python3.10/site-packages/httpx/_client.py", line 825, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/lib/python3.10/site-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
  File "/usr/lib/python3.10/site-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/usr/lib/python3.10/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/usr/lib/python3.10/site-packages/httpx/_client.py", line 1014, in _send_single_request
    response = transport.handle_request(request)
  File "/usr/lib/python3.10/site-packages/httpx/_transports/default.py", line 249, in handle_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/lib/python3.10/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [Errno 104] Connection reset by peer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/hf", line 7, in <module>
    sys.exit(main())
  File "/usr/lib/python3.10/site-packages/huggingface_hub/cli/hf.py", line 58, in main
    app()
  File "/usr/lib/python3.10/site-packages/typer/main.py", line 338, in __call__
    raise e
  File "/usr/lib/python3.10/site-packages/typer/main.py", line 321, in __call__
    return get_command(self)(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1442, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/typer/core.py", line 728, in main
    return _main(
  File "/usr/lib/python3.10/site-packages/typer/core.py", line 197, in _main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1830, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1226, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 794, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/typer/main.py", line 703, in wrapper
    return callback(**use_params)
  File "/usr/lib/python3.10/site-packages/huggingface_hub/cli/download.py", line 188, in download
    _print_result(run_download())
  File "/usr/lib/python3.10/site-packages/huggingface_hub/cli/download.py", line 149, in run_download
    return snapshot_download(
  File "/usr/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 89, in _inner_fn
    return fn(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 324, in snapshot_download
    raise LocalEntryNotFoundError(
huggingface_hub.errors.LocalEntryNotFoundError: An error happened while trying to locate the files on the Hub and we cannot find the appropriate snapshot folder for the specified revision on the local disk. Please check your internet connection and try again.

System info

hf env

Copy-and-paste the text below in your GitHub issue.

- huggingface_hub version: 1.1.7
- Platform: Linux-5.14.0-503.40.1.el9_5.x86_64-x86_64-with-glibc2.42
- Python version: 3.10.19
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Running in Google Colab Enterprise ?: No
- Token path ?: /hf/token
- Has saved token ?: False
- Configured git credential helpers: 
- Installation method: unknown
- httpx: 0.28.1
- hf_xet: 1.2.0
- gradio: N/A
- tensorboard: N/A
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: /hf/hub
- HF_ASSETS_CACHE: /hf/assets
- HF_TOKEN_PATH: /hf/token
- HF_STORED_TOKENS_PATH: /hf/stored_tokens
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_DISABLE_XET: False
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10
- HF_XET_HIGH_PERFORMANCE: False

smothiki avatar Dec 09 '25 01:12 smothiki

I was able to download the models successfully with version 0.36.0

smothiki avatar Dec 09 '25 01:12 smothiki

Hi @smothiki, I suspect something is wrong with the proxy setup. From your gist, I see that you are checking the proxy using

curl -x http://localhost:3128/ http://httpbin.org/ip

Can you try with

curl -x http://localhost:3128/ https://huggingface.co

instead? (note the https).

There might be something ongoing with the https connect tunneling. The curl command should help investigate if that's the case or not.

Wauplin avatar Dec 09 '25 11:12 Wauplin

I think the failure is simply caused by typos in proxy host-names (fooservuce / rfooservce).

If they run it like this , they should be able to see it working !

export HTTPS_PROXY="http://localhost:3128"
export HTTP_PROXY="http://localhost:3128"
hf download TinyLlama/TinyLlama-1.1B-Chat-v1.0

instead of :

HTTPS_PROXY="http://fooservuce:3128" HTTP_PROXY="http://rfooservce:3128" hf download TinyLlama/TinyLlama-1.1B-Chat-v1.0

As the error-log shows

httpcore.ConnectError: [Errno 104] Connection reset by peer

which is thrown before any HTTP request is sent.

Error 104 means the TCP connection itself was rejected and the empty proxy access log confirms no packet ever reached the proxy.

That happens when the resolver can’t find the host names you put in the URL (fooservuce / rfooservce → NXDOMAIN).

akshatvishu avatar Dec 09 '25 15:12 akshatvishu

My bad fooservuce and rfooservce are an example and doesn't;t reflect the actual proxy. Can you check the logs of squid proxy setup ? Does it show huggingface.co is going through the proxy you can do tail -f /var/log/squid/access.log or car cat /var/log/squid/access.log should show something like

1765245915.651      0 10.17.144.34 TCP_DENIED/403 3857 CONNECT cas-bridge.xethub.hf.co:443 - - HIER_NONE/- text/html
1765245916.490      0 10.17.144.34 TCP_DENIED/403 3857 CONNECT cas-bridge.xethub.hf.co:443 - - HIER_NONE/- text/html

smothiki avatar Dec 09 '25 23:12 smothiki

If your machine has direct internet access, this test may yield misleading results—even if HTTPS proxy environment variables are set. The Hugging Face CLI (hf) might bypass these variables and download directly via the open connection. The only definitive way to verify that traffic is routing through the proxy is to inspect the proxy server's access logs.

smothiki avatar Dec 09 '25 23:12 smothiki

Have you tried this? https://github.com/huggingface/huggingface_hub/issues/3603#issuecomment-3631866235

Also, the proxy should also proxy calls to cas-bridge.xethub.hf.co indeed (backend storage)

Wauplin avatar Dec 10 '25 09:12 Wauplin

This can also be due to httpx prioritizes the NO_PROXY environment variable over HTTPS_PROXY.

  • Ensure NO_PROXY is unset or does not contain huggingface.co.
  • HTTPS_PROXY uses the http:// scheme (e.g., http://proxy:3128), as httpx can fail silently or go direct if the scheme is missing or https://.

Can try running this directly to see if this the case :

# This forces NO_PROXY to be empty and ensures the correct http:// scheme
env NO_PROXY="" HTTPS_PROXY="http://<ACTUAL_PROXY_IP>:3128" hf download TinyLlama/TinyLlama-1.1B-Chat-v1.0

Also, if you're still getting TCP_DENIED/403 then check that your Squid configuration isn't blocking blocks the LFS/Xet backend (xethub.hf.co), sometimes it only allow huggingface.co

# Old: acl allowed_http_sites dstdom_regex -i (^|\.)huggingface\.co
# New: Allow both huggingface.co AND hf.co (needed for xethub/cas-bridge)
acl allowed_http_sites dstdom_regex -i (^|\.)(huggingface|hf)\.co

akshatvishu avatar Dec 10 '25 11:12 akshatvishu

Hey! 👋 I looked into this and wanted to share some findings.

The switch from requests tohttpxin v1.0.0 likely changed how proxy environment variables are handled. A few things to check:

1. Environment variable case - Try setting both lowercase versions:

export http_proxy="http://fooservice:3128"
export https_proxy="http://fooservice:3128"
export HTTP_PROXY="http://fooservice:3128"
export HTTPS_PROXY="http://fooservice:3128"

2. Verifyhttpx sees the proxy - Can you run this quick test?

import httpx
import os
print("HTTP_PROXY:", os.environ.get("HTTP_PROXY"))
print("HTTPS_PROXY:", os.environ.get("HTTPS_PROXY"))

Test if httpx picks up the proxy

client = httpx.Client(trust_env=True)
print("Client proxies:", client._proxies)

3. Workaround - Explicitly configure the proxy:

from huggingface_hub.utils import set_client_factory
import httpx
import os
def proxy_aware_client_factory():
    proxy = os.environ.get("HTTPS_PROXY") or os.environ.get("HTTP_PROXY")
    return httpx.Client(
        proxy=proxy,
        follow_redirects=True,
        timeout=httpx.Timeout(30.0, write=60.0),
    )
set_client_factory(proxy_aware_client_factory)
If the proxy isn't being picked up even with trust_env=True, this might be a regression that needs an explicit fix in 
default_client_factory()

Tanishq1030 avatar Dec 12 '25 15:12 Tanishq1030

unsetting no proxy actually worked. But isn;t no proxy should be considered only if huggingface domains are specified?

smothiki avatar Dec 18 '25 16:12 smothiki

Glad you got it working @smothiki. To be fair we rely on httpx to read environment variables and we don't want to change the default behavior. I'm not surprise if NO_PROXY takes precedence over the rest in all cases.

Wauplin avatar Dec 18 '25 17:12 Wauplin