Failed to download model with docker behind proxy

[daixiang0]

System Info

docker image: 1.4 model: teknium/OpenHermes-2.5-Mistral-7B

Information

• [X] Docker
• [ ] The CLI directly

Tasks

• [X] An officially supported command
• [ ] My own modifications

Reproduction

~/tdi# docker run -e http_proxy=$http_proxy -e https_proxy=$https_proxy --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:1.4 --model-id $model 2024-04-11T14:14:28.352097Z INFO text_generation_launcher: Args { model_id: "teknium/OpenHermes-2.5-Mistral-7B", revision: None, validation_workers: 2, sharded: None, num_shard: None, quantize: None, speculate: None, dtype: None, trust_remote_code: false, max_concurrent_requests: 128, max_best_of: 2, max_stop_sequences: 4, max_top_n_tokens: 5, max_input_length: 1024, max_total_tokens: 2048, waiting_served_ratio: 1.2, max_batch_prefill_tokens: 4096, max_batch_total_tokens: None, max_waiting_tokens: 20, max_batch_size: None, enable_cuda_graphs: false, hostname: "95997621edea", port: 80, shard_uds_path: "/tmp/text-generation-server", master_addr: "localhost", master_port: 29500, huggingface_hub_cache: Some("/data"), weights_cache_override: None, disable_custom_kernels: false, cuda_memory_fraction: 1.0, rope_scaling: None, rope_factor: None, json_output: false, otlp_endpoint: None, cors_allow_origin: [], watermark_gamma: None, watermark_delta: None, ngrok: false, ngrok_authtoken: None, ngrok_edge: None, tokenizer_config_path: None, disable_grammar_support: false, env: false } 2024-04-11T14:14:28.352213Z INFO download: text_generation_launcher: Starting download process. Error: DownloadError 2024-04-11T14:14:31.454429Z ERROR download: text_generation_launcher: Download encountered an error: urllib3.exceptions.SSLError: [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

File "/opt/conda/lib/python3.10/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen(

File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 847, in urlopen retries = retries.increment(

File "/opt/conda/lib/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/teknium/OpenHermes-2.5-Mistral-7B (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/opt/conda/bin/text-generation-server", line 8, in sys.exit(app())

File "/opt/conda/lib/python3.10/site-packages/text_generation_server/cli.py", line 128, in download_weights utils.weight_files(model_id, revision, extension)

File "/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/hub.py", line 151, in weight_files filenames = weight_hub_files(model_id, revision, extension)

File "/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/hub.py", line 110, in weight_hub_files info = api.model_info(model_id, revision=revision)

File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn return fn(*args, **kwargs)

File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 1921, in model_info r = get_session().get(path, headers=headers, timeout=timeout, params=params)

File "/opt/conda/lib/python3.10/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs)

File "/opt/conda/lib/python3.10/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs)

File "/opt/conda/lib/python3.10/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs)

File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 63, in send return super().send(request, *args, **kwargs)

File "/opt/conda/lib/python3.10/site-packages/requests/adapters.py", line 517, in send raise SSLError(e, request=request)

requests.exceptions.SSLError: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/teknium/OpenHermes-2.5-Mistral-7B (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))"), '(Request ID: e30a461b-7756-4f94-8234-0c5edff6645e)')

Expected behavior

Work.

[daixiang0]

BTW, is it possible to download by hand and put it somewhere?

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[remmen-io]

Simply download your model to a location (here models-cache) and set HUGGINGFACE_HUB_CACHE=/models-cache and MODEL_ID=/models-cache/deepseek-coder-33B-instruct-AWQ

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.