ollama icon indicating copy to clipboard operation
ollama copied to clipboard
Published 3 months ago • verified publisher icon ollama

Model download finally fails behind company firewall

Open berndgoetz opened this issue 1 year ago • 1 comments

What is the issue?

I want to make ollama available to our developer community in our company in order to learn to use the technology. We got ollama.com/* whitelisted through our company firewall, and it actually pretty much works quite well, but at the end of the model download, the process gets stuck:

pulling 42ba7f8a01dd... 100% ▕███████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 557 B
verifying sha256 digest
Error: digest mismatch, file must be downloaded again: want sha256:8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246, got sha256:1dd340187fa2df434fb562efa41b824c8b1ef6f3f3f3bc401280f80d8adea61e

After some digging, it seems that our malware scanner runs into a false positive with a file served from a cloudflare location

image

The malware software reports this: "Trickling"...

image

Any chance to get this changed/fixed?

OS

Windows

GPU

AMD

CPU

AMD

Ollama version

0.1.30

berndgoetz avatar May 28 '24 19:05 berndgoetz

Are you able to pull small models, or does every model get wedged by the scanner?

Since models are so large, we try to download many chunks in parallel to maximize throughput, so perhaps that approach is causing the scanner to get backlogged to a point where it has to start rejecting requests? If you can talk to the folks that run the scanner in your setup, it would be interesting to know if it's overloaded when you're trying to pull a model.

Server logs might shed a little more light on what the http client is seeing as it's downloading. Is the chunk actually corrupt, or was the http exchange truncated in such a way that we thought it was finished. Our client is supposed to be smart to lost connections and resume where it left off, so it feels like the scanner might be behaving in a way that causes our resume logic to get confused.

#2006 might be relevant once implemented.

dhiltgen avatar Jul 26 '24 00:07 dhiltgen

I'm getting the same error. I tried pulling a smaller model like tinyllama:

$ ollama pull tinyllama
pulling manifest
pulling 2af3b81862c6... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 637 MB
pulling af0ddbdaaa26... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏   70 B
pulling c8472cd9daed... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏   31 B
pulling fa956ab37b8c... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏   98 B
pulling 6331358be52a... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏  483 B
verifying sha256 digest
Error: digest mismatch, file must be downloaded again: want sha256:2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816, got sha256:198c7dc595d2b1235c2ac7d54ed0f10422dbe294b5750739e133cc7b55ef6947

and the server log only adds these lines for this particular pull:

$ tail -f ~/.ollama/logs/server.log
...
[GIN] 2024/10/23 - 17:56:51 | 200 |       56.75µs |       127.0.0.1 | HEAD     "/"
time=2024-10-23T17:56:54.587+02:00 level=INFO source=download.go:175 msg="downloading 2af3b81862c6 in 7 100 MB part(s)"
[GIN] 2024/10/23 - 17:57:56 | 200 |          1m4s |       127.0.0.1 | POST     "/api/pull"

I see some errors in higher in the log, but I'm not sure if this is related:

Full log... 
update check failed - TypeError: fetch failed

2024/10/22 21:46:42 routes.go:1158: INFO server config env="map[HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/Users/foo/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: http_proxy: https_proxy: no_proxy:]"
time=2024-10-22T21:46:42.427+02:00 level=INFO source=images.go:754 msg="total blobs: 0"
time=2024-10-22T21:46:42.428+02:00 level=INFO source=images.go:761 msg="total unused blobs removed: 0"
time=2024-10-22T21:46:42.428+02:00 level=INFO source=routes.go:1205 msg="Listening on 127.0.0.1:11434 (version 0.3.14)"
time=2024-10-22T21:46:42.430+02:00 level=INFO source=common.go:135 msg="extracting embedded files" dir=/var/folders/hw/1y7j_mnd1qs8lf9y27gpd5v80000gp/T/ollama1183217415/runners
time=2024-10-22T21:46:42.456+02:00 level=INFO source=common.go:49 msg="Dynamic LLM libraries" runners=[metal]
time=2024-10-22T21:46:42.552+02:00 level=INFO source=types.go:123 msg="inference compute" id=0 library=metal variant="" compute="" driver=0.0 name="" total="21.3 GiB" available="21.3 GiB"
[GIN] 2024/10/22 - 21:49:37 | 200 |     163.792µs |       127.0.0.1 | HEAD     "/"
time=2024-10-22T21:49:41.538+02:00 level=INFO source=download.go:175 msg="downloading dde5aa3fc5ff in 16 126 MB part(s)"
time=2024-10-22T21:53:10.117+02:00 level=INFO source=download.go:175 msg="downloading 966de95ca8a6 in 1 1.4 KB part(s)"
time=2024-10-22T21:53:12.436+02:00 level=INFO source=download.go:175 msg="downloading fcc5a6bec9da in 1 7.7 KB part(s)"
time=2024-10-22T21:53:14.836+02:00 level=INFO source=download.go:175 msg="downloading a70ff7e570d9 in 1 6.0 KB part(s)"
time=2024-10-22T21:53:18.001+02:00 level=INFO source=download.go:175 msg="downloading 56bb8bd477a5 in 1 96 B part(s)"
time=2024-10-22T21:53:20.340+02:00 level=INFO source=download.go:175 msg="downloading 34bb5ab01051 in 1 561 B part(s)"
[GIN] 2024/10/22 - 21:53:22 | 200 |         3m44s |       127.0.0.1 | POST     "/api/pull"
[GIN] 2024/10/22 - 21:55:12 | 200 |      98.958µs |       127.0.0.1 | HEAD     "/"
time=2024-10-22T21:55:16.441+02:00 level=INFO source=download.go:175 msg="downloading dde5aa3fc5ff in 16 126 MB part(s)"
[GIN] 2024/10/22 - 21:58:42 | 200 |         3m30s |       127.0.0.1 | POST     "/api/pull"
[GIN] 2024/10/22 - 21:59:53 | 200 |      89.708µs |       127.0.0.1 | HEAD     "/"
time=2024-10-22T21:59:56.980+02:00 level=INFO source=download.go:175 msg="downloading daec91ffb5dd in 12 100 MB part(s)"
time=2024-10-22T22:01:59.348+02:00 level=INFO source=download.go:175 msg="downloading a406579cd136 in 1 1.1 KB part(s)"
time=2024-10-22T22:02:01.874+02:00 level=INFO source=download.go:175 msg="downloading 0c4c9c2a325f in 1 337 B part(s)"
[GIN] 2024/10/22 - 22:02:03 | 200 |         2m10s |       127.0.0.1 | POST     "/api/pull"
[GIN] 2024/10/22 - 22:15:16 | 204 |     137.042µs |       127.0.0.1 | OPTIONS  "/api/chat"
[GIN] 2024/10/22 - 22:15:16 | 404 |    2.478416ms |       127.0.0.1 | POST     "/api/chat"
[GIN] 2024/10/22 - 22:16:06 | 200 |      57.792µs |       127.0.0.1 | HEAD     "/"
time=2024-10-22T22:16:10.265+02:00 level=INFO source=download.go:175 msg="downloading dde5aa3fc5ff in 16 126 MB part(s)"
[GIN] 2024/10/22 - 22:19:31 | 200 |         3m25s |       127.0.0.1 | POST     "/api/pull"
update check failed - TypeError: fetch failed
update check failed - TypeError: fetch failed
[GIN] 2024/10/22 - 23:46:42 | 200 |         139µs |       127.0.0.1 | HEAD     "/"
[GIN] 2024/10/22 - 23:46:42 | 200 |    1.587333ms |       127.0.0.1 | GET      "/api/tags"
[GIN] 2024/10/22 - 23:46:45 | 200 |     114.833µs |       127.0.0.1 | HEAD     "/"
[GIN] 2024/10/22 - 23:46:45 | 200 |       297.5µs |       127.0.0.1 | GET      "/api/ps"
[GIN] 2024/10/22 - 23:56:12 | 200 |      99.625µs |       127.0.0.1 | HEAD     "/"
time=2024-10-22T23:56:17.014+02:00 level=INFO source=download.go:175 msg="downloading 2af3b81862c6 in 7 100 MB part(s)"
time=2024-10-22T23:57:17.892+02:00 level=INFO source=download.go:175 msg="downloading af0ddbdaaa26 in 1 70 B part(s)"
time=2024-10-22T23:57:20.158+02:00 level=INFO source=download.go:175 msg="downloading c8472cd9daed in 1 31 B part(s)"
time=2024-10-22T23:57:22.440+02:00 level=INFO source=download.go:175 msg="downloading fa956ab37b8c in 1 98 B part(s)"
time=2024-10-22T23:57:24.802+02:00 level=INFO source=download.go:175 msg="downloading 6331358be52a in 1 483 B part(s)"
[GIN] 2024/10/22 - 23:57:26 | 200 |         1m13s |       127.0.0.1 | POST     "/api/pull"
update check failed - TypeError: fetch failed
update check failed - TypeError: fetch failed
update check failed - TypeError: fetch failed
update check failed - TypeError: fetch failed
update check failed - TypeError: fetch failed
update check failed - TypeError: fetch failed
update check failed - TypeError: fetch failed
update check failed - TypeError: fetch failed
update check failed - TypeError: fetch failed
update check failed - TypeError: fetch failed
update check failed - TypeError: fetch failed
update check failed - TypeError: fetch failed
update check failed - TypeError: fetch failed
update check failed - TypeError: fetch failed
update check failed - TypeError: fetch failed
update check failed - TypeError: fetch failed
update check failed - TypeError: fetch failed
[GIN] 2024/10/23 - 17:44:11 | 200 |      88.625µs |       127.0.0.1 | HEAD     "/"
time=2024-10-23T17:44:16.227+02:00 level=INFO source=download.go:175 msg="downloading 2af3b81862c6 in 7 100 MB part(s)"
[GIN] 2024/10/23 - 17:45:34 | 200 |         1m22s |       127.0.0.1 | POST     "/api/pull"

I'm behind a corporate proxy, but I don't see any logs of this being blocked. What else could be a problem? Is there any workaround for this?

laughedelic avatar Oct 23 '24 16:10 laughedelic

Could you try ollama-downloader? It has worked for me even behind a HTTPS proxy with a self-signed certificate, behind which ollama pull did not work for a single model, failing with the SHA256 check despite being able to download all BLOBs. (Disclaimer: I am the maintainer of the ollama-downloader project.)

anirbanbasu avatar Aug 23 '25 12:08 anirbanbasu