Can't connect to Ollama on Windows

[xpilasneo4j]

Using the GenAI stack from Docker and having built my Ollama on Windows, I tried to run the stack and I have this message

genai-stack-pull-model-1  | pulling ollama model llama2 using http://localhost:11434
genai-stack-pull-model-1  | Error: could not connect to ollama server, run 'ollama serve' to start it

But my ollama is running, I can use it in command line, I can pull llama2 in command line... then all seems OK on the Ollama side (except it's Windows and not really supported (yet) by Ollama) 2023/11/01 17:38:54 routes.go:678: Listening on 127.0.0.1:11434 (version 0.0.0)

[xpilasneo4j]

Checking the file pull_model.Dockerfile, I see the below (process/shell {:env {"OLLAMA_HOST" url} :out :inherit :err :inherit} (format "./bin/ollama pull %s" llm)) I don't believe that will work on windows or it has to follow the same path with a bin/ directory I changed the ./bin into my windows path to Ollama server and it worked

Then it's a question of making sure the path to call Ollama is recognized by Windows

[jexp]

@mchiang0610 either we need to figure out a way to generalize this or at least make a clear call-out in the readme?

[prillcode]

So does Ollama run on Windows in the WSL linux subsystem that Docker runs in?

[xpilasneo4j]

Ollama can run on both:

• you can run it on WSL
• you can download the code and build it on Windows to run natively

[icejean]

Checking the file pull_model.Dockerfile, I see the below (process/shell {:env {"OLLAMA_HOST" url} :out :inherit :err :inherit} (format "./bin/ollama pull %s" llm)) I don't believe that will work on windows or it has to follow the same path with a bin/ directory I changed the ./bin into my windows path to Ollama server and it worked

Then it's a question of making sure the path to call Ollama is recognized by Windows

How to fix the issue with replaceing the ./bin into my windows path? Does it need to replace all ./bin in file pull_model.Dockerfile? For example, my windows path is C:\Users\Jean\AppData\Local\Programs\Ollama.

#syntax = docker/dockerfile:1.4

FROM ollama/ollama:latest AS ollama
FROM babashka/babashka:latest

# just using as a client - never as a server
COPY --from=ollama /bin/ollama ./bin/ollama

COPY <<EOF pull_model.clj
(ns pull-model
  (:require [babashka.process :as process]
            [clojure.core.async :as async]))

(try
  (let [llm (get (System/getenv) "LLM")
        url (get (System/getenv) "OLLAMA_BASE_URL")]
    (println (format "pulling ollama model %s using %s" llm url))
    (if (and llm url (not (#{"gpt-4" "gpt-3.5" "claudev2"} llm)))

      ;; ----------------------------------------------------------------------
      ;; just call `ollama pull` here - create OLLAMA_HOST from OLLAMA_BASE_URL
      ;; ----------------------------------------------------------------------
      ;; TODO - this still doesn't show progress properly when run from docker compose

      (let [done (async/chan)]
        (async/go-loop [n 0]
          (let [[v _] (async/alts! [done (async/timeout 5000)])]
            (if (= :stop v) :stopped (do (println (format "... pulling model (%ss) - will take several minutes" (* n 10))) (recur (inc n))))))
        (process/shell {:env {"OLLAMA_HOST" url} :out :inherit :err :inherit} (format "bash -c './bin/ollama show %s --modelfile > /dev/null || ./bin/ollama pull %s'" llm llm))
        (async/>!! done :stop))

      (println "OLLAMA model only pulled if both LLM and OLLAMA_BASE_URL are set and the LLM model is not gpt")))
  (catch Throwable _ (System/exit 1)))
EOF

ENTRYPOINT ["bb", "-f", "pull_model.clj"]

[icejean]

Well, addressed already. All ./bin need to be replaced, and should use OLLAMA_BASE_URL=http://host.docker.internal:11434 to reference to llama2/3 model running on local Windows machine. pull_model.Dockerfile:

#syntax = docker/dockerfile:1.4

FROM ollama/ollama:latest AS ollama
FROM babashka/babashka:latest

# just using as a client - never as a server
#COPY --from=ollama /bin/ollama ./bin/ollama
COPY --from=ollama /bin/ollama C:/Users/Jean/AppData/Local/Programs/Ollama/ollama



COPY <<EOF pull_model.clj
(ns pull-model
  (:require [babashka.process :as process]
            [clojure.core.async :as async]))

(try
  (let [llm (get (System/getenv) "LLM")
        url (get (System/getenv) "OLLAMA_BASE_URL")]
    (println (format "pulling ollama model %s using %s" llm url))
    (if (and llm url (not (#{"gpt-4" "gpt-3.5" "claudev2"} llm)))

      ;; ----------------------------------------------------------------------
      ;; just call `ollama pull` here - create OLLAMA_HOST from OLLAMA_BASE_URL
      ;; ----------------------------------------------------------------------
      ;; TODO - this still doesn't show progress properly when run from docker compose

      (let [done (async/chan)]
        (async/go-loop [n 0]
          (let [[v _] (async/alts! [done (async/timeout 5000)])]
            (if (= :stop v) :stopped (do (println (format "... pulling model (%ss) - will take several minutes" (* n 10))) (recur (inc n))))))
        #(process/shell {:env {"OLLAMA_HOST" url} :out :inherit :err :inherit} (format "bash -c './bin/ollama show %s --modelfile > /dev/null || ./bin/ollama pull %s'" llm llm))
        (process/shell {:env {"OLLAMA_HOST" url} :out :inherit :err :inherit} (format "bash -c 'C:/Users/Jean/AppData/Local/Programs/Ollama/ollama show %s --modelfile > /dev/null || C:/Users/Jean/AppData/Local/Programs/Ollama/ollama pull %s'" llm llm))        
        (async/>!! done :stop))

      (println "OLLAMA model only pulled if both LLM and OLLAMA_BASE_URL are set and the LLM model is not gpt")))
  (catch Throwable _ (System/exit 1)))
EOF

ENTRYPOINT ["bb", "-f", "pull_model.clj"]

.env:

#*****************************************************************
# LLM and Embedding Model
#*****************************************************************
LLM=llama3 #or any Ollama model tag,llama2, gpt-4, gpt-3.5, or claudev2
EMBEDDING_MODEL=sentence_transformer # sentence_transformer or google-genai-embedding-001 openai, ollama, or aws

#*****************************************************************
# Ollama
#*****************************************************************
#MAC & Windows, Access services on local machine through host.docker.internal
OLLAMA_BASE_URL=http://host.docker.internal:11434
#Linux
# OLLAMA_BASE_URL=http://llm:11434