feat: CA truststore support

[GillesBodart]

Bug Report

Description

When you run the docker image in an enterprise context, based on the company policy, you may have SSL interception in order to anlyse the traffic

Bug Summary: Impossible to add some CA to the internal trustore used to make the REST API request

Steps to Reproduce: need to have SSL interception enable on your laptop that breaks the SSL chain

Expected Behavior: Possibility to add CA to the used truststore

Actual Behavior:

CA can't be added

Environment

• **Windows 11 running docker image ghcr.io/open-webui/open-webui:main
• **Browser Chrome

Reproduction Details

Confirmation:

• [* ] I have read and followed all the instructions provided in the README.md.
• [* ] I am on the latest version of both Open WebUI and Ollama.
• [* ] I have included the browser console logs.
• [ *] I have included the Docker container logs.

Logs and Screenshots

Browser Console Logs: { "detail": "Something went wrong :/\nHTTPSConnectionPool(host='api.openai.com', port=443): Max retries exceeded with url: /v1/images/generations (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)')))" }

Docker Container Logs: INFO: 172.17.0.1:50270 - "GET /ollama/api/tags HTTP/1.1" 200 OK

INFO:apps.openai.main:get_all_models()

ERROR:apps.openai.main:Connection error: Cannot connect to host api.openai.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)')]

INFO:apps.openai.main:models: {'data': []}

INFO:apps.openai.main:get_all_models()

ERROR:apps.openai.main:Connection error: Cannot connect to host api.openai.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)')]

INFO:apps.openai.main:models: {'data': []}

INFO: 172.17.0.1:50270 - "GET /openai/api/models HTTP/1.1" 200 OK

INFO: 172.17.0.1:50270 - "GET /litellm/api/v1/models HTTP/1.1" 200 OK

INFO: 172.17.0.1:50294 - "GET /_app/immutable/nodes/8.0396dff0.js HTTP/1.1" 200 OK

INFO: 172.17.0.1:50298 - "GET /ollama/api/version HTTP/1.1" 200 OK

INFO: 172.17.0.1:50304 - "GET /ollama/api/version HTTP/1.1" 200 OK

INFO: 172.17.0.1:50304 - "GET /ollama/urls HTTP/1.1" 200 OK

INFO: 172.17.0.1:50304 - "GET /ollama/api/version HTTP/1.1" 200 OK

INFO: 172.17.0.1:50304 - "GET /litellm/api/model/info HTTP/1.1" 200 OK

INFO: 172.17.0.1:50316 - "GET /api/config HTTP/1.1" 200 OK

INFO: 172.17.0.1:50316 - "GET /api/v1/auths/ HTTP/1.1" 200 OK

INFO:apps.ollama.main:get_all_models()

INFO: 172.17.0.1:50316 - "GET /ollama/api/tags HTTP/1.1" 200 OK

INFO:apps.openai.main:get_all_models()

ERROR:apps.openai.main:Connection error: Cannot connect to host api.openai.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)')]

INFO:apps.openai.main:models: {'data': []}

INFO:apps.openai.main:get_all_models()

ERROR:apps.openai.main:Connection error: Cannot connect to host api.openai.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)')]

INFO:apps.openai.main:models: {'data': []}

INFO: 172.17.0.1:50316 - "GET /openai/api/models HTTP/1.1" 200 OK

INFO: 172.17.0.1:50316 - "GET /litellm/api/v1/models HTTP/1.1" 200 OK

INFO: 172.17.0.1:50316 - "GET /api/v1/modelfiles/ HTTP/1.1" 200 OK

INFO: 172.17.0.1:50316 - "GET /api/v1/prompts/ HTTP/1.1" 200 OK

INFO: 172.17.0.1:50316 - "GET /api/v1/documents/ HTTP/1.1" 200 OK

INFO: 172.17.0.1:50316 - "GET /api/v1/chats/tags/all HTTP/1.1" 200 OK

INFO:apps.ollama.main:get_all_models()

Screenshots (if applicable):

Installation Method

Docker vanilla install with Open API key

Additional Information

[Include any additional details that may help in understanding and reproducing the issue. This could include specific configurations, error messages, or anything else relevant to the bug.]

Note

If the bug report is incomplete or does not follow the provided instructions, it may not be addressed. Please ensure that you have followed the steps outlined in the README.md and troubleshooting.md documents, and provide all necessary information for us to reproduce and address the issue. Thank you!

[strikeoncmputrz]

This would be an excellent feature. I'd rather not have to run my openai compatible inference server in http mode but I'm using a private public key infrastructure

[tjbck]

Feel free to make a PR!

[theobjectivedad]

For folks stuck on this, here is a quick hack you can use ... basically you can just map /etc/ssl/certs/ca-certificates.crt in the container to a another file that contains your own trust chain.

In this example, I have my internal root CA trusted on the host machine and can map it directly into the container (don't forget to make it read only just in case):

docker run -it --rm \
  --publish=3000:8080 \
  --volume=open-webui:/app/backend/data \
  --volume=/etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro \
  ghcr.io/open-webui/open-webui:main

For a larger-scale deployment you can basically do the same thing, ex in Kubernetes maybe you are using something like trust-manager to manage your truststore within the cluster. You can mount the trust-manager managed bundle to /etc/ssl/certs/ca-certificates.crt inside the open webui pod.

This isn't a best practice and and environment variable configuration to add an additional truststore would be cleaner but this will work in the interim. As an aside, I tried to set REQUESTS_CA_BUNDLE in the environment but it didn't work for me.

[hanahige]

I'm not able to get the fix above to work, I've pulled my CA bundle from my AD Domain and added it into the container as mentioned above, however I still get a SSL: CERTIFICATE_VERIFY_FAILED error. ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006) however I did notice that the above method allows me to curl the .well-known address without having to ignore validation. It just seems like Open WebUI doesn't also respect that the CA root for my domain is trusted once added to the container CA trust. I'm running Open WebUI 0.3.7, and Authentik 2024.4.7 behind a NGINX reverse proxy.

[KizzyCode]

FWIW, you can force python to use the system certificate store via REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt in the environment.

My (trimmed down) docker-compose.yml looks like this:

services:
  openwebui:
    image: ghcr.io/open-webui/open-webui:main
    volumes:
      - /var/containers/openwebui:/app/backend/data:rw
      - /etc/containers/openwebui/compusrv.crt:/etc/ssl/certs/ca-certificates.crt:ro
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    environment:
      - WEBUI_NAME=compusrv
      - ENABLE_SIGNUP=False
      - ENABLE_COMMUNITY_SHARING=False
      - WEBUI_SESSION_COOKIE_SAME_SITE=strict
      - WEBUI_SESSION_COOKIE_SECURE=True
      - ENABLE_OLLAMA_API=False
      - REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

where /etc/containers/openwebui/compusrv.crt is simply my self-signed certificate (i.e. you don't even need to build a full truststore, if you don't want to).

[nomppy]

As an alternative to what @KizzyCode said and @theobjectivedad said about setting the REQUESTS_CA_BUNDLE env + mounting the host store, you can also add the following commands in the Dockerfile

COPY <CorporateSSL.crt> /usr/local/share/ca-certificates/
RUN update-ca-certificates

btw, I also have set

ENV PIP_CERT=/etc/ssl/certs/ca-certificates.crt \
    REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

Note that if you're building with the Dockerfile, you will also need to add the cert for the frontend build stage

[christiangierschner]

FWIW, you can force python to use the system certificate store via REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt in the environment.

My (trimmed down) docker-compose.yml looks like this:

services:
  openwebui:
    image: ghcr.io/open-webui/open-webui:main
    volumes:
      - /var/containers/openwebui:/app/backend/data:rw
      - /etc/containers/openwebui/compusrv.crt:/etc/ssl/certs/ca-certificates.crt:ro
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    environment:
      - WEBUI_NAME=compusrv
      - ENABLE_SIGNUP=False
      - ENABLE_COMMUNITY_SHARING=False
      - WEBUI_SESSION_COOKIE_SAME_SITE=strict
      - WEBUI_SESSION_COOKIE_SECURE=True
      - ENABLE_OLLAMA_API=False
      - REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

where /etc/containers/openwebui/compusrv.crt is simply my self-signed certificate (i.e. you don't even need to build a full truststore, if you don't want to).

Sadly, this does not work for me...

i still get SSL: CERTIFICATE_VERIFY_FAILED if i hit "Continue with OAuth2" on login screen

however if i exec into the container, this works:

>>> import requests
>>> response = requests.get('https://internal-keycloak-url/auth')
>>> print(response)
<Response [200]>

seems like webui is somehow not using the rootca

any ideas?

[nomppy]

What does your compose file look like?

[christiangierschner]

services:
  ollama:
    image: ghcr.io/open-webui/open-webui:v0.3.11
    volumes:
      - ollama-webui:/app/backend/data
      - /usr/local/share/ca-certificates/rootca.crt:/etc/ssl/certs/ca-certificates.crt:ro
    environment:
      OLLAMA_BASE_URL: http://ollama-api:11434
      ENABLE_OAUTH_SIGNUP: true
      ENABLE_LOGIN_FORM: false
      ENABLE_SIGNUP: false
      OAUTH_MERGE_ACCOUNTS_BY_EMAIL: true
      OAUTH_USERNAME_CLAIM: name
      OAUTH_CLIENT_ID: ollama
      OAUTH_CLIENT_SECRET:
      OPENID_PROVIDER_URL: https://$KEYCLOAKURL/auth/realms/$REALMNAME/.well-known/openid-configuration
      OAUTH_PROVIDER_NAME: Keycloak
      REQUESTS_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt
    extra_hosts:
      - host.docker.internal:host-gateway
    restart: unless-stopped
    networks:
      - backend
      - frontend

    labels:
      - traefik.enable=true
      - traefik.http.routers.ollama.rule=Host(`$HOST_NAME`)
      - traefik.http.routers.ollama.tls.certresolver=internal-acme
      - traefik.http.routers.ollama.service=ollama
      - traefik.http.routers.ollama.entrypoints=https
      - traefik.http.services.ollama.loadbalancer.server.port=8080

[nomppy]

Is /usr/local/share/ca-certificates/roocat.crt the complete CA used by your system. If it's just your custom root CA, then you are not giving python any of the usual certs usually present in /etc/ssl/certs/ca-certificates.crt.

Can you try update-ca-certificates on your host and then mapping /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro?

[christiangierschner]

i tried both...none works our hosts all have our rootca implmented with update-ca-certificates

i get Internal Server Error and SSL: CERTIFICATE_VERIFY_FAILED

services:
  ollama:
    image: ghcr.io/open-webui/open-webui:v0.3.12
    volumes:
      - ollama-webui:/app/backend/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
      - /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro
      - /usr/local/share/ca-certificates/rootca.crt:/usr/local/share/ca-certificates/rootca.crt
    environment:
      OLLAMA_BASE_URL: http://ollama-api:11434
      ENABLE_OAUTH_SIGNUP: true
      ENABLE_LOGIN_FORM: false
      ENABLE_SIGNUP: false
      OAUTH_MERGE_ACCOUNTS_BY_EMAIL: true
      OAUTH_USERNAME_CLAIM: name
      OAUTH_CLIENT_ID: ollama
      OAUTH_CLIENT_SECRET:
      OPENID_PROVIDER_URL: https://$KEYCLOAKURL/auth/realms/$REALMNAME/.well-known/openid-configuration
      OAUTH_PROVIDER_NAME: Keycloak
      REQUESTS_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt
    extra_hosts:
      - host.docker.internal:host-gateway
    restart: unless-stopped
    networks:
      - backend
      - frontend

    labels:
      - traefik.enable=true
      - traefik.http.routers.ollama.rule=Host(`$HOST_NAME`)
      - traefik.http.routers.ollama.tls.certresolver=ukd-internal
      - traefik.http.routers.ollama.service=ollama
      - traefik.http.routers.ollama.entrypoints=https
      - traefik.http.services.ollama.loadbalancer.server.port=8080

[nomppy]

• Can you manually check that /etcssl/certs/ca-certificates.crt contains the cert in rootca.crt?
• How did you obtain rootca.crt?
• Do you can SSL error only when you click login with oAuth, or also for other things? I.e., can you try to log in with username/password and see if you get an error for requests to e.g. OpenAI?

[bkev]

After struggling with this myself, thanks for the pointers in here.

What fixed this for me was

I copied the root.crt (from Caddy) to my Linux device and put it in /usr/local/share/ca-certificates

I then ran sudo update-ca-certificates which said it had added one cert.

After playing about, the only options that seemed to work me for me was adding these to my docker run command

-v /link/to/my/cert.crt:/etc/ssl/certs/ca-certificates.crt:ro

AND -e REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

After that, the Openweb-ui accepted the connections and it worked.

[christiangierschner]

• Can you manually check that /etcssl/certs/ca-certificates.crt contains the cert in rootca.crt?

  • How did you obtain rootca.crt?

  • Do you can SSL error only when you click login with oAuth, or also for other things? I.e., can you try to log in with username/password and see if you get an error for requests to e.g. OpenAI?

Yes, it contains the cert. I got it from our rootca server and we use it in all our servers and deployments via ansible. It works.

I only get the error, when i click on login with OAuth. Open-webui then connects to our internal keycloak secured with a signed cert from our rootca.

[ebakoba]

@christiangierschner and all the future people struggling with OAuth and custom CA certs.

It was able to deduce from the logs that OAuth does not use requests as its client module, but httpx instead. httpx documentation explains that custom CA certs could be used by using environment variable SSL_CERT_FILE. I was able to resolve my cert issue by adding:

environment:
  # ...
  # other values
  - SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

to docker compose.