tls: failed to verify certificate: x509: certificate signed by unknown authority

[berupp]

I am trying to circumvent the server certificate check:

conf := z.New(url.Hostname())
c, err := client.New(ctx, conf,
	client.WithAuth(client.DefaultServiceUserAuthentication(machineKeyFile, oidc.ScopeOpenID, client.ScopeZitadelAPI())),
        client.WithGRPCDialOptions(grpc.WithTransportCredentials(credentials.NewTLS(&tls.Config{InsecureSkipVerify: true}))),
)

Unfortunately, it seems zitadel-go is ignoring the provided tls.Config and performs the server certificate validation anyways.

Expectation

zitadel-go offering grpc.DialOptions to customize the underlying grpc connection should honour the provided configuration options.

[fforootd]

Hm since it does not work may I recommend a workaround?

To my understanding Go uses the systems CA store, so adding the signing cert there from the internal CA should allow to make it work.

[berupp]

I figured it out, but there is some severe lack of documentation paired with implicit logic that makes this quite annoying to figure out.

Firstly, it seems that zitadel-go uses the http.DefaultClient when setting up the connection initially. Overwriting the tls.Config on http.DefaultTransport allowed me to successfully set up the client and get past the error.

Fortunately zitadel-go, allows you to provide your own http.Client so you don't have to modify the DefaultTransport. With my own client and tls.Config with InsecureSkipVerify:true, I am now successfully connected.

But then, when sending an actual API request, it fails again with the same error. Now HERE is where the

client.WithGRPCDialOptions(grpc.WithTransportCredentials(credentials.NewTLS(&tls.Config{InsecureSkipVerify: true}))),

is actually required and used.

So the following allows me to actually skip verification of the certificate chain and interact with Zitadel.

conf := z.New(URL.Hostname())
	c, err := client.New(ctx, conf,
		client.WithAuth(func(ctx context.Context, issuer string) (oauth2.TokenSource, error) {
			return profile.NewJWTProfileTokenSourceFromKeyFile(ctx, zitadelURL, machineKeyFile, []string{oidc.ScopeOpenID, client.ScopeZitadelAPI()},
				profile.WithHTTPClient(&http.Client{Transport: &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}}}))
		}),
		client.WithGRPCDialOptions(grpc.WithTransportCredentials(credentials.NewTLS(&tls.Config{InsecureSkipVerify: true}))),
	)
	if err != nil {
		return nil, fmt.Errorf("unable to initialize Zitadel client: %w", err)
	}

It is quite unfortunate that this is documented so poorly and essentially requires you to debug through the code in order to understand what is happening.

[BillyBolton]

@fforootd Any update on this?

[fforootd]

Sorry to say but I did not work on this.

@hifabienne @muhlemmer this could be a good issue to look into for someone new

[amitavaghosh2]

docker-composes:

zitadel service account

services:
  zitadel:
    # The user should have the permission to write to ./machinekey
    user: "${UID:-1000}"
    restart: 'always'
    networks:
      - 'zitadel'
    image: 'ghcr.io/zitadel/zitadel:latest'
    command: 'start-from-init --masterkey "MasterkeyNeedsToHave32Characters" --tlsMode disabled'
    environment:
      ZITADEL_DATABASE_POSTGRES_HOST: db
      ZITADEL_DATABASE_POSTGRES_PORT: 5432
      ZITADEL_DATABASE_POSTGRES_DATABASE: zitadel
      ZITADEL_DATABASE_POSTGRES_USER_USERNAME: zitadel
      ZITADEL_DATABASE_POSTGRES_USER_PASSWORD: zitadel
      ZITADEL_DATABASE_POSTGRES_USER_SSL_MODE: disable
      ZITADEL_DATABASE_POSTGRES_ADMIN_USERNAME: postgres
      ZITADEL_DATABASE_POSTGRES_ADMIN_PASSWORD: postgres
      ZITADEL_DATABASE_POSTGRES_ADMIN_SSL_MODE: disable
      # ZITADEL_EXTERNALSECURE: false
      ZITADEL_EXTERNALPORT: 443
      ZITADEL_EXTERNALSECURE: true
      ZITADEL_TLS_ENABLED: false
      ZITADEL_EXTERNALDOMAIN: 127.0.0.1.sslip.io
      ZITADEL_LOGSTORE_ACCESS_STDOUT_ENABLED: true
      ZITADEL_FIRSTINSTANCE_MACHINEKEYPATH: /machinekey/zitadel-admin-sa.json
      ZITADEL_FIRSTINSTANCE_ORG_MACHINE_MACHINE_USERNAME: zitadel-admin-sa
      ZITADEL_FIRSTINSTANCE_ORG_MACHINE_MACHINE_NAME: Admin
      ZITADEL_FIRSTINSTANCE_ORG_MACHINE_MACHINEKEY_TYPE: 1
    depends_on:
      db:
        condition: 'service_healthy'
    ports:
      - '8080:8080'
    volumes:
      - ./machinekey:/machinekey
    healthcheck:
      test: ["CMD", "/app/zitadel", "ready"]
      interval: '10s'
      timeout: '5s'
      retries: 5
      start_period: '10s'

  db:
    restart: 'always'
    image: postgres:16-alpine
    environment:
      PGUSER: postgres
      POSTGRES_PASSWORD: postgres
    networks:
      - 'zitadel'
    healthcheck:
      test: ["CMD-SHELL", "pg_isready", "-d", "zitadel", "-U", "postgres"]
      interval: '10s'
      timeout: '30s'
      retries: 5
      start_period: '20s'

networks:
  zitadel:

**nginx config

services:
  proxy-external-tls:
    image: "nginx:mainline-alpine"
    volumes:
      - "./nginx-external-tls.conf:/etc/nginx/nginx.conf:ro"
      - "./selfsigned.crt:/etc/certs/selfsigned.crt:ro"
      - "./selfsigned.key:/etc/certs/selfsigned.key:ro"
    ports:
      - "443:443"
    networks:
      - 'zitadel'
    depends_on:
      zitadel:
        condition: 'service_healthy'

networks:
  zitadel:

Rest of the setup script looks like so.

#!/bin/bash

# copied from zitadel docs

# Download the configuration files.
ZITADEL_CONFIG_FILES="https://raw.githubusercontent.com/zitadel/zitadel/main/docs/docs/self-hosting/manage/reverseproxy"

download() {
	URL="$1"
	SAVE_PATH="$2"

	[[ ! -f "$SAVE_PATH" ]] && curl "$URL" > "$SAVE_PATH"
}

download "https://raw.githubusercontent.com/zitadel/zitadel/main/docs/docs/self-hosting/deploy/docker-compose-sa.yaml" "zitadel-sa.compose.yml"

download "${ZITADEL_CONFIG_FILES}/nginx/docker-compose.yaml" "docker-compose-nginx.yaml"

download "${ZITADEL_CONFIG_FILES}/nginx/nginx-external-tls.conf" "nginx-external-tls.conf"

# Generate a self signed certificate and key.
openssl req -x509 -batch -subj "/CN=127.0.0.1.sslip.io/O=ZITADEL Demo" -nodes -newkey rsa:2048 -keyout ./selfsigned.key -out ./selfsigned.crt 2>/dev/null

# Run the database, ZITADEL and NGINX.
docker compose -f zitadel-sa.compose.yml -f docker-compose-nginx.yaml up --wait db zitadel proxy-external-tls

# Test that gRPC and HTTP APIs work. Empty brackets like {} means success.
# Make sure you have the grpcurl cli installed on your machine https://github.com/fullstorydev/grpcurl?tab=readme-ov-file#installation
# grpcurl --insecure 127.0.0.1.sslip.io:443 zitadel.admin.v1.AdminService/Healthz
curl --insecure https://127.0.0.1.sslip.io:443/admin/v1/healthz -v

The curl works but, the client-go doesn't work.

package main

import (
	"context"
	"flag"
	"log"
	"os"

	"github.com/zitadel/oidc/v3/pkg/oidc"
	"golang.org/x/exp/slog"

	"github.com/zitadel/zitadel-go/v3/pkg/client"
	"github.com/zitadel/zitadel-go/v3/pkg/client/zitadel/management"
	"github.com/zitadel/zitadel-go/v3/pkg/zitadel"
)

var (
	// flags to be provided for running the example server
	domain  = flag.String("domain", "", "your ZITADEL instance domain (in the form: <instance>.zitadel.cloud or <yourdomain>)")
	keyPath = flag.String("key", "", "path to your key.json")
)

func main() {
	flag.Parse()

	ctx := context.Background()

	// Initiate the API client by providing at least a zitadel configuration.
	// You can also directly set an authorization option, resp. provide its authentication mechanism,
	// by passing the downloaded service user key:
	api, err := client.New(ctx, zitadel.New(*domain),
		client.WithAuth(client.DefaultServiceUserAuthentication(*keyPath, oidc.ScopeOpenID, client.ScopeZitadelAPI())),
	)
	if err != nil {
		log.Fatal(err)
	}

	// In this example we will just use the ManagementService to retrieve the users organisation,
	// but you can use the API for all the other services (Admin, Auth, User, Session, ...) too.
	resp, err := api.ManagementService().GetMyOrg(ctx, &management.GetMyOrgRequest{})
	if err != nil {
		slog.Error("cannot retrieve the organisation", "error", err)
		os.Exit(1)
	}
	slog.Info("retrieved the organisation", "orgID", resp.GetOrg().GetId(), "name", resp.GetOrg().GetName())
}

$> go run main.go -domain 127.0.0.1.sslip.io -key ./machinekey/zitadel-admin-sa.json

Output:

2025/03/11 16:50:19 OpenID Provider Configuration Discovery has failed
Get "https://127.0.0.1.sslip.io/.well-known/openid-configuration": tls: failed to verify certificate: x509: certificate signed by unknown authority
exit status 1

why? I deleted a couple of stuff from the original docker-compose, https://zitadel.com/docs/self-hosting/manage/reverseproxy/nginx#tls-mode-external , What am I doing wrong!

[kilianstallz]

+1 for me!

I could absolutely not figure out how to correctly overwrite this behavior in the client configurations.

So as a workaround i needed to set the global http client handling to skip tls verification in dev.

func init() {
    http.DefaultTransport.(*http.Transport).TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
}

[danielloader]

Even this only gets me past the .well-known jwks lookup.

Now I'm facing the following error:

2025/08/06 20:13:41 Failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: credentials: cannot check peer: missing selected ALPN property. If you upgraded from a grpc-go version earlier than 1.67, your TLS connections may have stopped working due to ALPN enforcement. For more details, see: https://github.com/grpc/grpc-go/issues/434"
exit status 1

Example code:

package main

import (
	"context"
	"crypto/tls"
	"log"
	"net/http"

	"github.com/zitadel/oidc/v3/pkg/oidc"
	"github.com/zitadel/zitadel-go/v3/pkg/client"
	"github.com/zitadel/zitadel-go/v3/pkg/client/zitadel/org/v2"
	"github.com/zitadel/zitadel-go/v3/pkg/zitadel"
)

func init() {
	http.DefaultTransport.(*http.Transport).TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
}

func main() {
	ctx := context.Background()
	authOption := client.DefaultServiceUserAuthentication(
		"./local/secrets/iam-owner-service-account.json",
		oidc.ScopeOpenID,
		client.ScopeZitadelAPI(),
	)

	client, err := client.New(
		ctx,
		zitadel.New("login.127.0.0.1.sslip.io", zitadel.WithInsecureSkipVerifyTLS()),
		client.WithAuth(authOption),
	)
	if err != nil {
		log.Fatalf("Failed: %s", err.Error())
	}

	staffOrgId := "000000000000000001"
	_, err = client.OrganizationServiceV2().AddOrganization(ctx, &org.AddOrganizationRequest{
		Name:  "example",
		OrgId: &staffOrgId,
	})
	if err != nil {
		log.Fatalf("Failed: %s", err.Error())
	}

}

Edit: Got it working by adding the ALPN configuration to my envoy reverse proxy:

# rest of config omitted above
      transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
          common_tls_context:
            # Add ALPN protocols for HTTP/2 and HTTP/1.1
            alpn_protocols: ["h2", "http/1.1"]
            tls_certificates:
            - certificate_chain:
                filename: "/etc/envoy/certs/cert.pem"
              private_key:
                filename: "/etc/envoy/certs/key.pem"