wait4x Support self-signed TLS certiticate on HTTP component

Oct 07 '22 13:10 mortymacs

try mkcert and it will be accepted as valid in your system https://github.com/FiloSottile/mkcert

Nov 10 '22 21:11 lotyp

Hi @lotyp Thanks for the suggestion, but the idea is to use the already prepared certificates. For example, some APIs need certificate files prepared by the company and it's not public certificates. This issue is about those types of certificates.

Nov 11 '22 20:11 mortymacs

@mortymacs would an approach similar to https://forfuncsake.github.io/post/2017/08/trust-extra-ca-cert-in-go-app/ work? maybe using SystemCertPool instead of NewCertPool?

Nov 14 '22 07:11 sbaeurle

@sbaeurle I was thinking the same to use SystemCertPool by supporting additional CA file, but maybe it should support these 3 type of inputs like helm install command:

--ca-file
--cert-file
--key-file

What's your idea?

Nov 14 '22 08:11 mortymacs

Probably a good approach, what kind of formats would you expect for each argument? pem, crt and cem?

Nov 14 '22 21:11 sbaeurle

We're looking for something like: this https://gist.github.com/michaljemala/d6f4e01c4834bf47a9c4 What's your opinion?

Nov 15 '22 07:11 mortymacs

Looks like a good approach! The only open question on my side would be how to handle the system root store when we provide a custom certificate. Do we just append to the "internal" root store or do we override it and only accept our root cert?

Nov 15 '22 14:11 sbaeurle

It should be append in the system root store by using SystemCertPool.

Nov 15 '22 15:11 mortymacs

So, would you like to send a PR or I send it?

Nov 15 '22 15:11 mortymacs

I can take a look at it. I'm probably quite booked this week, but next week looks promising and I will try to adjust the certificate handling then :)

Nov 16 '22 16:11 sbaeurle

@sbaeurle awesome!

Nov 17 '22 20:11 mortymacs

Today tried to use wait4x on my local domains, like https://router.docker, or https://auth.wod.docker And I don't understand, is this problem, because I've installed custom certs with mkcert on my machine, and wait4x can't see them?

But, in browser, I can open these domains:

I'm using traefik locally, with mkcert

I have macos Ventura 13.0.1

Interesting, that if I try, shipped curl with macos, it works and can access domains:

I'm wondering, is this related to custom SSL certs ?

Currently, can't use wait4x to ping services using http

Nov 29 '22 08:11 vbnetadmin

@vbnetadmin please add -l debug to the wait4x command, so we can verify if the incorrect ssl validation is indeed the problem

PS. logs in text format are better than screenshots, so we can copy and compare it more easily

Nov 29 '22 09:11 sbaeurle

Output:

Wait4X with debug

➜ wait4x http https://router.docker -l debug
2022-11-29T12:43:09+02:00 INF [HTTP] Checking the https://router.docker ...
2022-11-29T12:43:09+02:00 DBG Get "https://router.docker": dial tcp: lookup router.docker on 100.64.0.2:53: no such host
2022-11-29T12:43:10+02:00 INF [HTTP] Checking the https://router.docker ...
2022-11-29T12:43:10+02:00 DBG Get "https://router.docker": dial tcp: lookup router.docker on 100.64.0.2:53: no such host
2022-11-29T12:43:11+02:00 INF [HTTP] Checking the https://router.docker ...
2022-11-29T12:43:11+02:00 DBG Get "https://router.docker": dial tcp: lookup router.docker on 100.64.0.2:53: no such host
2022-11-29T12:43:12+02:00 INF [HTTP] Checking the https://router.docker ...
2022-11-29T12:43:12+02:00 DBG Get "https://router.docker": dial tcp: lookup router.docker on 100.64.0.2:53: no such host
2022-11-29T12:43:13+02:00 INF [HTTP] Checking the https://router.docker ...
2022-11-29T12:43:13+02:00 DBG Get "https://router.docker": dial tcp: lookup router.docker on 100.64.0.2:53: no such host
2022-11-29T12:43:14+02:00 INF [HTTP] Checking the https://router.docker ...
2022-11-29T12:43:14+02:00 DBG Get "https://router.docker": dial tcp: lookup router.docker on 100.64.0.2:53: no such host
2022-11-29T12:43:15+02:00 INF [HTTP] Checking the https://router.docker ...
2022-11-29T12:43:15+02:00 DBG Get "https://router.docker": dial tcp: lookup router.docker on 100.64.0.2:53: no such host
2022-11-29T12:43:16+02:00 INF [HTTP] Checking the https://router.docker ...
2022-11-29T12:43:16+02:00 DBG Get "https://router.docker": dial tcp: lookup router.docker on 100.64.0.2:53: no such host
2022-11-29T12:43:17+02:00 INF [HTTP] Checking the https://router.docker ...
2022-11-29T12:43:17+02:00 DBG Get "https://router.docker": dial tcp: lookup router.docker on 100.64.0.2:53: no such host
2022-11-29T12:43:18+02:00 INF [HTTP] Checking the https://router.docker ...
2022-11-29T12:43:18+02:00 DBG Get "https://router.docker": dial tcp: lookup router.docker on 100.64.0.2:53: no such host

Ping:

ping router.docker
PING router.docker (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.051 ms
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.104 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.260 ms
64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.311 ms

Brew curl:

➜ curl -L router.docker
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

macOS default curl:

➜ /usr/bin/curl -L https://router.docker
<!DOCTYPE html><html><head><title>Traefik</title><meta charset=utf-8><meta name=description content="Traefik UI"><meta name=format-detection content="telephone=no"><meta name=msapplication-tap-highlight content=no><meta name=viewport content="user-scalable=no,initial-scale=1,maximum-scale=1,minimum-scale=1,width=device-width"><link rel=icon type=image/png href=statics/app-logo-128x128.png><link rel=icon type=image/png sizes=16x16 href=statics/icons/favicon-16x16.png><link rel=icon type=image/png sizes=32x32 href=statics/icons/favicon-32x32.png><link rel=icon type=image/png sizes=96x96 href=statics/icons/favicon-96x96.png><link rel=icon type=image/ico href=statics/icons/favicon.ico><link rel=apple-touch-icon href=statics/icons/apple-icon-152x152.png><link rel=apple-touch-icon sizes=152x152 href=statics/icons/apple-icon-152x152.png><link rel=apple-touch-icon sizes=167x167 href=statics/icons/apple-icon-167x167.png><link rel=apple-touch-icon sizes=180x180 href=statics/icons/apple-icon-180x180.png><link href=css/041f4ca8.f6e3d404.css rel=prefetch><link href=css/05759188.7a9c0db7.css rel=prefetch><link href=css/09354cb8.0e433876.css rel=prefetch><link href=css/09995f48.60caa8f0.css rel=prefetch><link href=css/0c428e32.ed2ba60f.css rel=prefetch><link href=css/18c6e965.5d8876c8.css rel=prefetch><link href=css/3e3ce03c.9b22e80f.css rel=prefetch><link href=css/46fd955e.9b22e80f.css rel=prefetch><link href=css/491024e9.9b22e80f.css rel=prefetch><link href=css/4f0a01e4.f8382310.css rel=prefetch><link href=css/52875482.9b22e80f.css rel=prefetch><link href=css/54b4dbdf.6e23d447.css rel=prefetch><link href=css/6d73c73c.9b22e80f.css rel=prefetch><link href=css/77d911b4.9b22e80f.css rel=prefetch><link href=css/797e4f23.9b22e80f.css rel=prefetch><link href=css/b34404c8.9b22e80f.css rel=prefetch><link href=js/041f4ca8.4f70d06d.js rel=prefetch><link href=js/05759188.f1376142.js rel=prefetch><link href=js/09354cb8.2e39e534.js rel=prefetch><link href=js/09995f48.a140ed0a.js rel=prefetch><link href=js/0c428e32.92dcacf8.js rel=prefetch><link href=js/18c6e965.5dc22ece.js rel=prefetch><link href=js/2d21e8fd.c00ac0e6.js rel=prefetch><link href=js/3e3ce03c.293f4729.js rel=prefetch><link href=js/46fd955e.343a8488.js rel=prefetch><link href=js/491024e9.5ba4955c.js rel=prefetch><link href=js/4f0a01e4.e6772caf.js rel=prefetch><link href=js/52875482.65e645a6.js rel=prefetch><link href=js/54b4dbdf.915c5773.js rel=prefetch><link href=js/6d73c73c.d8a27bbb.js rel=prefetch><link href=js/77d911b4.8d5002b7.js rel=prefetch><link href=js/797e4f23.771f63ae.js rel=prefetch><link href=js/b34404c8.7a4b9be8.js rel=prefetch><link href=css/app.3d2d5994.css rel=preload as=style><link href=js/app.45ca101c.js rel=preload as=script><link href=js/runtime.db7dd4a3.js rel=preload as=script><link href=js/vendor.f60bcb86.js rel=preload as=script><link href=css/app.3d2d5994.css rel=stylesheet></head><body><div id=q-app></div><script type=text/javascript src=js/app.45ca101c.js></script><script type=text/javascript src=js/runtime.db7dd4a3.js></script><script type=text/javascript src=js/vendor.f60bcb86.js></script></body></html>%

Nov 29 '22 10:11 vbnetadmin

Hi Patrick,

This is the DNS problem. Wait4x tries to lookup the domain via 100.64.0.2 DNS server which is depends on your system configuration. Would you please try with dig router.docker @100.64.0.2 and let us know the result?

Nov 29 '22 10:11 mortymacs

@mortymacs

➜ dig router.docker @100.64.0.2

; <<>> DiG 9.10.6 <<>> router.docker @100.64.0.2 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11785 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION: ;router.docker. IN A

;; Query time: 35 msec ;; SERVER: 100.64.0.2#53(100.64.0.2) ;; WHEN: Tue Nov 29 13:02:07 EET 2022 ;; MSG SIZE rcvd: 31

Nov 29 '22 11:11 vbnetadmin

I'm using dnsmasq on my mac, and my ansible role https://github.com/wayofdev/ansible-role-dnsmasq to setup dns on mac machines.

➜ cat /etc/resolver/docker nameserver 127.0.0.1

➜ pwd /opt/homebrew/etc

➜ cat dnsmasq.conf address=/docker/127.0.0.1 address=/mac/127.0.0.1

Nov 29 '22 11:11 vbnetadmin

@mortymacs

➜ dig router.docker @100.64.0.2

; <<>> DiG 9.10.6 <<>> router.docker @100.64.0.2 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11785 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION: ;router.docker. IN A

;; Query time: 35 msec ;; SERVER: 100.64.0.2#53(100.64.0.2) ;; WHEN: Tue Nov 29 13:02:07 EET 2022 ;; MSG SIZE rcvd: 31

So, as it says, it couldn't fine anything.

I'm using dnsmasq on my mac, and my ansible role https://github.com/wayofdev/ansible-role-dnsmasq to setup dns on mac machines.

➜ cat /etc/resolver/docker nameserver 127.0.0.1

➜ pwd /opt/homebrew/etc

➜ cat dnsmasq.conf address=/docker/127.0.0.1 address=/mac/127.0.0.1

Well, this issue is not related to Wait4X, but, in order to address your issue, we need to know how do you run Wait4X? by Docker or binary on your local machine?

Nov 29 '22 12:11 mortymacs

@mortymacs tried locally using binary and through docker (in separate image)

➜ which wait4x /usr/local/bin/wait4x

Quick result, using docker image:

docker run --rm --name='wait4x' \
    atkrad/wait4x:latest http https://router.docker -l debug
2022-11-29T12:13:32Z INF [HTTP] Checking the https://router.docker ...
2022-11-29T12:13:32Z DBG Get "https://router.docker": dial tcp 127.0.0.1:443: connect: connection refused
2022-11-29T12:13:33Z INF [HTTP] Checking the https://router.docker ...
2022-11-29T12:13:33Z DBG Get "https://router.docker": dial tcp 127.0.0.1:443: connect: connection refused
2022-11-29T12:13:35Z INF [HTTP] Checking the https://router.docker ...
2022-11-29T12:13:35Z DBG Get "https://router.docker": dial tcp 127.0.0.1:443: connect: connection refused
2022-11-29T12:13:36Z INF [HTTP] Checking the https://router.docker ...
2022-11-29T12:13:36Z DBG Get "https://router.docker": dial tcp 127.0.0.1:443: connect: connection refused

But, as you can see, using docker image for wait4x, it shows proper IP address

As a workaround, temporarily I can compile wait4x inside keycloak container (where I need to check for readiness)

Update:

Ok, this looks more promising, by adding flag --net=host I get this result:

➜ docker run --rm --net=host --name='wait4x'
atkrad/wait4x:latest http http://router.docker -l debug 2022-11-29T12:19:35Z INF [HTTP] Checking the http://router.docker ... 2022-11-29T12:19:35Z DBG Get "https://router.docker/": x509: certificate signed by unknown authority 2022-11-29T12:19:36Z INF [HTTP] Checking the http://router.docker ... 2022-11-29T12:19:36Z DBG Get "https://router.docker/": x509: certificate signed by unknown authority 2022-11-29T12:19:37Z INF [HTTP] Checking the http://router.docker ... 2022-11-29T12:19:37Z DBG Get "https://router.docker/": x509: certificate signed by unknown authority 2022-11-29T12:19:38Z INF [HTTP] Checking the http://router.docker ... 2022-11-29T12:19:38Z DBG Get "https://router.docker/": x509: certificate signed by unknown authority 2022-11-29T12:19:39Z INF [HTTP] Checking the http://router.docker ... 2022-11-29T12:19:39Z DBG Get "https://router.docker/": x509: certificate signed by unknown authority

Nov 29 '22 12:11 vbnetadmin

Temporary, working solution

docker-compose.yaml:

...
  keycloak:
    image: quay.io/keycloak/keycloak:20.0.1
    container_name: ${COMPOSE_PROJECT_NAME_SAFE}_keycloak
    restart: on-failure
    ports:
      - "8765:8080"
...

Makefile:

_kc-wait:
	$(DOCKER) run --rm --net=host --name='wait4x' atkrad/wait4x:latest \
		http $(shell docker inspect --format 'http://127.0.0.1:{{ (index (index .NetworkSettings.Ports "8080/tcp") 0).HostPort }}' wod_keycloak) \
		-l debug \
		-t 1m
.PHONY: _kc-wait

Result:

➜ make _kc-wait docker run --rm --net=host --name='wait4x' atkrad/wait4x:latest
http http://127.0.0.1:8765
-l debug
-t 1m 2022-11-29T12:59:19Z INF [HTTP] Checking the http://127.0.0.1:8765 ...

Btw, wait4x does not work, if container port of target isn't statically exposed:

    ports:
      - "8080"

➜ make _kc-wait docker run --rm --net=host --name='wait4x' atkrad/wait4x:latest
http http://127.0.0.1:64584
-l debug
-t 1m 2022-11-29T13:02:32Z INF [HTTP] Checking the http://127.0.0.1:64584 ... 2022-11-29T13:02:32Z DBG Get "http://127.0.0.1:64584": dial tcp 127.0.0.1:64584: connect: connection refused 2022-11-29T13:02:33Z INF [HTTP] Checking the http://127.0.0.1:64584 ... 2022-11-29T13:02:33Z DBG Get "http://127.0.0.1:64584": dial tcp 127.0.0.1:64584: connect: connection refused 2022-11-29T13:02:34Z INF [HTTP] Checking the http://127.0.0.1:64584 ... 2022-11-29T13:02:34Z DBG Get "http://127.0.0.1:64584": dial tcp 127.0.0.1:64584: connect: connection refused 2022-11-29T13:02:35Z INF [HTTP] Checking the http://127.0.0.1:64584 ... 2022-11-29T13:02:35Z DBG Get "http://127.0.0.1:64584": dial tcp 127.0.0.1:64584: connect: connection refused 2022-11-29T13:02:36Z INF [HTTP] Checking the http://127.0.0.1:64584 ... 2022-11-29T13:02:36Z DBG Get "http://127.0.0.1:64584": dial tcp 127.0.0.1:64584: connect: connection refused

docker compose ps NAME COMMAND SERVICE STATUS PORTS wod_keycloak "/opt/keycloak/bin/k…" keycloak running (healthy) 8443/tcp, 0.0.0.0:64584->8080/tcp

Nov 29 '22 13:11 vbnetadmin

@vbnetadmin I'm not sure if you understand networking and docker correctly (see https://docs.docker.com/network/). Your first approach using a docker container and the localhost address may never work per default (since each container usually has it's own virtual network adapter).

If instead of 127.0.0.1 you would use the actual address of the service/host you want to check (either the IP address or the compose service name) then wait4x would be able to find a route to the service and verify it.

Nov 30 '22 08:11 sbaeurle

@vbnetadmin, If you can't fix the problem, you can use the --insecure-skip-tls-verify flag to skip TLS verification.

Dec 06 '22 08:12 atkrad

wait4x wait4x copied to clipboard

Support self-signed TLS certiticate on HTTP component

wait4x
wait4x copied to clipboard