brupop-apiserver liveness/readiness probe failed connection refused

[guohaolee]

I'm currently using the operator version v0.2.2.

However, the pod failed to startup returning the following error:

Warning Unhealthy 5s (x2 over 35s) kubelet Liveness probe failed: Get "https://100.64.165.98:8443/ping": dial tcp 100.64.165.98:8443: connect: connection refused Warning Unhealthy 5s (x3 over 35s) kubelet Readiness probe failed: Get "https://100.64.165.98:8443/ping": dial tcp 100.64.165.98:8443: connect: connection refused

I have cert manager running in cert-manager namespace.

I'm deploying this on AWS EKS 1.23

Is there something I missed that needs to be whitelisted?

Image I'm using: v0.2.2

[somnusfish]

Hi guohaolee@,

Thanks for reaching out. Would you mind provide more information here?

1. What yaml file did you use to deploy bottlerocket update operator 0.2.2
2. How did you deploy bottlerocket update operator 0.2.2, did you directly upgrade from 0.2.1 or did you deploy from scratch?
3. Do you have all issuer and secret generator? You can check by running

➜  ~ kubectl get clusterissuer
NAME                READY   AGE
selfsigned-issuer   True    6d20h  <------- You should see this cluster issuer

➜  ~ kubectl get issuer -n brupop-bottlerocket-aws
NAME           READY   AGE
my-ca-issuer   True    21h   <------------ You should see this issuer

➜  ~ kubectl get secret -n brupop-bottlerocket-aws
NAME                                            TYPE                                  DATA   AGE
brupop-tls                                      kubernetes.io/tls                     3      21h  <---- 

4. Is your self-signed certificate bootstrapped? You can check that by running:

# Get the secret:
➜  ~ kubectl get secret brupop-tls -o yaml
# Decode the tls.crt
echo $tls.crt_value | base64 --decode > tls.crt
# Check the cert
➜  ~ openssl x509 -in tls.crt -text -noout

The Issuer should be the same as Subject, and X509v3 extensions: should show CA:TRUE 5. What do you get by running kubectl describe pod $your-brupop-apiserver? 6. What do you get by running kubectl logs pod $your-brupop-apiserver?

[guohaolee]

Hi @somnusfish. apologies for the late reply and thank you for attending my issue.

1. This is the yaml I pull and installed from: link 2. This is a fresh install in the cluster. So I installed it directly 3. Here are the issuer and secret generator

NAME                READY   AGE
selfsigned-issuer   True    2m4s

kubectl get issuer -n brupop-bottlerocket-aws
NAME           READY   AGE
my-ca-issuer   True    2m20s

kubectl get secret -n brupop-bottlerocket-aws
NAME                                            TYPE                                  DATA   AGE
brupop-agent-service-account-token-pnbb4        kubernetes.io/service-account-token   3      2m42s
brupop-apiserver-service-account-token-7ml9p    kubernetes.io/service-account-token   3      2m42s
brupop-controller-service-account-token-7cnvg   kubernetes.io/service-account-token   3      2m41s
brupop-tls                                      kubernetes.io/tls                     3      2m27s
default-token-ptjk8                             kubernetes.io/service-account-token   3      2m44s

4. Here's the field from the tls

Issuer: CN=my-selfsigned-ca
Subject: CN=my-selfsigned-ca
X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Subject Key Identifier:
                EF:9D:38:37:A4:4E:36:F9:63:DB:E3:91:EF:B5:BA:09:64:76:63:6D
            X509v3 Subject Alternative Name:
                DNS:brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, DNS:brupop-apiserver.brupop-bottlerocket-aws.svc

5. Here are the output of events from the kubectl describe

Events:
  Type     Reason       Age                From               Message
  ----     ------       ----               ----               -------
  Normal   Scheduled    11m                default-scheduler  Successfully assigned brupop-bottlerocket-aws/brupop-apiserver-68c5d7764f-pz9mg to ip-10-9-101-250.us-west-2.compute.internal
  Warning  FailedMount  11m (x4 over 11m)  kubelet            MountVolume.SetUp failed for volume "bottlerocket-tls-keys" : secret "brupop-tls" not found
  Normal   Pulling      11m                kubelet            Pulling image "public.ecr.aws/bottlerocket/bottlerocket-update-operator:v0.2.1"
  Normal   Pulled       11m                kubelet            Successfully pulled image "public.ecr.aws/bottlerocket/bottlerocket-update-operator:v0.2.1" in 1.509719539s
  Normal   Created      11m (x2 over 11m)  kubelet            Created container brupop
  Normal   Started      11m (x2 over 11m)  kubelet            Started container brupop
  Normal   Pulled       11m                kubelet            Container image "public.ecr.aws/bottlerocket/bottlerocket-update-operator:v0.2.1" already present on machine
  Warning  Unhealthy    10m (x6 over 11m)  kubelet            Readiness probe failed: Get "https://10.9.106.72:8443/ping": dial tcp 10.9.106.72:8443: connect: connection refused
  Warning  Unhealthy    10m (x6 over 11m)  kubelet            Liveness probe failed: Get "https://10.9.106.72:8443/ping": dial tcp 10.9.106.72:8443: connect: connection refused
  Normal   Killing      10m (x2 over 11m)  kubelet            Container brupop failed liveness probe, will be restarted
  Warning  BackOff      95s (x32 over 9m)  kubelet            Back-off restarting failed container

6. Here are the output of kubectl logs

{"v":0,"name":"apiserver","msg":"[APPLY_DEBUG_OVERRIDES - START]","level":40,"hostname":"brupop-apiserver-68c5d7764f-pz9mg","pid":1,"time":"2022-08-18T20:02:07.111156755+00:00","target":"kube_client::config","line":285,"file":"/src/.cargo/registry/src/github.com-1ecc6299db9ec823/kube-client-0.71.0/src/config/mod.rs","self":"Config { cluster_url: https://kubernetes.default.svc/, default_namespace: \"brupop-bottlerocket-aws\", root_cert: Some([[48, 130, 2, 254, 48, 130, 1, 230, 160, 3, 2, 1, 2, 2, 1, 0, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 21, 49, 19, 48, 17, 6, 3, 85, 4, 3, 19, 10, 107, 117, 98, 101, 114, 110, 101, 116, 101, 115, 48, 30, 23, 13, 50, 50, 48, 56, 49, 55, 50, 51, 50, 54, 51, 51, 90, 23, 13, 51, 50, 48, 56, 49, 52, 50, 51, 50, 54, 51, 51, 90, 48, 21, 49, 19, 48, 17, 6, 3, 85, 4, 3, 19, 10, 107, 117, 98, 101, 114, 110, 101, 116, 101, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 209, 168, 45, 26, 62, 158, 41, 243, 73, 39, 99, 226, 187, 29, 177, 1, 124, 20, 207, 222, 105, 225, 159, 247, 121, 61, 202, 210, 243, 24, 204, 229, 26, 90, 14, 54, 45, 249, 183, 106, 32, 186, 97, 245, 206, 49, 65, 5, 95, 25, 173, 120, 178, 229, 233, 11, 68, 138, 4, 79, 129, 138, 176, 195, 250, 212, 64, 107, 40, 18, 68, 5, 231, 103, 44, 25, 68, 249, 167, 158, 99, 93, 205, 177, 207, 23, 136, 251, 156, 215, 28, 233, 163, 223, 197, 224, 115, 203, 120, 172, 158, 142, 63, 231, 60, 189, 131, 219, 75, 40, 152, 164, 140, 184, 128, 173, 1, 186, 194, 64, 254, 111, 195, 205, 203, 21, 60, 107, 121, 233, 24, 200, 76, 22, 220, 182, 177, 245, 12, 220, 253, 133, 13, 161, 124, 177, 170, 21, 238, 38, 96, 217, 179, 20, 119, 0, 97, 6, 18, 200, 214, 208, 247, 176, 57, 61, 99, 115, 41, 179, 197, 221, 245, 39, 87, 87, 1, 191, 144, 81, 122, 27, 167, 218, 112, 7, 35, 178, 253, 35, 141, 243, 34, 227, 3, 6, 156, 110, 96, 194, 115, 141, 250, 176, 223, 211, 147, 216, 56, 133, 188, 134, 191, 68, 89, 51, 60, 52, 94, 78, 74, 42, 178, 84, 233, 23, 113, 174, 159, 209, 154, 95, 226, 156, 149, 51, 4, 124, 178, 100, 46, 227, 230, 156, 160, 210, 232, 124, 75, 181, 62, 16, 202, 9, 5, 239, 2, 3, 1, 0, 1, 163, 89, 48, 87, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 2, 164, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 207, 203, 51, 118, 153, 154, 156, 72, 174, 133, 5, 160, 145, 137, 189, 182, 221, 198, 109, 82, 48, 21, 6, 3, 85, 29, 17, 4, 14, 48, 12, 130, 10, 107, 117, 98, 101, 114, 110, 101, 116, 101, 115, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 189, 243, 98, 5, 88, 244, 31, 83, 172, 183, 43, 145, 173, 137, 62, 44, 166, 184, 63, 182, 247, 167, 102, 242, 137, 26, 123, 90, 70, 185, 10, 34, 185, 134, 92, 36, 47, 219, 62, 113, 100, 198, 86, 65, 91, 143, 10, 69, 208, 113, 147, 47, 238, 124, 36, 116, 115, 174, 189, 41, 206, 155, 34, 187, 28, 169, 162, 159, 242, 172, 200, 28, 65, 43, 10, 13, 130, 183, 140, 0, 28, 21, 169, 111, 47, 221, 105, 78, 52, 129, 164, 8, 229, 39, 117, 127, 111, 99, 155, 129, 200, 87, 31, 53, 9, 55, 234, 115, 61, 67, 115, 39, 153, 17, 203, 50, 90, 92, 220, 101, 56, 220, 89, 185, 136, 17, 88, 164, 222, 42, 2, 208, 45, 15, 160, 129, 185, 54, 145, 205, 125, 199, 174, 83, 15, 1, 246, 240, 126, 70, 3, 115, 226, 155, 179, 189, 238, 8, 160, 219, 222, 197, 86, 165, 9, 163, 87, 24, 43, 187, 132, 65, 102, 113, 117, 5, 45, 52, 12, 80, 239, 153, 114, 192, 58, 244, 239, 117, 71, 60, 35, 213, 100, 213, 22, 175, 196, 247, 121, 91, 133, 243, 162, 33, 156, 185, 107, 226, 182, 22, 170, 152, 119, 130, 227, 167, 189, 2, 176, 86, 231, 29, 203, 37, 107, 224, 42, 177, 192, 16, 218, 39, 43, 130, 66, 186, 197, 84, 129, 251, 50, 68, 231, 143, 16, 92, 239, 254, 143, 239, 102, 76, 250, 157, 196, 36]]), timeout: Some(295s), accept_invalid_certs: false, auth_info: AuthInfo { username: None, password: None, token: None, token_file: Some(\"/var/run/secrets/kubernetes.io/serviceaccount/token\"), client_certificate: None, client_certificate_data: None, client_key: None, client_key_data: None, impersonate: None, impersonate_groups: None, auth_provider: None, exec: None }, proxy_url: None }"}
{"v":0,"name":"apiserver","msg":"[APPLY_DEBUG_OVERRIDES - END]","level":40,"hostname":"brupop-apiserver-68c5d7764f-pz9mg","pid":1,"time":"2022-08-18T20:02:07.111218347+00:00","target":"kube_client::config","line":285,"file":"/src/.cargo/registry/src/github.com-1ecc6299db9ec823/kube-client-0.71.0/src/config/mod.rs","elapsed_milliseconds":0,"self":"Config { cluster_url: https://kubernetes.default.svc/, default_namespace: \"brupop-bottlerocket-aws\", root_cert: Some([[48, 130, 2, 254, 48, 130, 1, 230, 160, 3, 2, 1, 2, 2, 1, 0, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 21, 49, 19, 48, 17, 6, 3, 85, 4, 3, 19, 10, 107, 117, 98, 101, 114, 110, 101, 116, 101, 115, 48, 30, 23, 13, 50, 50, 48, 56, 49, 55, 50, 51, 50, 54, 51, 51, 90, 23, 13, 51, 50, 48, 56, 49, 52, 50, 51, 50, 54, 51, 51, 90, 48, 21, 49, 19, 48, 17, 6, 3, 85, 4, 3, 19, 10, 107, 117, 98, 101, 114, 110, 101, 116, 101, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 209, 168, 45, 26, 62, 158, 41, 243, 73, 39, 99, 226, 187, 29, 177, 1, 124, 20, 207, 222, 105, 225, 159, 247, 121, 61, 202, 210, 243, 24, 204, 229, 26, 90, 14, 54, 45, 249, 183, 106, 32, 186, 97, 245, 206, 49, 65, 5, 95, 25, 173, 120, 178, 229, 233, 11, 68, 138, 4, 79, 129, 138, 176, 195, 250, 212, 64, 107, 40, 18, 68, 5, 231, 103, 44, 25, 68, 249, 167, 158, 99, 93, 205, 177, 207, 23, 136, 251, 156, 215, 28, 233, 163, 223, 197, 224, 115, 203, 120, 172, 158, 142, 63, 231, 60, 189, 131, 219, 75, 40, 152, 164, 140, 184, 128, 173, 1, 186, 194, 64, 254, 111, 195, 205, 203, 21, 60, 107, 121, 233, 24, 200, 76, 22, 220, 182, 177, 245, 12, 220, 253, 133, 13, 161, 124, 177, 170, 21, 238, 38, 96, 217, 179, 20, 119, 0, 97, 6, 18, 200, 214, 208, 247, 176, 57, 61, 99, 115, 41, 179, 197, 221, 245, 39, 87, 87, 1, 191, 144, 81, 122, 27, 167, 218, 112, 7, 35, 178, 253, 35, 141, 243, 34, 227, 3, 6, 156, 110, 96, 194, 115, 141, 250, 176, 223, 211, 147, 216, 56, 133, 188, 134, 191, 68, 89, 51, 60, 52, 94, 78, 74, 42, 178, 84, 233, 23, 113, 174, 159, 209, 154, 95, 226, 156, 149, 51, 4, 124, 178, 100, 46, 227, 230, 156, 160, 210, 232, 124, 75, 181, 62, 16, 202, 9, 5, 239, 2, 3, 1, 0, 1, 163, 89, 48, 87, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 2, 164, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 207, 203, 51, 118, 153, 154, 156, 72, 174, 133, 5, 160, 145, 137, 189, 182, 221, 198, 109, 82, 48, 21, 6, 3, 85, 29, 17, 4, 14, 48, 12, 130, 10, 107, 117, 98, 101, 114, 110, 101, 116, 101, 115, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 189, 243, 98, 5, 88, 244, 31, 83, 172, 183, 43, 145, 173, 137, 62, 44, 166, 184, 63, 182, 247, 167, 102, 242, 137, 26, 123, 90, 70, 185, 10, 34, 185, 134, 92, 36, 47, 219, 62, 113, 100, 198, 86, 65, 91, 143, 10, 69, 208, 113, 147, 47, 238, 124, 36, 116, 115, 174, 189, 41, 206, 155, 34, 187, 28, 169, 162, 159, 242, 172, 200, 28, 65, 43, 10, 13, 130, 183, 140, 0, 28, 21, 169, 111, 47, 221, 105, 78, 52, 129, 164, 8, 229, 39, 117, 127, 111, 99, 155, 129, 200, 87, 31, 53, 9, 55, 234, 115, 61, 67, 115, 39, 153, 17, 203, 50, 90, 92, 220, 101, 56, 220, 89, 185, 136, 17, 88, 164, 222, 42, 2, 208, 45, 15, 160, 129, 185, 54, 145, 205, 125, 199, 174, 83, 15, 1, 246, 240, 126, 70, 3, 115, 226, 155, 179, 189, 238, 8, 160, 219, 222, 197, 86, 165, 9, 163, 87, 24, 43, 187, 132, 65, 102, 113, 117, 5, 45, 52, 12, 80, 239, 153, 114, 192, 58, 244, 239, 117, 71, 60, 35, 213, 100, 213, 22, 175, 196, 247, 121, 91, 133, 243, 162, 33, 156, 185, 107, 226, 182, 22, 170, 152, 119, 130, 227, 167, 189, 2, 176, 86, 231, 29, 203, 37, 107, 224, 42, 177, 192, 16, 218, 39, 43, 130, 66, 186, 197, 84, 129, 251, 50, 68, 231, 143, 16, 92, 239, 254, 143, 239, 102, 76, 250, 157, 196, 36]]), timeout: Some(295s), accept_invalid_certs: false, auth_info: AuthInfo { username: None, password: None, token: None, token_file: Some(\"/var/run/secrets/kubernetes.io/serviceaccount/token\"), client_certificate: None, client_certificate_data: None, client_key: None, client_key_data: None, impersonate: None, impersonate_groups: None, auth_provider: None, exec: None }, proxy_url: None }"}
{"v":0,"name":"apiserver","msg":"server exited","level":50,"hostname":"brupop-apiserver-68c5d7764f-pz9mg","pid":1,"time":"2022-08-18T20:02:35.339467124+00:00","target":"apiserver::api","line":190,"file":"apiserver/src/api/mod.rs"}

[somnusfish]

Hi @guohaolee ,

Thanks for providing these information. The error was likely due to the link file was still use 0.2.1 image which didn't have the SSL support. I have opened a pull request to fix the link.

You can use the new yaml file for deployment.

Please let me know if you still facing any issue after using the new file.

[guohaolee]

Hi @somnusfish

Thanks for the update!. It works now running on image 0.2.2. Appreciate your help

Also another request on here if its not too hard, can you also add the yaml file as an asset when releasing the tag? So we can download from the tag version? Something like https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases/tag/v2.4.3 where the yaml file is part of the asset in the tag.

[gthao313]

Hi @guohaolee Thanks for your suggestion. We'll support that.

[MiLk]

Hi,

Would it be possible to make a new release with the updated manifest?

I tried to update to 0.2.2 with the tag, and it took me a while to figure out this was the issue.

It would make more sense to have the latest tag working. I'm not sure what has been done in https://github.com/bottlerocket-os/bottlerocket-update-operator/pull/239 is what most people expect.

Thank you.

[jpmcb]

@gthao313 and I have updated the v0.2.2 tag to point to the right images (sorry for the delay on this! Had to do abit of git surgery to update an inplace tag)

I've also added the generated yaml as a static file in our release artifacts. Going forward, we'll continue to do this.

[MiLk]

Thank you!