[BUG] InvalidProxyConfig

[jhcook]

General information

• OS: macOS
• Hypervisor: vfkit
• Did you run crc setup before starting it (Yes/No)? Yes
• Running CRC on: Laptop and Desktop

CRC version

CRC version: 2.7.1+8b30b973
OpenShift version: 4.11.0
Podman version: 4.1.1

CRC status

DEBU CRC version: 2.7.1+8b30b973                  
DEBU OpenShift version: 4.11.0                    
DEBU Podman version: 4.1.1                        
DEBU Running 'crc status'                         
DEBU Checking file: /Users/jcook/.crc/machines/crc/.crc-exist 
DEBU Checking file: /Users/jcook/.crc/machines/crc/.crc-exist 
DEBU Running SSH command: df -B1 --output=size,used,target /sysroot | tail -1 
DEBU Using ssh private keys: [/Users/jcook/.crc/machines/crc/id_ecdsa /Users/jcook/.crc/cache/crc_vfkit_4.11.0_amd64/id_ecdsa_crc] 
DEBU SSH command results: err: ssh: handshake failed: read tcp 127.0.0.1:49301->127.0.0.1:2222: read: connection reset by peer, output:  
DEBU Cannot get root partition usage: ssh command error:
command : df -B1 --output=size,used,target /sysroot | tail -1
err     : ssh: handshake failed: read tcp 127.0.0.1:49301->127.0.0.1:2222: read: connection reset by peer 
DEBU cannot get OpenShift status: Get "https://api.crc.testing:6443/apis/config.openshift.io/v1/clusteroperators": read tcp 127.0.0.1:49361->127.0.0.1:6443: read: connection reset by peer 
CRC VM:          Running
OpenShift:       Unreachable (v4.11.0)
Podman:          
Disk Usage:      0B of 0B (Inside the CRC VM)
Cache Usage:     37.14GB
Cache Directory: /Users/jcook/.crc/cache

CRC config

- consent-telemetry                     : no
- cpus                                  : 6
- disk-size                             : 100
- enable-cluster-monitoring             : true
- http-proxy                            : http://192.168.123.13:3128
- https-proxy                           : http://192.168.123.13:3128
- memory                                : 30208
- no-proxy                              : *.local,*.test,*.testing,.amazonaws.com,vagrantcloud-files-production.s3-accelerate.amazonaws.com,.vagrantcloud.com,192.168.0.0/16,10.0.0.0/8,172.16.0.0/12
- proxy-ca-file                         : /Users/jcook/repo/kubernetes-dev-env/cert.pem

Host Operating System

ProductName:    macOS
ProductVersion: 12.5.1
BuildVersion:   21G83

Steps to reproduce

1. crc start

Expected

A big fat happy OpenShift Local instance which I had until the recent update to 2.5.1.

Actual

INFO Starting openshift instance... [waiting for the cluster to stabilize] INFO 3 operators are progressing: image-registry, network, openshift-controller-manager INFO 3 operators are progressing: image-registry, network, openshift-controller-manager INFO 3 operators are progressing: image-registry, network, openshift-controller-manager INFO 3 operators are progressing: image-registry, network, openshift-controller-manager INFO 3 operators are progressing: image-registry, network, openshift-controller-manager INFO 3 operators are progressing: image-registry, network, openshift-controller-manager INFO 3 operators are progressing: image-registry, network, openshift-controller-manager INFO 3 operators are progressing: image-registry, network, openshift-controller-manager INFO 3 operators are progressing: image-registry, network, openshift-controller-manager INFO 2 operators are progressing: image-registry, openshift-controller-manager INFO 3 operators are progressing: authentication, image-registry, openshift-controller-manager INFO 3 operators are progressing: authentication, image-registry, openshift-controller-manager INFO 3 operators are progressing: authentication, image-registry, openshift-controller-manager INFO 4 operators are progressing: authentication, console, image-registry, openshift-controller-manager ERRO Cluster is not ready: cluster operators are still not stable after 18m3.483863048s INFO Waiting for the proxy configuration to be applied... INFO 2 operators are progressing: kube-apiserver, openshift-controller-manager INFO Adding crc-admin and crc-developer contexts to kubeconfig... ERRO Cannot update kubeconfig: read tcp 127.0.0.1:64054->127.0.0.1:6443: read: connection reset by peer

Logs

time="2022-08-30T12:17:54+01:00" level=debug msg="network operator is degraded, Reason: InvalidProxyConfig"

Before gather the logs try following if that fix your issue

$ crc delete -f
$ crc cleanup
$ crc setup
$ crc start --log-level debug

Please consider posting the output of crc start --log-level debug on http://gist.github.com/ and post the link in the issue.

[cfergeau]

*.local,*.test,*.testing,.amazonaws.com,vagrantcloud-files-production.s3-accelerate.amazonaws.com,.vagrantcloud.com,192.168.0.0/16,10.0.0.0/8,172.16.0.0/12

I'm not sure about this no-proxy string, could you try with just *.local,*.test,*.testing,.amazonaws.com,vagrantcloud-files-production.s3-accelerate.amazonaws.com,.vagrantcloud.com and see if it's better?

[jhcook]

I have and same outcome. The problem seems to be the cert.pem. It was working fine, then I updated to 2.5.1 and mayhem. When removed, it works better although it doesn't work completely because I need the proxy cert.

$ openssl x509 -inform DER -in cert.cer -outform PEM -out cert.pem

[cfergeau]

You need the proxy cert because the proxy is reencrypting https communication using the CA from this cert?

[jhcook]

You need the proxy cert because the proxy is reencrypting https communication using the CA from this cert?

Correct

[praveenkumar]

@jhcook Can you provide following details (do mask sensitive info if there)?

$ oc get proxy -oyaml
$ oc get cm user-ca-bundle -n openshift-config

Edit: Can you also share the debug logs?

[jhcook]

@jhcook Can you provide following details (do mask sensitive info if there)?

$ oc get proxy -oyaml
$ oc get cm user-ca-bundle -n openshift-config

It never complete so I am unable to provide that info.

Edit: Can you also share the debug logs?

Attached debug.log

Debug log ends with INFO Waiting for the proxy configuration to be applied....

[praveenkumar]

@jhcook Are you sure that - proxy-ca-file: /Users/jcook/repo/kubernetes-dev-env/cert.pem is valid one and not corrupted because that is another reason to have https://bugzilla.redhat.com/show_bug.cgi?id=1844736 InvalidProxyConfig. Please verify it once.

[jhcook]

@praveenkumar this file was working with 2.5.0 and stopped working immediately after upgrade to 2.5.1

$ file cert.pem
cert.pem: PEM certificate

The file is perfectly fine.

[praveenkumar]

@jhcook Another guess is https://go.dev/doc/go1.18#sha1 ( suggested by @cfergeau ) because I just tired 4.11.0 bundle with proxy and it did pass without any InvalidProxyConfig error.

[jhcook]

$ openssl x509 -noout -text -in cert.pem | grep 'Signature Algorithm'
    Signature Algorithm: sha256WithRSAEncryption
    Signature Algorithm: sha256WithRSAEncryption

[praveenkumar]

@jhcook Thanks for all the details, I will try that on mac also ( I tired on linux) and update.

[praveenkumar]

It never complete so I am unable to provide that info.

@jhcook even it doesn't complete can you ssh to the VM using https://github.com/code-ready/crc/wiki/Debugging-guide and then try to use /opt/kubeconfig to get that info?

<crc_vm>$ oc --kubecconfig /opt/kubeconfig get proxy -oyaml
<crc_vm>$ oc --kubecconfig /opt/kubeconfig get cm user-ca-bundle -n openshift-config
<crc_vm>$ oc --kubecconfig /opt/kubeconfig  get  co

[jhcook]

@praveenkumar I have yet to get your info, but I have backed off the proxy cert an injected it and updated trusted ca bundle. That works, but alas the network operator bombs out and SSH connectivity to the instance is lost.

I will sleep on it and concentrate on the Kubernetes objects you point out and the guidance here.

You can see my code here.

[cfergeau]

If you are allowed to share your cert.pem with us, we could try to reproduce your issues locally.

[praveenkumar]

@jhcook Thanks for the issue, I hope it is fixed for you since we put some fixes on proxy side on latest version of crc and if not please create new one.

/close

[openshift-ci[bot]]

@praveenkumar: Closing this issue.

In response to this:

@jhcook Thanks for the issue, I hope it is fixed for you since we put some fixes on proxy side on latest version of crc and if not please create new one.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.