oracle
PEM data was not found in buffer with valid RSA key in v1.31.0
Description
The OCI Cloud Controller Manager (CCM) v1.31.0 fails to initialize with "PEM data was not found in buffer" despite a valid RSA private key being provided at the specified path. The key is readable by the CCM process (confirmed via pre-start debug), works with the OCI CLI, and matches the fingerprint in the config. Tried with standard /etc/oci in-container mount and custom /oci-config mount. Tried regular PEM key and converted RSA key, always the same error from the cloud controller container. Tried using multiple different versions of CCM.
Environment
- CCM Version:
ghcr.io/oracle/cloud-provider-oci:v1.31.0 - Kubernetes: v1.31.0
- OCI Region: us-ashburn-1
- Date: March 21, 2025
Steps to Reproduce
- Generate an RSA key:
bash
openssl rsa -in /root/.oci/richard.tylka.clean.pem -out /root/.oci/richard.tylka.rsa.pem -traditional
kubectl create secret generic oci-credential
--from-file=cloud-provider.yaml
--from-file=key.pem=/root/.oci/richard.tylka.rsa.pem
-n kube-system
cloud-provider.yaml:
auth: region: us-ashburn-1 tenancy: ocid1.tenancy.oc1..aaaaaaaa6bpzdsduwbti65kqm7gvubwvrspi7626cxona2t6fgguxn3ftvxa user: ocid1.user.oc1..aaaaaaaaahcwxw3yxfayfxm666prjwi4667zhvnh4ddz5sdzclar3tjpww5q key: /oci-config/key.pem fingerprint: 76:75:60:81:15:20:95:b6:3a:cb:29:c6:38:ad:83:76 compartment: ocid1.compartment.oc1..aaaaaaaameevqllpkruwsepv2ps3otacm7vgx257uoi2o7sgeischov64hja vcn: ocid1.vcn.oc1.iad.amaaaaaausti56qaepofn65jytrp37n5o4jopfbvxnquomr2dt46khs24s4a
oci-ccm.yaml:
oci-ccm.txt kubectl logs -n kube-system -l k8s-app=oci-cloud-controller-manager -c oci-cloud-controller-manager --tail=-1 total 4 -rw-r--r-- 1 root root 1675 Mar 21 18:45 key.pem -----BEGIN RSA PRIVATE KEY----- MIIEogIBAAKCAQEAtu3ghDp8/LUR6Jc2GNHsDvcFRUgjB72g0yNG9HmPxas785Bu F6DD5p9WjtpyLuwh2/MH87eDPtJzdmVGX2huI9BS130G4f4e0sGqBGIT0XMhlL+a 9zUAANhZavRWOY6+9qNKdB+MhPMsw5W3Pqc1g4MOjlRBlYVUa60lHZk6OEyi2SqI lUziYbwBBSQUH7xWkk6ObbH+kdSEabaGhBUvmQ6X+2cYdJAZyrF23L1vAHjmS26H ...truncated... JU6F8NeWiPiqUQ/kAO8FmG/3gYCIA6WDy0WmyOj7OvG2cs9yKw2hzu7lHw9sk+4D 6qvRAoGAG0TXp5Ub+gpzXx1wQj6pvb8IQKyazROtXst4M0bsqYmrJgw/fwUNoNh9 6Ks0N2W5gSLme5C8Irxi1mZPwntd5/jbwpYdbDCUA7svevM3CclHf5xLQwJtdg+i bruvA3e2A1Gw8fp29l9OCrIwNsIUQPt4yfRryWpX/VjBYJlzZ40= -----END RSA PRIVATE KEY----- 2025-03-21T18:56:23.336Z INFO oci-cloud-controller-manager/main.go:69 oci-cloud-controller-manager {"version": "411bfeb2", "build": "411bfeb2"} I0321 18:56:24.870569 1 serving.go:386] Generated self-signed cert in-memory I0321 18:56:25.538715 1 serving.go:386] Generated self-signed cert in-memory W0321 18:56:25.538742 1 client_config.go:659] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0321 18:56:26.555771 1 requestheader_controller.go:247] Loaded a new request header values for RequestHeaderAuthRequestController 2025-03-21T18:56:26.560Z INFO metadata/instance_metadata.go:76 Metadata endpoint /opc/v2/instance/ returned response successfully 2025-03-21T18:56:26.571Z INFO metadata/instance_metadata.go:76 Metadata endpoint /opc/v2/instance/ returned response successfully 2025-03-21T18:56:26.571Z INFO client/utils.go:95 OCI using read rate limit configuration: QPS=20, bucket=5 {"component": "cloud-controller-manager"} 2025-03-21T18:56:26.571Z INFO client/utils.go:99 OCI using write rate limit configuration: QPS=20, bucket=5 {"component": "cloud-controller-manager"} F0321 18:56:26.571981 1 main.go:82] Cloud provider could not be initialized: could not init cloud provider "oci": NewComputeClientWithConfigurationProvider: can not create client, bad configuration: PEM data was not found in buffer
Additional Info Key works with OCI CLI using the same fingerprint.
Tested with PKCS#8 and RSA formats, symlinks and real files, various paths (/etc/oci/key.pem, /oci-config/key.pem).
Debug container confirms key presence.
Full logs available upon request.
Suspected Cause Bug in NewComputeClientWithConfigurationProvider failing to parse a valid RSA key.
