genCA and genSignedCert might create invalid certificates

[dereulenspiegel]

With Helm 3 we are deploying a mutating webhook service in kubernetes. For that we need to generate a CA and a server certificate. We di this via the template functions genCA and genSignedCert from sprig. Usually this works pretty fine, but our last deployment generated the following CA cert and certificate: CA:

LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM5RENDQWR5Z0F3SUJBZ0lRTEdENHpCMXFxUnJuclg1Y3hxNzMvekFOQmdrcWhraUc5dzBCQVFzRkFEQVUKTVJJd0VBWURWUVFERXdsemNXeGlaV1V0WTJFd0hoY05NakV3TVRJMU1URXhNekUzV2hjTk16RXdNVEl6TVRFeApNekUzV2pBVU1SSXdFQVlEVlFRREV3bHpjV3hpWldVdFkyRXdnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCCkR3QXdnZ0VLQW9JQkFRRFRyanhYSlVUdjlYY0Z6MEZHaWw0UU8yRENwQW5EYjV3NDNhL0hOREttYloxaHhiUEsKbE03R1FPQnRIWGoreWk0UUJLa21mMFZ5SHl2elllekRGSjVVTUNMSVZYTEtsSklvTnFZekhHM2VsMFhLMnlaTApzUzFqd1FOWGVEeGp1OWNMcTFTL3VyRFFSaHQ4VkhlUGhLUmwxVGpLaGsrQk15NEhpdDZlR2l3ZW9HT0NVbmM4CjBhTlYvKzVPSmdBRmlFYUVVdEU2Y2svdmVaYmYyM2QyUjlRZVZpOGV6WGNCSDFMbUpldDUrallKSzhmbXBCbmQKQlVLTS9XbGhJRlZqbUlqTzZrRk5lMXJEbHNEMUxQb3pHMHFTeTg4MHlJRkNDN0czMjR3blgvem81ZXlwWkNpegpFUXlGaGhDQUUrYkRLV1FiY2svY2h0QzlZRG1rSGxLUnJyV1ZBZ01CQUFHalFqQkFNQTRHQTFVZER3RUIvd1FFCkF3SUNwREFkQmdOVkhTVUVGakFVQmdnckJnRUZCUWNEQVFZSUt3WUJCUVVIQXdJd0R3WURWUjBUQVFIL0JBVXcKQXdFQi96QU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUFSYlNpMUorNHMzT3k5dUFzZTR4OGxIZVh6T3N3bWt6dApRU3ZxclQzS3FzVmFndWFyc1BZeVhhQmt1WXBhQTI3Y3hQWGFtRThEQWNJUCtRL3c5WTc3dFM4Nmc0ZTJMckxkCmpkTWtHQjdvUC84eHgvL2kzR3FZaVQrcit0ZVZTTE81T3k4VTlqZFd4WHBNRmhIYjZmVVpUakw4WVNLRnFTVncKY2tlNGZ0c1Buc2lKUE1TYy9scWNRVHk1RFRxeisxZkhuZkZtc0MwRmd6czN4Sk8rQXM3TTIycExIMlROTkZHRwpnSGVud0pseEVUVlViZ3FxUTg5VWowOXRXR08wZ3F2R0w4MFhPeUU5SnBpbmNaZWh0MHYyQTZVYzhFRUpLcHR0CjU2Um1vSjJRcnVJb2JsWTJ3MzRQRW5HQXUwN0I0V2hSOS8xVWg2eXorZVdXZWdwRVVyL2RRQT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K

Certificate:

LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURURENDQWpTZ0F3SUJBZ0lSQU01TW5sQ0VVbzVjcnJvMkRSak81bDR3RFFZSktvWklodmNOQVFFTEJRQXcKRkRFU01CQUdBMVVFQXhNSmMzRnNZbVZsTFdOaE1CNFhEVEl4TURFeU5URXdOVEV6TVZvWERUTXhNREV5TXpFdwpOVEV6TVZvd0ZURVRNQkVHQTFVRUF4TUtjM0ZzWW1WbExYTjJZekNDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFECmdnRVBBRENDQVFvQ2dnRUJBTFRjNXc2Y1RBZDB5eTdTdFBHbTBEMnZ2R01DVGxEVEcxR0lYOSsyaVJpV3U0WlcKUHlJOVd3SUtVTkpQeFJxVmtwMzAyYUFKNStTRVNWbVJMZ00raHZnR3lBRmJZMUUxekJJdlRacEQ1Tm83eXlmVgpYZXhKV1FHeHIvc2p6all1VzRVVGg1bFdaY0tPdUpvVU4wekYySm93MUNXdjFlMEY0NEsySmhMVmozZkJrUm4xCm5pSFdIb0lwU1g3Z1loWU84ZVE1WCs3VXZsMjl4eDd2cWdwM0VaL3NON0xic0hNV3d0bDcwdFRET1o0aWRzZTAKbTlValpwZnBQcU5neGU4cUpNUlEyZXhJZUVVNmk3bU1xTDhaYi9sT2tkcTY1YXFwRE9Jb0t2eGNxTTNjdzcrTQpTSEF5SDk1SVJvbDVucGwrMVFtZStqNys5V3BhemVDRVNUMW00ZlVDQXdFQUFhT0JsekNCbERBT0JnTlZIUThCCkFmOEVCQU1DQmFBd0hRWURWUjBsQkJZd0ZBWUlLd1lCQlFVSEF3RUdDQ3NHQVFVRkJ3TUNNQXdHQTFVZEV3RUIKL3dRQ01BQXdWUVlEVlIwUkJFNHdUSUlpYzNGc1ltVmxMWE4yWXk1d2JHRjBabTl5YlMxcGJtWnlZWE4wY25WagpkSFZ5WllJbWMzRnNZbVZsTFhOMll5NXdiR0YwWm05eWJTMXBibVp5WVhOMGNuVmpkSFZ5WlM1emRtTXdEUVlKCktvWklodmNOQVFFTEJRQURnZ0VCQUNHb1dHNGJiaUZRU0tycFN5c08zaCtKdzBOUmJEMDR2TFV1K2EzN0JRbVIKU1hEMVVWSjRKL3lobU1FTG9xZzRwUmxkN2Rmb0ZrdEYrMXAwdEFEQVFpRU5VT3U1S05MaEhUQlB4NXlBdm9KQwpRbmd6dFBjQTY2MlhZT0RSVGI1RVk3Ykl1dWhoNDVYL01hNFBaL0h2cWxnYnIzckRORnozekZ4QUV5V2VMQVZWCkY0czhNVW93VXJycDVFNThyYVpaTUxSdmVUZ1Azdzc0TFAvbTlwUHJMdm9aTDRTbkxUdWVPNDhHUmRjaDhhV1EKQkl3TTR3RVlIZjBVbkFRc2s0NllpOXIzcTlUeitnaTVraCttRHBQWXV1Q1htZXVJYzZ6dUF5S0FGeWRUQ3NGKwp2WG84UGd1RVVWS2Z5a1ZjclMxWUxybmpsZGhmR3lpeTE3SWR4YlpsVzdzPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==

While on the first look these are completely valid certificates, the kubernetes api server was not able to validate the server certificate of our mutating webhook server. Further inspection via openssl verify returned the following error:

sqlbee.cert: CN = sqlbee-svc
error 7 at 0 depth lookup:certificate signature failure
4438066796:error:04FFF06A:rsa routines:CRYPTO_internal:block type is not 01:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-22.260.1/libressl-2.6/crypto/rsa/rsa_pk1.c:103:
4438066796:error:04FFF072:rsa routines:CRYPTO_internal:padding check failed:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-22.260.1/libressl-2.6/crypto/rsa/rsa_eay.c:680:
4438066796:error:0DFFF006:asn1 encoding routines:CRYPTO_internal:EVP lib:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-22.260.1/libressl-2.6/crypto/asn1/a_verify.c:155:

Further investigation revealed, that this is some kind of padding error within the certificate. Regenerating everything solved the issue, so my guess is, that genCA or genSignedCert generate random values (of course they are randomly generating keys) which might need some special form of padding in certain cases, but this isn't handled. So actually using these certificates and CA is not possible. Since we never encountered this problem before and I couldn't find a previous issue describing this problem the problem might be very rare. Nonetheless it was not easy to find the root cause for our failing mutating webhook server, so we think it might be worth investigating this. Unfortunately we can't provide a solution right now, as we are not really familiar enough with this project. In any case we will provide every information necessary (and not mentioned here) to help in solving this issue. Thanks for this awesome project 👍