storage/sources/postgres: recurrent CI issues with postgres-cdc-tests due to SSL errors

[vmarcos]

What version of Materialize are you using?

v0.26.1-dev (ca392a37f)

How did you install Materialize?

Built from source

What is the issue?

Every so often, the Postgres CDC Tests fail in CI due to SSL issues. The relevant error is reported during a source creation in pg-cdc-ssl.td:

> CREATE SOURCE "mz_source" FROM POSTGRES CONNECTION pgconn PUBLICATION 'mz_source';
 
pg-cdc-ssl.td:163:1: error: executing query failed: db error: ERROR: error performing TLS handshake: error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus:crypto/rsa/rsa_ossl.c:553:, error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib:crypto/asn1/a_verify.c:170:, error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1919:: certificate signature failure: error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus:crypto/rsa/rsa_ossl.c:553:, error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib:crypto/asn1/a_verify.c:170:, error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1919:: certificate signature failure: error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus:crypto/rsa/rsa_ossl.c:553:, error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib:crypto/asn1/a_verify.c:170:, error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1919:: error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus:crypto/rsa/rsa_ossl.c:553:, error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib:crypto/asn1/a_verify.c:170:, error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1919:: ERROR: error performing TLS handshake: error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus:crypto/rsa/rsa_ossl.c:553:, error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib:crypto/asn1/a_verify.c:170:, error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1919:: certificate signature failure: error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus:crypto/rsa/rsa_ossl.c:553:, error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib:crypto/asn1/a_verify.c:170:, error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1919:: certificate signature failure: error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus:crypto/rsa/rsa_ossl.c:553:, error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib:crypto/asn1/a_verify.c:170:, error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1919:: error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus:crypto/rsa/rsa_ossl.c:553:, error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib:crypto/asn1/a_verify.c:170:, error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1919:
     |
 162 |   DATABASE postgres;
 163 | > CREATE SOURCE "mz_source"
     | ^ 

The error has been recurrent in CI and temporarily fixed by PRs that change files or image versions, supposedly to circumvent an OpenSSL bug related to expiry time-based behavior. Some relevant PRs and discussions include:

• https://github.com/MaterializeInc/materialize/pull/13765
• https://github.com/MaterializeInc/materialize/pull/13956
• https://materializeinc.slack.com/archives/CM7ATT65S/p1658378763034179?thread_ts=1658325277.305769&cid=CM7ATT65S
• https://materializeinc.slack.com/archives/CM7ATT65S/p1659468392732789?thread_ts=1659468257.686669&cid=CM7ATT65S
• https://materializeinc.slack.com/archives/CM7ATT65S/p1659526386362929

This issue is filed as a request to investigate and implement a long-term fix or workaround for this issue.

Relevant log output

No response

[JLDLaughlin]

yesterday this part of the test was disabled by @danhhz.

[JLDLaughlin]

cc @uce

[guswynn]

We havent seen this issue in a long time, as far as i can tell! @vmarcos let me know if you see it again!

[benesch]

Looks like we didn't see this for a while because we disabled the test. Oops! Reopening to track re-enabling the test.