SSL CA error when creating an AwsKmsClient from AWS Lambda in python

[rsivek]

Deployment

Tink is being deployed into an AWS Lambda function with the python3.8 runtime. Tink and dependencies are pre-downloaded and packaged with the lambda source:

• tink-1.5.0
• protobuf-3.17.1
• absl_py-0.12.0
• six-1.16.0

Describe the bug

When attempting to create an AwsKmsClient, I receive an error indicating a problem with the CA certificate.

Relevant line:

kms_client = awskms.AwsKmsClient(kms_master_uri, '')

Error:

INVALID_ARGUMENT: AWS KMS decryption failed with error: AWS error code: 99, : curlCode: 77, Problem with the SSL CA cert (path? access rights?) with address : <<IP_ADDRESS>> with address : <<IP_ADDRESS>>

This only occurs when running in a lambda environment. Code works perfectly fine on my local machine with dependencies installed with pip (Mac).

To Reproduce

Run a python script in AWS lambda (or using the lambci/lambda Docker image) and attempt to create an AwsKmsClient with a valid KMS URI.

Expected behavior

AWS client connects to KMS successfully.

Attempted Fixes

Based on what I've read, it seems the underlying libcurl library is not checking the proper location for CA certs. The aws-lambda-cpp repo suggests setting the caFile config but it doesn't look like I have any way to set this from python.

I've attempted setting the SSL_CERT_FILE, SSL_CERT_DIR, CURL_CA_BUNDLE, and REQUESTS_CA_BUNDLE environment variables to the suggested path (which I've verified is a valid cert bundle file in the environment), but none of these have been successful.

[rsivek]

Packaged dependencies were downloaded directly from Pypi from the following locations:

• tink-1.5.0-cp38-cp38-manylinux2014_x86_64.whl
• protobuf-3.17.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl
• absl_py-0.12.0-py3-none-any.whl
• six-1.16.0-py2.py3-none-any.whl

[rsivek]

Just tried running this same setup on a CentOS instance and received the same error... so maybe it's not Lambda specific

[rsivek]

I also see the same error with tink installed via pip into a virtualenv on CentOS

[rsivek]

I've verified that the problem is due to the library looking for CA certificates at /etc/ssl/certs/ca-certificates.crt, whereas in some environments (like CentOS and AWS Lambda) that file does not exist (it's instead located at /etc/ssl/certs/ca-bundle.crt).

When I symlink ca-certificates.crt to ca-bundle.crt it works perfectly on the CentOS instance. However, I am unsure if that solution is feasible in a Lambda environment.

[thaidn]

Thanks for the report.

It seems that it's caused by this line: https://github.com/google/tink/blob/master/cc/third_party/curl.BUILD.bazel#L413. I'm not sure how to fix this. @tholenst, @kste thoughts?

I found a workaround. If you can set an environment variable, try:

export CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

[rsivek]

@thaidn thanks for your response. Unfortunately setting the CURL_CA_BUNDLE environment bundle does not seem to have any effect, as mentioned in my original comment above.

[thaidn]

@nanowizard I see.

I chatted with @tholenst and @chuckx this morning and @chuckx said that he's found a temporary workaround. There's a setting that we can add to cURL that would make it accept the environment variable.

Long-term speaking we'll have to revise how we build and distribute Tink C++ and Python binding.

[chuckx]

The temporary workaround (https://github.com/chuckx/tink/commit/e409e16c19724deb4abdb698b85f1ba7ca5e31bc) I stumbled across has a couple drawbacks:

• It's a compile time setting that was removed in curl 7.55.0. It's only available to us because we're depending on curl 7.49.1. These are both really old versions (~2016/2017) and the dependency should be updated. But then we'd have to maintain a local patch to reintroduce the option, which isn't ideal.
• From manual testing, it results in requiring the CURL_CA_BUNDLE environment variable to be set. Otherwise, certificate validation will always fail.

After looking at the AWS SDK documentation, an alternative approach is allowing more user customization of the AWS ClientConfiguration object, specifically the caPath and caFile attributes.

I'll follow up with a path forward for fixing this issue.

In the meantime, if you're really itching for a short term workaround, I can share a Python 3.8 binary wheel that was built based on the commit linked to above.

[rsivek]

Apologies I thought I has mentioned this, but as a workaround we just ended up deploying a custom lambda docker image where we symlinked the ca file tink was looking for.

[chuckx]

Ah, good to know. My understanding was that you were unsure if that workaround was usable in the AWS Lambda environment.

[morambro]

Tink Python (github.com/tink-crypto/tink-py) AWS KMS extension no longer depends on Tink C++ AWS KMS, so I think this issue no longer applies. Please feel free to open a new issue in https://github.com/tink-crypto/tink-py/issues if further clarifications are needed.