cURL error 77: error setting certificate verify locations

[benferns]

Hi - I've been getting the below odd exception very sporadically (once every few hundred invocations of this lambda). For now this is a bit of a placeholder issue for others to search out, as a) its not really reproducible yet, and b) I'm not even sure its related to this package.

If no-one else pops up with similar issues, or if I find no other info, I'll close this in a few weeks. Below is the full exception:

   "message": "Error executing \"Invoke\" on \"https://lambda.ap-southeast-2.
amazonaws.com/2015-03-31/functions/SC-XXXXXX-production-Sidecar-Sharp
ResizeImageLambda%3Aactive/invocations\"; AWS HTTP error: cURL error 77: 
error setting certificate verify locations:\n  CAfile: /opt/lib/curl/cert.pem\n  
CApath: none (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for 
https://lambda.ap-southeast-2.amazonaws.com/2015-03-31/functions/SC-
XXXXXX-production-Sidecar-SharpResizeImageLambda%3Aactive/invocations",
   "code": 0,
   "file": "/tmp/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:195",
   "trace": [
       "/tmp/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:97",
       "/tmp/vendor/guzzlehttp/promises/src/Promise.php:204",
       "/tmp/vendor/guzzlehttp/promises/src/Promise.php:169",
       "/tmp/vendor/guzzlehttp/promises/src/RejectedPromise.php:42",
       "/tmp/vendor/guzzlehttp/promises/src/TaskQueue.php:48",
       "/tmp/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:158",
       "/tmp/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:183",
       "/tmp/vendor/guzzlehttp/promises/src/Promise.php:248",
       "/tmp/vendor/guzzlehttp/promises/src/Promise.php:224",
       "/tmp/vendor/guzzlehttp/promises/src/Promise.php:269",
       "/tmp/vendor/guzzlehttp/promises/src/Promise.php:226",
       "/tmp/vendor/guzzlehttp/promises/src/Promise.php:269",
       "/tmp/vendor/guzzlehttp/promises/src/Promise.php:226",
       "/tmp/vendor/guzzlehttp/promises/src/Promise.php:62",
       "/tmp/vendor/hammerstone/sidecar/src/Results/PendingResult.php:48",
       "/tmp/vendor/hammerstone/sidecar/src/Manager.php:157",
       "/tmp/vendor/hammerstone/sidecar/src/Manager.php:156",
       ```

[aarondfrancis]

Hmm this is intriguing indeed. Thanks for reporting. We'll see if anyone else comes across it!

[maurocasas]

I've had this issue with my Laradock configuration.

All our lambdas are invoked through Horizon, and Horizon runs independently in it's container.

The issue lived inside Horizon's container.

When working with sidecar invokes, I stop my horizon container and run the daemon directly from the container hosting nginx+php-fpm, this will avoid trigger CURL issues.

Hope this helps!

[aarondfrancis]

@maurocasas interesting. Good to have another data point, for sure.

Do we have any idea what actually triggers the CURL errors? Is it intermittent? Should I just wrap the calls, catch CURL certificate issues, and try again?

[benferns]

@maurocasas what would you say the max number of invocations you're doing at once is when you see the issue?

I think curl opens a file descriptor per active stream, so it could be that we're hitting the max (1024 in Lambda and many other environments) and then the cert access fails due to running out of them, but reports the file as missing.

In my original case, the very top-end would be around that number - 700 to 900 invocations in an executeMany batch. This is just a wild stab in the dark though!

[aarondfrancis]

700 to 900 invocations in an executeMany batch

🤯 I can't believe that.

The file descriptor thing is a pretty good lead. I may look into that some more.

[benferns]

🤯 I can't believe that.

Reasonable usage estimates don't survive first contact with a motivated customer! In this case whereas most of my customers have 10-20 textures per file, this person has 200+ in a single file, that need to be sized to 4096, 2048, 1024 etc. within a quick processing duration!

Its a tough error to google, for sure, as a lot of people have the most simple version of the issue (that a cert file genuinely is missing). The intermittent nature, when running on Laravel Vapor (where curl certs actually missing would be a much louder issue!) makes me suspect its access, rather than existence of the file.

[geetpurwar]

I am also getting same error right after 50 invocations only.

Index.js

exports.handler = async function (event){
    console.log('HelloWorld');
}
return {
    status: 'done'
}

ExecuteMany to invoke multiple. Thats taking too long to execute and finally resulting in url error.

$results = \App\Sidecar\Screenshot::executeManyAsync(100);
foreach($results as $result){
dump($result->settled());
}

Error:

Aws\Lambda\Exception\LambdaException
Error executing "Invoke" on "https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/SC-Laravel-local-Sidecar-Screenshot%3Aactive/invocations"; AWS HTTP error: cURL error 7: (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/SC-Laravel-local-Sidecar-Screenshot%3Aactive/invocations

I am trying to experiment with some other executions and will keep you posted.

[aarondfrancis]

@geetpurwar interesting, thanks for the data point. Let me know if you discover anything else.

Also, I think your return {status: done} needs to be inside your handler function if you want to use that back on the Laravel side.

[aarondfrancis]

@geetpurwar have you run into this at all anymore or can I close it?

[geetpurwar]

@aarondfrancis You can close it. I haven't observed it anymore.