LAMBDA_RUNTIME Failed to get next invocation. No Response from endpoint

[samuelkeenvil]

Hi guys!

I just writing some lambdas in c++ using the aws-lambda-cpp

i checking my log and i have this Error in my log:

[ERROR] [1553201323958] LAMBDA_RUNTIME Failed to get next invocation. No Response from endpoint

do you know if this is normal? because i check and debug my code but i don't found nothing.

what my lambda do is extract the body from SQS request, and make a slack post with the message.

for that i use curl and have the ssl disable

other thing is i dont see a print message that i have in the main after the

invocation_response handler(invocation_request const &req) {
return invocation_response::success();
}

this is my main:

int main() {
    //use AWS namespace and libs
    using namespace Aws;
    //set the log for the sdk
    SDKOptions options;
    //init AWS API
    InitAPI(options);
    {
        auto handler_function = [](aws::lambda_runtime::invocation_request const &req) {
            return handler(req);
        };
        run_handler(handler_function);
    }
    // shutdown the aws api
    std::cout << "ShutdownAPI api" << "\n";
    ShutdownAPI(options);

    std::cout << "exit" << "\n";

    //exit
    return 0;
}

i dont see the Shutdown and exit print in the cloud-watch log

so the question the ShutdownAPI(options); and the return 0 is never called?

thank you!!!

[marcomagdy]

The I/O is buffered. You should see the messages if you add std::cout.flush() at the end, or append << std::endl which has the side-effect of flushing.

[marcomagdy]

Are you seeing the message [ERROR] [1553201323958] LAMBDA_RUNTIME Failed to get next invocation. No Response from endpoint in your logs on every invocation? or sometimes? I'd like to get more data on this to diagnose what's happening.

[samuaz]

@marcomagdy sorry for delay, this is my personal github account, i see this in every invocation, "all works fine" but in see that message in every invocation, let me know what data you need if you want i can share the code of my function

[webbnh]

"all works fine" but in see that message in every invocation

Me, too.

Every Lambda invocation starts out like this:

Lambda function invocation starting.
[WARN] 2019-07-03 22:05:46 FileSystemUtils [139730358032256] Home dir not stored in environment, trying to fetch manually from the OS.
Lambda function calling Lambda runtime.
START RequestId: cb1f23f2-4df9-4093-985c-d2a49d2f9354 Version: $LATEST
[ERROR] [1562191546320] LAMBDA_RUNTIME Failed to get next invocation. No Response from endpoint
Entering my_handler.
[...]

And, for Lambda invocations which get reused, it shows up on the next invocation too:

[...]
END RequestId: f1be42e8-84e2-4ca5-b877-3ac84d33b9d1
REPORT RequestId: f1be42e8-84e2-4ca5-b877-3ac84d33b9d1 Init Duration: 202.89 ms Duration: 9216.62 ms Billed Duration: 9500 ms Memory Size: 256 MB Max Memory Used: 80 MB
START RequestId: 384e8a2e-2230-4642-81db-c91f725f8e69 Version: $LATEST
[ERROR] [1562191572175] LAMBDA_RUNTIME Failed to get next invocation. No Response from endpoint
Entering my_handler.
[...]

Also, the warning:

[WARN] 2019-07-03 22:05:44 FileSystemUtils [140509995136896] Home dir not stored in environment, trying to fetch manually from the OS.

shows up at various points, as well, apparently without negative effect.

[marcomagdy]

@webbnh the last warning is from the AWS SDK. Not Lambda. @samuaz I'm unable to reproduce the problem, and I feel it's yet-another-libcurl-problem that does not happen when using a different libcurl version.

I'm going to resolve all the libcurl problems by bundling a good version with this runtime.

[deepak-1530]

@marcomagdy Curious to see when this will be resolved. I am also stuck at the same spot

[marcomagdy]

The problem is I need to reproduce the problem and I haven't been able to so far. What version of libcurl are you using?

If a newer version of libcurl resolves this issue, I will put up a PR that builds a known good version of libcurl and link to it as a CMake option. But I need your help to make sure that's the root cause first.

[deepak-1530]

I am using version 7.47.0

[marcomagdy]

Try building the latest release of libcurl from source and install it somewhere on your system that doesn't conflict with your OS (pass --prefix /home//my-libcurl-install to the configure command) and then build the runtime normally and pass -DCMAKE_PREFIX_PATH="/home/<username>/my-libcurl-install" to your CMake command.

Meanwhile, I will try to reproduce with libcurl 7.47.0 Which distro does that come as a default with btw?

[webbnh]

# curl-config --version
libcurl 7.47.0

# uname -a
Linux aa6d84702054 4.14.131-linuxkit #1 SMP Fri Jul 19 12:31:17 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

That's a Docker container built from ubuntu:16.04 with apt-get install -y libcurl4-openssl-dev.

[ghost]

I am facing same error. I am using nodejs12.x as runtime with serverless framework.

I have 39 lambda functions in the stack but only 1 shows this error. I get following lines:

INFO Unexpected Top Level Error: Error: Failed to get next invocation, error -1
Error: Runtime exited with error: signal: segmentation fault (core dumped) Runtime.ExitError
[ERROR] [1621229600432] LAMBDA_RUNTIME Failed to post handler success response. Http response code: 403.
[ERROR] [1621229600449] LAMBDA_RUNTIME Failed to get next invocation. No Response from endpoint

The only difference is that this particular lambda writes a file to S3. Maybe that is what causes this error? The file is successfully updated though. I am using aws-sdk's putObject to write the file. The file is a small json file with size 131.0 B (as shown in AWS S3 Management Console).

[david-wb]

Any luck with this? I'm having the same issue in a python lambda.

[AltafHussain4748]

@david-wb

How did you solve that?

[hbobenicio]

I'm trying to run the demo example from the docs and I'm facing the same error. here is what I found: First, I got this error:

[ERROR] [1660998782441] LAMBDA_RUNTIME Failed to get next invocation. No Response from endpoint
[ERROR] [1660998782581] LAMBDA_RUNTIME Failed to get next invocation. No Response from endpoint
[ERROR] [1660998782664] LAMBDA_RUNTIME Failed to get next invocation. No Response from endpoint
[ERROR] [1660998782664] LAMBDA_RUNTIME Exhausted all retries. This is probably a bug in libcurl v7.68.0 Exiting!

After I grep the runtime source code, I found that this is comming from here: https://github.com/awslabs/aws-lambda-cpp/blob/095ff84911d0f3dcd0f428c4a5b49124ee66ee6b/src/runtime.cpp#L243 But the actual CURL "error" is just logged as a debug level log (which IMHO shouldn't be a debug level. it should be an error level message. No error context must be truncated...) https://github.com/awslabs/aws-lambda-cpp/blob/095ff84911d0f3dcd0f428c4a5b49124ee66ee6b/src/runtime.cpp#L242 So after recompiling my lambda with -DAWS_LAMBDA_LOG=2 (maybe it would be better to have a runtime log level configuration instead of needing to recompile it, but well... let's focus on the problem here...) I got the missing context:

[INFO] [1660999392927] LAMBDA_RUNTIME Initializing the C++ Lambda Runtime version 0.0.0
[DEBUG] [1660999392928] LAMBDA_RUNTIME Making request to http:///2018-06-01/runtime/invocation/next
[DEBUG] [1660999392960] LAMBDA_RUNTIME CURL DBG: Could not resolve host: 2018-06-01

[DEBUG] [1660999392960] LAMBDA_RUNTIME CURL DBG: Closing connection 0

[DEBUG] [1660999392960] LAMBDA_RUNTIME Completed request to http:///2018-06-01/runtime/invocation/next
[DEBUG] [1660999392960] LAMBDA_RUNTIME CURL returned error code 6 - Couldn't resolve host name
[ERROR] [1660999392960] LAMBDA_RUNTIME Failed to get next invocation. No Response from endpoint
[DEBUG] [1660999392960] LAMBDA_RUNTIME Making request to http:///2018-06-01/runtime/invocation/next
[DEBUG] [1660999392988] LAMBDA_RUNTIME CURL DBG: Could not resolve host: 2018-06-01

[DEBUG] [1660999392988] LAMBDA_RUNTIME CURL DBG: Closing connection 1

[DEBUG] [1660999392988] LAMBDA_RUNTIME Completed request to http:///2018-06-01/runtime/invocation/next
[DEBUG] [1660999392988] LAMBDA_RUNTIME CURL returned error code 6 - Couldn't resolve host name
[ERROR] [1660999392988] LAMBDA_RUNTIME Failed to get next invocation. No Response from endpoint
[DEBUG] [1660999392988] LAMBDA_RUNTIME Making request to http:///2018-06-01/runtime/invocation/next
[DEBUG] [1660999393011] LAMBDA_RUNTIME CURL DBG: Could not resolve host: 2018-06-01

[DEBUG] [1660999393011] LAMBDA_RUNTIME CURL DBG: Closing connection 2

[DEBUG] [1660999393011] LAMBDA_RUNTIME Completed request to http:///2018-06-01/runtime/invocation/next
[DEBUG] [1660999393011] LAMBDA_RUNTIME CURL returned error code 6 - Couldn't resolve host name
[ERROR] [1660999393011] LAMBDA_RUNTIME Failed to get next invocation. No Response from endpoint
[ERROR] [1660999393011] LAMBDA_RUNTIME Exhausted all retries. This is probably a bug in libcurl v7.68.0 Exiting!

Clearly requesting http:///2018-06-01/runtime/invocation/next would never going to work (and it's not a curl issue either). After checking the source how I set's this URL I found that the runtime creates this endpoint based on the environment variable AWS_LAMBDA_RUNTIME_API which, in my case, is clearly empty. https://github.com/awslabs/aws-lambda-cpp/blob/095ff84911d0f3dcd0f428c4a5b49124ee66ee6b/src/runtime.cpp#L405 I'm wondering if this environment variable is required for this runtime to work. Because if it is, we get no clue about it just from the logs... Maybe this runtime could improve the logging about this of better handling the endpoint configuration/setting process to avoid this confusion.

[marcomagdy]

That environment variable is set by AWS. See https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html#configuration-envvars-runtime

I am not 100% sure how that environment variable is empty for you.

[hbobenicio]

Thanks for the tip, marcomagdy! Just tried running locally on my machine. That explains a lot =) Sorry if this is dummy... I was just curious what would happen before uploading it to AWS. So I got these messages and found this issue. I'm a total beginner on aws lambda too.