Logs API in Node JS does not process the logs to S3 Immediately.

[vignesh-uj]

Not sure if it is the limitation, but the logs are uploaded to S3 only during the INVOKE and SHUTDOWN event which makes me wait to receive logs of current invocation in the next one or during the shut down event.

Also tried uploading the batch to S3 as soon as the request is received at the local server, could able to upload them except the very last log in the lambda function.

Is there any possible solution to this ?

[julianwood]

There is a new platform.runtimeDone message which Lambda sends to subscribed logs extensions. You can wait for this before sending the logs up to S3. https://aws.amazon.com/blogs/compute/performance-and-functionality-improvements-for-aws-lambda-extensions/ I will look to update the example in this repo but hopefully this helps you move forward.

[waynerobinson]

Even with platform.runtimeDone the logs don't get sent to the extension until the next call in a subsequent (not always the next) execution of the function.

So you're waiting until 1 or more future calls of a function or until the entire function gets shutdown before you get logs for it.

The platform should constantly be pushing logs to the listening HTTP server rather than only during the invoke step (which happens at the beginning of a function) or in some type of synchronous way at the end of a function call.

[julianwood]

Are you looking for and waiting until you receive a platform.runtimeDone log message before your extension sends its /next Extensions API call?

[Borduhh]

Trying to build a test case with Grafana Loki's HTTP API and seeing the same thing. Is it possible to process the logs in the same Lambda invocation?

On the first invocation, we receive nothing. On the second one, we get the details of the first invocation:

[
  {
    "time": "2022-06-22T19:55:58.555Z",
    "type": "platform.start",
    "record": { "requestId": "16b2a55e-a0eb-40cd-9415-76f8a481c8dc", "version": "$LATEST" }
  },
  {
    "time": "2022-06-22T19:55:58.674Z",
    "type": "platform.logsSubscription",
    "record": {
      "name": "nodejs-example-logs-api-extension",
      "state": "Subscribed",
      "types": ["platform", "function"]
    }
  },
  {
    "time": "2022-06-22T19:55:58.674Z",
    "type": "platform.extension",
    "record": {
      "name": "nodejs-example-logs-api-extension",
      "state": "Ready",
      "events": ["SHUTDOWN", "INVOKE"]
    }
  },
  {
    "time": "2022-06-22T19:55:58.711Z",
    "type": "function",
    "record": "2022-06-22T19:55:58.676Z\t16b2a55e-a0eb-40cd-9415-76f8a481c8dc\tINFO\tInside Lambda function\n"
  },
  {
    "time": "2022-06-22T19:55:58.733Z",
    "type": "platform.end",
    "record": { "requestId": "16b2a55e-a0eb-40cd-9415-76f8a481c8dc" }
  },
  {
    "time": "2022-06-22T19:55:58.733Z",
    "type": "platform.report",
    "record": {
      "requestId": "16b2a55e-a0eb-40cd-9415-76f8a481c8dc",
      "metrics": {
        "durationMs": 56.63,
        "billedDurationMs": 57,
        "memorySizeMB": 128,
        "maxMemoryUsedMB": 83,
        "initDurationMs": 644.87
      }
    }
  }
]