confluentinc
How to use this package to AWS Lambda?
We’re interested in using this package within an AWS Lambda environment, but we’re unsure whether it’s compatible or how to properly deploy it.
Is there any documentation or guidance on how to get started with Lambda support for this package?
Thanks in advance!
We're interested in this as well.
Currently, our lambdas error as soon as the '@confluentinc/kafka-javascript' package is imported.
This is the error:
INIT_REPORT Init Duration: 1652.67 ms Phase: init Status: error Error Type: Runtime.Unknown
To me, it reads like the runtime of the lambda is incompatible with the package.
This is our lambda runtime configuration:
- Memory 1024 MB
- Runtime: Node.js 22.x
- Architecture: arm64
I can find Node 22 and arm64 on here, https://docs.confluent.io/kafka-clients/javascript/current/overview.html but I'm not sure if that matches lambdas architecture...
I have two preliminary suggestions:
- is it possible to increase the verbosity of the available logging? The error is quite uninformative
- Seeing this document https://docs.aws.amazon.com/lambda/latest/dg/nodejs-package.html#nodejs-package-create-dependencies it seems like you have to make a zip file with the installed node modules that's sent to Lambda. While installing, our package checks the current architecture to determine which prebuilt binary to fetch. So, for instance, you create your zip file on a macOS system with Node 18, it won't work on linux with Node 22. If this is the case, then you could use a docker image with the right OS, node version, and architecture to create the zip file for submission. I've seen similar errors earlier in different contexts.
Hi @milindl , thanks for the response. I hope there’s a guideline in the documentation that shows how to deploy to Lambda—either including the steps to create a layer or providing an official layer for people to use.
Hi @milindl
is it possible to increase the verbosity of the available logging? The error is quite uninformative
Good suggestion, thanks! I increased the verbosity to the maximum possible for lambda. See attached logs at the end.
[...] So, for instance, you create your zip file on a macOS system with Node 18, it won't work on linux with Node 22. [...]
That's also a good hint, thanks a lot. All our lambdas are built in github actions, using the following runner configuration:
- OS: Ubuntu 24.04.2 LTS
- image: ubuntu-24.04 - 20250527.1.0
- Included Software: https://github.com/actions/runner-images/blob/ubuntu24/20250527.1/images/ubuntu/Ubuntu2404-Readme.md
- Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20250527.1
We use the same node version at buildtime & at runtime (node 22).
I tried both possible lambda runtime architectures (arm64 and x86_64) and both do currently not work (and produce the same error).
I also tried both available bundling options for the lambda (CJS and ESM, ESM being our default) and they produce different errors currently.
Lambda logs (JSON format) for the ESM bundling (same for arm64 and x86_64):
{
"time": "2025-06-02T08:33:13.231Z",
"type": "platform.initStart",
"record": {
"initializationType": "on-demand",
"phase": "invoke",
"runtimeVersion": "nodejs:22.v43",
"runtimeVersionArn": "arn:aws:lambda:eu-central-1::runtime:e4b97d4055c2afa813f5bda7d8e92ef971a7605f10be17ef7ae967743a21ba7d",
"functionName": "<FUNCTION_NAME>",
"functionVersion": "$LATEST"
}
}
file:///var/task/index.mjs:431265
if (fileName !== __filename) {
^
ReferenceError: __filename is not defined
(Use `node --trace-uncaught ...` to show where the exception was thrown)
Node.js v22.15.0
{
"time": "2025-06-02T08:33:16.034Z",
"type": "platform.initRuntimeDone",
"record": {
"initializationType": "on-demand",
"phase": "invoke",
"status": "error",
"errorType": "Runtime.ExitError"
}
}
{
"time": "2025-06-02T08:33:16.034Z",
"type": "platform.initReport",
"record": {
"initializationType": "on-demand",
"phase": "invoke",
"status": "error",
"errorType": "Runtime.ExitError",
"metrics": {
"durationMs": 2803.685
}
}
}
{
"time": "2025-06-02T08:33:16.034Z",
"type": "platform.start",
"record": {
"requestId": "2ea4f82b-5fda-483c-8ed8-427d5908c82a",
"functionArn": "<FUNCTION_ARN>",
"version": "$LATEST",
"tracing": {
"spanId": "69765a8e17d79988",
"type": "X-Amzn-Trace-Id",
"value": "Root=1-683d61c9-3a9685c768d1be1f76d7404a;Parent=27d52784c5996cbf;Sampled=1"
}
}
}
{
"time": "2025-06-02T08:33:16.035Z",
"type": "platform.runtimeDone",
"record": {
"requestId": "2ea4f82b-5fda-483c-8ed8-427d5908c82a",
"status": "error",
"errorType": "Runtime.ExitError",
"metrics": {
"durationMs": 0,
"producedBytes": 0
}
}
}
{
"time": "2025-06-02T08:33:16.035Z",
"type": "platform.report",
"record": {
"requestId": "2ea4f82b-5fda-483c-8ed8-427d5908c82a",
"metrics": {
"durationMs": 2805.027,
"billedDurationMs": 2806,
"memorySizeMB": 1024,
"maxMemoryUsedMB": 202
},
"tracing": {
"spanId": "69765a8e17d79988",
"type": "X-Amzn-Trace-Id",
"value": "Root=1-683d61c9-3a9685c768d1be1f76d7404a;Parent=27d52784c5996cbf;Sampled=1"
},
"status": "error",
"errorType": "Runtime.ExitError"
}
}
{
"time": "2025-06-02T08:33:16.063Z",
"type": "platform.initStart",
"record": {
"initializationType": "on-demand",
"phase": "init",
"runtimeVersion": "nodejs:22.v43",
"runtimeVersionArn": "arn:aws:lambda:eu-central-1::runtime:e4b97d4055c2afa813f5bda7d8e92ef971a7605f10be17ef7ae967743a21ba7d",
"functionName": "FUNCTION_NAME",
"functionVersion": "$LATEST"
}
}
file:///var/task/index.mjs:431265
if (fileName !== __filename) {
^
ReferenceError: __filename is not defined
(Use `node --trace-uncaught ...` to show where the exception was thrown)
Node.js v22.15.0
{
"time": "2025-06-02T08:33:17.668Z",
"type": "platform.initRuntimeDone",
"record": {
"initializationType": "on-demand",
"phase": "init",
"status": "error",
"errorType": "Runtime.ExitError"
}
}
{
"time": "2025-06-02T08:33:17.668Z",
"type": "platform.initReport",
"record": {
"initializationType": "on-demand",
"phase": "init",
"status": "error",
"errorType": "Runtime.ExitError",
"metrics": {
"durationMs": 1605.514
}
}
}
When looking at the bundled lambda's index.mjs, this is the code trace that produces the above error:
require_librdkafka -> require_bindings -> getFileName. Those are the relevant code parts:
// node_modules/.pnpm/@[email protected]/node_modules/@confluentinc/kafka-javascript/librdkafka.js
var require_librdkafka = __commonJS({
"node_modules/.pnpm/@[email protected]/node_modules/@confluentinc/kafka-javascript/librdkafka.js"(exports, module) {
var kafka = require_bindings()("confluent-kafka-javascript");
module.exports = kafka;
}
});
// node_modules/.pnpm/[email protected]/node_modules/bindings/bindings.js
var require_bindings = __commonJS({
"node_modules/.pnpm/[email protected]/node_modules/bindings/bindings.js"(exports, module) {
var fs = __require("fs");
var path = __require("path");
var fileURLToPath2 = require_file_uri_to_path();
var join = path.join;
var dirname = path.dirname;
var exists = fs.accessSync && function(path2) {
try {
fs.accessSync(path2);
} catch (e) {
return false;
}
return true;
} || fs.existsSync || path.existsSync;
var defaults = {
arrow: process.env.NODE_BINDINGS_ARROW || " \u2192 ",
compiled: process.env.NODE_BINDINGS_COMPILED_DIR || "compiled",
platform: process.platform,
arch: process.arch,
nodePreGyp: "node-v" + process.versions.modules + "-" + process.platform + "-" + process.arch,
version: process.versions.node,
bindings: "bindings.node",
try: [
// node-gyp's linked version in the "build" dir
["module_root", "build", "bindings"],
// node-waf and gyp_addon (a.k.a node-gyp)
["module_root", "build", "Debug", "bindings"],
["module_root", "build", "Release", "bindings"],
// Debug files, for development (legacy behavior, remove for node v0.9)
["module_root", "out", "Debug", "bindings"],
["module_root", "Debug", "bindings"],
// Release files, but manually compiled (legacy behavior, remove for node v0.9)
["module_root", "out", "Release", "bindings"],
["module_root", "Release", "bindings"],
// Legacy from node-waf, node <= 0.4.x
["module_root", "build", "default", "bindings"],
// Production "Release" buildtype binary (meh...)
["module_root", "compiled", "version", "platform", "arch", "bindings"],
// node-qbs builds
["module_root", "addon-build", "release", "install-root", "bindings"],
["module_root", "addon-build", "debug", "install-root", "bindings"],
["module_root", "addon-build", "default", "install-root", "bindings"],
// node-pre-gyp path ./lib/binding/{node_abi}-{platform}-{arch}
["module_root", "lib", "binding", "nodePreGyp", "bindings"]
]
};
function bindings(opts) {
if (typeof opts == "string") {
opts = { bindings: opts };
} else if (!opts) {
opts = {};
}
Object.keys(defaults).map(function(i2) {
if (!(i2 in opts)) opts[i2] = defaults[i2];
});
if (!opts.module_root) {
opts.module_root = exports.getRoot(exports.getFileName());
}
if (path.extname(opts.bindings) != ".node") {
opts.bindings += ".node";
}
var requireFunc = typeof __webpack_require__ === "function" ? __non_webpack_require__ : __require;
var tries = [], i = 0, l = opts.try.length, n, b, err2;
for (; i < l; i++) {
n = join.apply(
null,
opts.try[i].map(function(p) {
return opts[p] || p;
})
);
tries.push(n);
try {
b = opts.path ? requireFunc.resolve(n) : requireFunc(n);
if (!opts.path) {
b.path = n;
}
return b;
} catch (e) {
if (e.code !== "MODULE_NOT_FOUND" && e.code !== "QUALIFIED_PATH_RESOLUTION_FAILED" && !/not find/i.test(e.message)) {
throw e;
}
}
}
err2 = new Error(
"Could not locate the bindings file. Tried:\n" + tries.map(function(a) {
return opts.arrow + a;
}).join("\n")
);
err2.tries = tries;
throw err2;
}
module.exports = exports = bindings;
exports.getFileName = function getFileName(calling_file) {
var origPST = Error.prepareStackTrace, origSTL = Error.stackTraceLimit, dummy = {}, fileName;
Error.stackTraceLimit = 10;
Error.prepareStackTrace = function(e, st) {
for (var i = 0, l = st.length; i < l; i++) {
fileName = st[i].getFileName();
if (fileName !== __filename) {
if (calling_file) {
if (fileName !== calling_file) {
return;
}
} else {
return;
}
}
}
};
Error.captureStackTrace(dummy);
dummy.stack;
Error.prepareStackTrace = origPST;
Error.stackTraceLimit = origSTL;
var fileSchema = "file://";
if (fileName.indexOf(fileSchema) === 0) {
fileName = fileURLToPath2(fileName);
}
return fileName;
};
exports.getRoot = function getRoot(file) {
var dir = dirname(file), prev;
while (true) {
if (dir === ".") {
dir = process.cwd();
}
if (exists(join(dir, "package.json")) || exists(join(dir, "node_modules"))) {
return dir;
}
if (prev === dir) {
throw new Error(
'Could not find module root given file: "' + file + '". Do you have a `package.json` file? '
);
}
prev = dir;
dir = join(dir, "..");
}
};
}
});
Lambda logs (JSON format) for the CJS bundling (same for arm64 and x86_64):
{
"time": "2025-06-02T14:37:23.727Z",
"type": "platform.initStart",
"record": {
"initializationType": "on-demand",
"phase": "init",
"runtimeVersion": "nodejs:22.v43",
"runtimeVersionArn": "arn:aws:lambda:eu-central-1::runtime:fd2e05b324f99edd3c6e17800b2535deb79bcce74b7506d595a94870b3d9bd2e",
"functionName": "<FUNCTION_NAME>",
"functionVersion": "$LATEST"
}
}
{
"timestamp": "2025-06-02T14:37:25.373Z",
"level": "ERROR",
"message": {
"errorType": "Error",
"errorMessage": "Could not find module root given file: \"/var/task/index.js\". Do you have a `package.json` file? ",
"stackTrace": [
"Error: Could not find module root given file: \"/var/task/index.js\". Do you have a `package.json` file? ",
" at Function.getRoot (/var/task/index.js:427074:17)",
" at bindings (/var/task/index.js:427001:37)",
" at node_modules/.pnpm/@[email protected]/node_modules/@confluentinc/kafka-javascript/librdkafka.js (/var/task/index.js:427088:35)",
" at __require (/var/task/index.js:9:50)",
" at node_modules/.pnpm/@[email protected]/node_modules/@confluentinc/kafka-javascript/lib/client.js (/var/task/index.js:427562:17)",
" at __require (/var/task/index.js:9:50)",
" at node_modules/.pnpm/@[email protected]/node_modules/@confluentinc/kafka-javascript/lib/kafka-consumer.js (/var/task/index.js:428182:18)",
" at __require (/var/task/index.js:9:50)",
" at node_modules/.pnpm/@[email protected]/node_modules/@confluentinc/kafka-javascript/lib/rdkafka.js (/var/task/index.js:429503:25)",
" at __require (/var/task/index.js:9:50)"
]
}
}
{
"time": "2025-06-02T14:37:25.469Z",
"type": "platform.initRuntimeDone",
"record": {
"initializationType": "on-demand",
"phase": "init",
"status": "error",
"errorType": "Runtime.Unknown"
}
}
{
"time": "2025-06-02T14:37:25.469Z",
"type": "platform.initReport",
"record": {
"initializationType": "on-demand",
"phase": "init",
"status": "error",
"errorType": "Runtime.Unknown",
"metrics": {
"durationMs": 1742.111
}
}
}
{
"time": "2025-06-02T14:37:25.491Z",
"type": "platform.initStart",
"record": {
"initializationType": "on-demand",
"phase": "invoke",
"runtimeVersion": "nodejs:22.v43",
"runtimeVersionArn": "arn:aws:lambda:eu-central-1::runtime:fd2e05b324f99edd3c6e17800b2535deb79bcce74b7506d595a94870b3d9bd2e",
"functionName": "<FUNCTION_NAME>",
"functionVersion": "$LATEST"
}
}
{
"timestamp": "2025-06-02T14:37:27.903Z",
"level": "ERROR",
"message": {
"errorType": "Error",
"errorMessage": "Could not find module root given file: \"/var/task/index.js\". Do you have a `package.json` file? ",
"stackTrace": [
"Error: Could not find module root given file: \"/var/task/index.js\". Do you have a `package.json` file? ",
" at Function.getRoot (/var/task/index.js:427074:17)",
" at bindings (/var/task/index.js:427001:37)",
" at node_modules/.pnpm/@[email protected]/node_modules/@confluentinc/kafka-javascript/librdkafka.js (/var/task/index.js:427088:35)",
" at __require (/var/task/index.js:9:50)",
" at node_modules/.pnpm/@[email protected]/node_modules/@confluentinc/kafka-javascript/lib/client.js (/var/task/index.js:427562:17)",
" at __require (/var/task/index.js:9:50)",
" at node_modules/.pnpm/@[email protected]/node_modules/@confluentinc/kafka-javascript/lib/kafka-consumer.js (/var/task/index.js:428182:18)",
" at __require (/var/task/index.js:9:50)",
" at node_modules/.pnpm/@[email protected]/node_modules/@confluentinc/kafka-javascript/lib/rdkafka.js (/var/task/index.js:429503:25)",
" at __require (/var/task/index.js:9:50)"
]
}
}
{
"time": "2025-06-02T14:37:28.072Z",
"type": "platform.initRuntimeDone",
"record": {
"initializationType": "on-demand",
"phase": "invoke",
"status": "error",
"errorType": "Runtime.Unknown"
}
}
{
"time": "2025-06-02T14:37:28.073Z",
"type": "platform.initReport",
"record": {
"initializationType": "on-demand",
"phase": "invoke",
"status": "error",
"errorType": "Runtime.Unknown",
"metrics": {
"durationMs": 2581.309
}
}
}
{
"time": "2025-06-02T14:37:28.073Z",
"type": "platform.start",
"record": {
"requestId": "ab5fdd36-8d16-486c-8be2-7efb0c2ecca1",
"functionArn": "<FUNCTION_ARN>",
"version": "$LATEST",
"tracing": {
"spanId": "169d7dcf1f466182",
"type": "X-Amzn-Trace-Id",
"value": "Root=1-683db723-6a1d131b6e9d9ab108adf796;Parent=321f6f8ae8e37911;Sampled=1"
}
}
}
{
"time": "2025-06-02T14:37:28.073Z",
"type": "platform.runtimeDone",
"record": {
"requestId": "ab5fdd36-8d16-486c-8be2-7efb0c2ecca1",
"status": "error",
"errorType": "Runtime.Unknown",
"metrics": {
"durationMs": 0,
"producedBytes": 0
}
}
}
{
"time": "2025-06-02T14:37:28.073Z",
"type": "platform.report",
"record": {
"requestId": "ab5fdd36-8d16-486c-8be2-7efb0c2ecca1",
"metrics": {
"durationMs": 2582.303,
"billedDurationMs": 2583,
"memorySizeMB": 1024,
"maxMemoryUsedMB": 163
},
"tracing": {
"spanId": "169d7dcf1f466182",
"type": "X-Amzn-Trace-Id",
"value": "Root=1-683db723-6a1d131b6e9d9ab108adf796;Parent=321f6f8ae8e37911;Sampled=1"
},
"status": "error",
"errorType": "Runtime.Unknown"
}
}
{
"time": "2025-06-02T14:37:28.103Z",
"type": "platform.initStart",
"record": {
"initializationType": "on-demand",
"phase": "init",
"runtimeVersion": "nodejs:22.v43",
"runtimeVersionArn": "arn:aws:lambda:eu-central-1::runtime:fd2e05b324f99edd3c6e17800b2535deb79bcce74b7506d595a94870b3d9bd2e",
"functionName": "<FUNCTION_NAME>",
"functionVersion": "$LATEST"
}
}
{
"timestamp": "2025-06-02T14:37:29.507Z",
"level": "ERROR",
"message": {
"errorType": "Error",
"errorMessage": "Could not find module root given file: \"/var/task/index.js\". Do you have a `package.json` file? ",
"stackTrace": [
"Error: Could not find module root given file: \"/var/task/index.js\". Do you have a `package.json` file? ",
" at Function.getRoot (/var/task/index.js:427074:17)",
" at bindings (/var/task/index.js:427001:37)",
" at node_modules/.pnpm/@[email protected]/node_modules/@confluentinc/kafka-javascript/librdkafka.js (/var/task/index.js:427088:35)",
" at __require (/var/task/index.js:9:50)",
" at node_modules/.pnpm/@[email protected]/node_modules/@confluentinc/kafka-javascript/lib/client.js (/var/task/index.js:427562:17)",
" at __require (/var/task/index.js:9:50)",
" at node_modules/.pnpm/@[email protected]/node_modules/@confluentinc/kafka-javascript/lib/kafka-consumer.js (/var/task/index.js:428182:18)",
" at __require (/var/task/index.js:9:50)",
" at node_modules/.pnpm/@[email protected]/node_modules/@confluentinc/kafka-javascript/lib/rdkafka.js (/var/task/index.js:429503:25)",
" at __require (/var/task/index.js:9:50)"
]
}
}
{
"time": "2025-06-02T14:37:29.598Z",
"type": "platform.initRuntimeDone",
"record": {
"initializationType": "on-demand",
"phase": "init",
"status": "error",
"errorType": "Runtime.Unknown"
}
}
{
"time": "2025-06-02T14:37:29.598Z",
"type": "platform.initReport",
"record": {
"initializationType": "on-demand",
"phase": "init",
"status": "error",
"errorType": "Runtime.Unknown",
"metrics": {
"durationMs": 1494.505
}
}
}
For additional context, we use pnpm as a package manager.
The CJS error message "Could not find module root given file: \"/var/task/index.js\". Do you have a package.json file? " is (maybe?) understandable, since in our lambdas, we do not have a package.json (only the bundled index.js file for the function).
Hope that helps 👍
Would be great if you could find a solution for lambdas bundled with ESM. The architecture (arm64 or x86_64) is not that important to us.
but we’re unsure whether it’s compatible
Yes. It's compatible. I run Lambda with:
- Purely ESM (MJS extension);
- NodeJS 22.x;
- ARM64 container.
but we’re unsure whether it’s compatible
Yes. It's compatible. I run Lambda with:
- Purely ESM (MJS extension);
- NodeJS 22.x;
- ARM64 container.
Do you use Layer or you bundle the code using webpack or other tools?
To just use the library, I got it to work by creating a layer - it was straightforward, and I didn't encounter any errors. I'm posting the steps for it below, and then, I'll be figuring out if I can get it to work when the code is being bundled.
@felixmagnus , how are you bundling your code?
Layer creation (run it on the right architecture and with the target node version, using a docker container if necessary):
mkdir layer && cd layer
mkdir nodejs && cd nodejs
npm init -y
npm install @confluentinc/kafka-javascript
cd ../
zip -r layer.zip nodejs/
The zip file should look like this internally:
layer.zip
- nodejs
- package.json
- package-lock.json
- node_modules/ (lots of subfolders)
I created a layer by uploading this to create a new layer (make sure to match platform and version).
And then the following index.mjs file for the function, configured to use my layer, worked correctly (I'm just printing the library version to check the import):
import Kafka from '@confluentinc/kafka-javascript'
export const handler = async (event) => {
// TODO implement
const response = {
statusCode: 200,
body: Kafka.librdkafkaVersion,
};
return response;
};
Thanks for looking into this!
We're bundling our lambdas using AWS CDK, with the following settings / configs:
Monorepo / Package manager:
- pnpm
10.12.1 - turborepo
2.5.4
package.json
aws-cdkversion:2.1012.0aws-cdk-libversion:2.193.0
cdk.json
appcommand:npx tsx infra/main.ts
Lambda construct:
NodejsFunctionfromaws-cdk-lib/aws-lambda-nodejswith the following settings:
{
runtime: Runtime.NODEJS_22_X,
architecture: Architecture.ARM_64,
memorySize: 1024,
bundling: {
format: OutputFormat.ESM,
mainFields: ['module', 'main'],
},
...,
}
@milindl is there any more information I can provide?
I believe this could be solvable if librdkafka.js allowed overriding where to search for the bindings.
Which seems possible via module_root option https://github.com/TooTallNate/node-bindings/blob/master/bindings.js#L81
That way we could create an actual Lambda Layer with the proper binary from Releases that could be mounted on /opt and pass some env variable CONFLUENTINC_KAFKA_BINARY_PATH=/opt.
In the meantime, downloading the Release that works for your lambda arch and zipping it along the bundled lambda code seems to work.
