EndpointFunctions[fn] is not a function

[taoatmars]

Checkboxes for prior research

• [X] I've gone through Developer Guide and API reference
• [X] I've checked AWS Forums and StackOverflow.
• [X] I've searched for previous similar issues and didn't find any solution.

Describe the bug

aws-sdk recently switched from @aws-sdk/util-endpoints to @smithy/util-endpoints, when calling client-lambda inside a container, Error: endpointFunctions[fn] is not a function will appear when calling lambda functions.

SDK version number

@aws-sdk/[email protected]

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v18.18.2

Reproduction Steps

import { LambdaClient, InvokeCommand, InvocationType } from '@aws-sdk/client-lambda'; // ES Modules import
.
.
.
const input = {
	FunctionName: `some-function`,
	InvocationType: InvocationType.RequestResponse,
	Payload: JSON.stringify({
        . . .
         }),
};

const command = new InvokeCommand(input);
const response = await client.send(command);

Observed Behavior

@smithy/util-endpoints was not able to find a function call endpointFunctions

Expected Behavior

@smithy/util-endpoints able to locate and call endpointFunctions

Possible Solution

roll back to "@aws-sdk/client-lambda": "3.418.0"

Additional Information/Context

No response

[RanVaknin]

Hi @taoatmars,

I'm not able to reproduce the reported behavior. I used Docker to containerize my application and I can use the lambda client without any issues.

It's possible that your lock file has multiple copies of the SDK pinned at different versions.

If you want to inspect it you can try this:

npm ls @aws-sdk/client-lambda

Or let npm resolve the dependency tree by:

cd your-project-root
rm -rf node_modules && rm package-lock.json && npm install

to regenerate your lock file.
Then rebuild your image and run it.

Let me know how it goes.

All the best, Ran~

[github-actions[bot]]

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.

[seawatts]

I'm seeing the same issue. I don't believe this should be closed.

I reverted back to 3.4.28.0 and it seems to work again.

[ytimenkov]

I got this issue too. Turned out I had one of my local modules called aws (guess what it's responsible for 😁). So the line export * from "./aws"; in @aws-sdk/util-endpoints picked the wrong one. And the function which was not found was aws.partition. (Maybe it's the side effect of bundling ncc).

[seawatts]

Interesting. I don't think that is the issue I'm seeing.

[willmcclellan]

Seeing the same issue here but tracing to the client-secrets-manager:

@[email protected]/node_modules/@aws-sdk/client-secrets-manager/dist-es/endpoint/endpointResolver.js:4:12

[nagarjuna993]

Had the same issue with @aws-sdk/client-sts reverting to 3.428.0 fixed the issue for now.

[RanVaknin]

Hi folks,

Thanks for staying active on the thread.

Did any of you try the solution I included in this comment? If that didnt help can you please include a minimal repro code, or even better - a small repository that outlines the issue?

That will help me investigate this further.

Thanks, Ran~

[boenhoff]

From my experience it appears to be a bundling issue. I have experienced a similar issue with the @aws-sdk/client-dynamodb client. I'm using projen and esbuild to bundle my lambda locally with external packages @aws-sdk/* which are available within the lamba runtime.

Checking the packages inside the lambda runtime locally with docker run --rm -it --entrypoint /bin/bash amazon/aws-lambda-nodejs:20, cd /var/runtime, npm ls you can see that there are pretty old versions like @aws-sdk/[email protected]. But my bundler is using @aws-sdk/[email protected]. This version mismatch appears to create the error, at least on my side. Bundling everything together without external packages resolves the issue.

[ytimenkov]

@RanVaknin

Did any of you try the solution I included in this comment?

I did. But I solved my issue with a bit of persistence and printf debugging 😁 as I wrote, the problem was that ncc picked a wrong module (i.e. just skipped the right one). This was pretty clear from reading the resulting code. Others may have a different root cause...

[kopertop]

I just ran into this issue when upgrading to [email protected] which upgraded to [email protected].

I also have a module called aws (packages/core/aws to be specific) but renaming this to something else didn't have any impact. @ytimenkov how exactly did you fix the issue for you scenario? did you just remove your custom AWS module entirely, or rename it and it still worked?

[pedro-pina-cabrera]

Had the same issue with @aws-sdk/client-sts reverting to 3.428.0 fixed the issue for now.

This works for me too, I clean the package-lock.json, upgrade version and reinstall.

[sheeni17]

I just ran into this issue when upgrading to [email protected] which upgraded to [email protected].

I also have a module called aws (packages/core/aws to be specific) but renaming this to something else didn't have any impact. @ytimenkov how exactly did you fix the issue for you scenario? did you just remove your custom AWS module entirely, or rename it and it still worked?

Hey, I'm on the same boat, did you manage to find a solution?

[Makiavelo]

@sheeni17 What worked for me was to revert sst to version 2.35.1

[seawatts]

I got this issue too. Turned out I had one of my local modules called aws (guess what it's responsible for 😁). So the line export * from "./aws"; in @aws-sdk/util-endpoints picked the wrong one. And the function which was not found was aws.partition. (Maybe it's the side effect of bundling ncc).

The aws call I'm trying to make is getSignedUrl from @aws-sdk/s3-request-presigner and PutObjectCommand from @aws-sdk/client-s3

I don't have the same export * from "./aws" in my project. However, when I debug the program, it fails here: https://github.com/smithy-lang/smithy-typescript/blob/main/packages/util-endpoints/src/utils/callFunction.ts#L16 The function that is trying to be passed in is fn = aws.partition, aws.isVirtualHostableS3Bucket, and aws.parseArn The available functions to call are

booleanEquals
getAttr
isSet
isValidHostLabel
not
parseURL
normalizedPath
stringEquals
substring
uriEncode

The total ruleset is defined here: https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-s3/src/endpoint/ruleset.ts

This is where one of the conditions aws.partition is defined in that file https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-s3/src/endpoint/ruleset.ts#L29

The rule condition is coming from many rules, here are a couple.

• Partition does not support FIPS
• Unrecognized hardware type: "Expected hardware type o or e but got {hardwareType}"
• Invalid ARN: The outpost Id must only contain a-z, A-Z, 0-9 and -.

Going back in the history of that file, that condition has always been there. The smithy client must not be able to handle those functions, but the old utils must have been able to. I haven't dove into the old utils that shipped with this package. I believe the change happened with this PR https://github.com/aws/aws-sdk-js-v3/pull/5390

It looks like the latest version of the @aws-sdk/client-s3 that works before that pull request was 3.437.0

[anthgur]

In case this is helpful to someone else, I ran into a similar endpointFunctions[fn] is not a function issue at build time when I installed the latest @aws-sdk/client-ses version into an existing project. I was able to resolve the issue by setting all aws client packages for my project to the same 3.x minor version.

[josereymondaguilar]

Got same issue when using version 3.481.0, seems the recent 3.490.0 works. Both @aws-sdk/client-sns, @aws-sdk/client-sqs, @aws-sdk/client-s3 in same 490 version and that magically worked! 🤔

[changchang]

I have the same issue as https://github.com/aws/aws-sdk-js-v3/issues/5435#issuecomment-1854878590. It failed at handling fn = aws.partition.

Investigation

I can see the aws.partition is set up at https://github.com/aws/aws-sdk-js-v3/blob/main/packages/util-endpoints/src/aws.ts#L13

The logic is supposed to hit the branch https://github.com/smithy-lang/smithy-typescript/blob/main/packages/util-endpoints/src/utils/callFunction.ts#L13. But it hit https://github.com/smithy-lang/smithy-typescript/blob/main/packages/util-endpoints/src/utils/callFunction.ts#L16

From my local error stack: at callFunction (/<project-root-path>/node_modules/@aws-sdk/client-sts/node_modules/@smithy/util-endpoints/dist-cjs/index.js:271:31)

I can see there was a @smithy under @aws-sdk/client-sts/node_modules.

I believe this error is caused by the way we register customEndpointFunctions.aws. We update the customEndpointFunctions.aws from @smithy in aws-sdk.

I use sst to manage my stacks on top of CDK. They specify the @smithy/util-endpoints version as 1.0.5 in the latest sst version (2.39.6). https://github.com/sst/sst/commit/ff3abe36516b0e110978390178fe9c17fbac294e#diff-6a1935fc5ad82cc9ecc6277e01331eda95c2f1494e465569e5c325edcc8e596fR66 But the version in aws-sdk is 1.1.0.

Because of the version conflicts, @aws-sdk/client-sts and @aws-sdk/util-endpoints have their own @smithy/util-endpoints. Here is the layout for my local dependencies.

.
└── node_modules
    ├── @aws-sdk
    │   ├── client-sts
    │   │   └── node_modules
    │   │       └── @smithy
    │   │           └── util-endpoints (1.1.0)
    │   └── util-endpoints
    │       └── node_modules
    │           └── @smithy
    │               └── util-endpoints (1.1.0)
    └── @smithy
        └── util-endpoints (1.0.5, from sst)

In client-sts We import @aws-sdk/util-endpoints to register customerEndpointFunctions. (https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-sts/src/index.ts#L19) The customerEndpointFunctions is stored in @aws-sdk/util-endpoints/node_modules/@smithy/util-endpoints.

But when we resolve the endpoint, we import the resolveEndpoint function from @smithy/util-endpoints directly. (https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-sts/src/endpoint/endpointResolver.ts#L12) Then, this resolveEndpoint function will access customerEndpointFunctions in @aws-sdk/client-sts/node_modules/@smithy/util-endpoints, which is different from the one we registered above. Then we will get an empty customerEndpointFunctions.

Possible Solution

So one possible solution is to re-export resolveEndpoint in @aws-sdk/util-endpoints since we register customEndpointFunctions in it. The call stack will look like: @aws-sdk/client-sts -> @aws-sdk/util-endpoints -> @smithy/util-endpoints. But, ideally, maybe we should find a better way to manage the customEndpointFunctions. 🤔

For a temp workaround, I revert sst to the previous version (2.39.5), delete the package-lock.json and node_modules, and re-run the build. It can work. The extra @smithy/util-endpoints in sub-modules are gone. That means we should make sure we are using the same version for the @smithy/util-endpoints in our project.

If I resume sst 2.39.6, delete the package-lock.json and node_modules, and re-build, I can replicate the error.

Hope this can help with your investigation. @RanVaknin

[jfirebaugh]

I can confirm that this issue arises when package version resolution results in multiple versions of @smithy/util-endpoints being used at runtime. The way @smithy/util-endpoints is currently architected, with a global custom endpoint function registry, makes it unsafe to have multiple copies in use simultaneously. This should be fixed in aws-sdk.

In the meantime, a workaround is to use overrides (npm), pnpm.overrides (pnpm), or resolutions (yarn) to force @smithy/util-endpoints to be resolved to a single version only.

[janvorwerk]

I have the same issue here... when I upgrade my project from Next.js 13 to Next.js 14. Unfortunately, I am trying to reproduce it on a small project, but there, it works like a charm so far 😭

Unlike the comments above, I only have a single version of @smithy/util-endpoints => 1.1.1.

However, I am using a monorepo with pnpm workspaces.

And the fun fact is that it works well with Next.js 13.5.6 but not when I upgrade only my next dependency to 14.1.0. I wonder if some bundler options somehow could also generate two occurrences of customEndpointFunctions??

Here is the call-stack if this can help...

    stack: 'TypeError: endpointFunctions[fn] is not a function\n' +
      '    at callFunction (webpack-internal:///(rsc)/../../node_modules/.pnpm/@[email protected]/node_modules/@smithy/util-endpoints/dist-cjs/index.js:258:33)\n' +
      '    at evaluateCondition (webpack-internal:///(rsc)/../../node_modules/.pnpm/@[email protected]/node_modules/@smithy/util-endpoints/dist-cjs/index.js:266:19)\n' +
      '    at evaluateConditions (webpack-internal:///(rsc)/../../node_modules/.pnpm/@[email protected]/node_modules/@smithy/util-endpoints/dist-cjs/index.js:283:38)\n' +
      '    at evaluateErrorRule (webpack-internal:///(rsc)/../../node_modules/.pnpm/@[email protected]/node_modules/@smithy/util-endpoints/dist-cjs/index.js:383:41)\n' +
      '    at evaluateRules (webpack-internal:///(rsc)/../../node_modules/.pnpm/@[email protected]/node_modules/@smithy/util-endpoints/dist-cjs/index.js:419:13)\n' +
      '    at evaluateTreeRule (webpack-internal:///(rsc)/../../node_modules/.pnpm/@[email protected]/node_modules/@smithy/util-endpoints/dist-cjs/index.js:402:12)\n' +
      '    at evaluateRules (webpack-internal:///(rsc)/../../node_modules/.pnpm/@[email protected]/node_modules/@smithy/util-endpoints/dist-cjs/index.js:421:41)\n' +
      '    at evaluateTreeRule (webpack-internal:///(rsc)/../../node_modules/.pnpm/@[email protected]/node_modules/@smithy/util-endpoints/dist-cjs/index.js:402:12)\n' +
      '    at evaluateRules (webpack-internal:///(rsc)/../../node_modules/.pnpm/@[email protected]/node_modules/@smithy/util-endpoints/dist-cjs/index.js:421:41)\n' +
      '    at resolveEndpoint (webpack-internal:///(rsc)/../../node_modules/.pnpm/@[email protected]/node_modules/@smithy/util-endpoints/dist-cjs/index.js:452:22)\n' +
      '    at Object.defaultEndpointResolver [as endpointProvider] (webpack-internal:///(rsc)/../../node_modules/.pnpm/@[email protected]/node_modules/@aws-sdk/client-s3/dist-es/endpoint/endpointResolver.js:11:83)\n' +
      '    at getEndpointFromInstructions (webpack-internal:///(rsc)/../../node_modules/.pnpm/@[email protected]/node_modules/@smithy/middleware-endpoint/dist-cjs/index.js:136:35)\n' +
      '    at async eval (webpack-internal:///(rsc)/../../node_modules/.pnpm/@[email protected]/node_modules/@smithy/middleware-endpoint/dist-cjs/index.js:172:30)\n' +
      '    at async eval (webpack-internal:///(rsc)/../../node_modules/.pnpm/@[email protected]/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:107:28)\n' +
      '    at async eval (webpack-internal:///(rsc)/../../node_modules/.pnpm/@[email protected]/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:132:24)\n' +
      '    at async eval (webpack-internal:///(rsc)/../../node_modules/.pnpm/@[email protected]/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:40:34)\n' +
      '    at async SkS3Bucket.exists (webpack-internal:///(rsc)/../../packages/aws/s3.ts:45:30)\n' +

[fones]

Got the same issue, when I upgrade @aws-sdk/client-dynamodb to 3.503.1. I have removed node_modules and package-loc.json file and install again. Now works fine.

[jcollum-nutrien]

When I ran into this issue it was due to a mismatch of some sort in the versions of aws packages. I don't have more detail, I just bumped an sts package from 496 to 498 or something similar and ran yarn and it started working. rimraf your node modules dir and update all your aws packages to the current version -- that's the only advice I have here.

[DarrylBrooks97]

Can confirm this happens when upgrading from Next 13.5.6 -> 14.1.0 🥲

[janvorwerk]

Can confirm this happens when upgrading from Next 13.5.6 -> 14.1.0 🥲

@DarrylBrooks97 what is your setup? Let's try to find what our two cases have in common to try and create a small project for reproduction. If you agree, of course!

I am using a pnpm monorepo. The structure is as follows:

The root is in a folder called sk:

root project => .../sk

From there, I have two Next.js projects and one Expo/React Native project:

@sk/next_project_1 => .../sk/apps/next_project_1
@sk/next_project_2 => .../sk/apps/next_project_2
@sk/expo_project => .../sk/apps/expo_project

All these depend on packages listed below... one of which is my AWS wrapper which depends on

    "@aws-sdk/client-s3": "^3.499.0",
    "@aws-sdk/client-sns": "^3.499.0",

My shared packages:

  @sk/api => .../sk/packages/api
  @sk/assets => .../sk/packages/assets
  @sk/auth => .../sk/packages/auth
  @sk/aws => .../sk/packages/aws       <=== AWS dep here
  @sk/business => .../sk/packages/business
  @sk/context => .../sk/packages/context
  @sk/drizzle => .../sk/packages/drizzle
  @sk/fit => .../sk/packages/fit
  @sk/i18n => .../sk/packages/i18n
  @sk/logging => .../sk/packages/logging
  @sk/notifications => .../sk/packages/notifications
  @sk/schema => .../sk/packages/schema
  @sk/utils => .../sk/packages/utils
  @sk/webview => .../sk/packages/webview

  @sk/eslint-config => .../sk/tooling/eslint
  @sk/prettier-config => .../sk/tooling/prettier
  @sk/tailwind-config => .../sk/tooling/tailwind
  @sk/tsconfig => .../sk/tooling/typescript

We could try and send the output of pnpn ls -r --depth=0 --json... but for now, I am afraid that this might be a bit overwhelming.

[seawatts]

I can confirm that this issue arises when package version resolution results in multiple versions of @smithy/util-endpoints being used at runtime. The way @smithy/util-endpoints is currently architected, with a global custom endpoint function registry, makes it unsafe to have multiple copies in use simultaneously. This should be fixed in aws-sdk.

In the meantime, a workaround is to use overrides (npm), pnpm.overrides (pnpm), or resolutions (yarn) to force @smithy/util-endpoints to be resolved to a single version only.

Pinning the package to 1.1.2 worked for me:

// package.json

"pnpm": {
    "overrides": {
      "@smithy/util-endpoints": "1.1.2"
    }
  },

pnpm why @smithy/util-endpoints

sst 2.40.3
└─┬ @aws-sdk/client-cloudformation 3.511.0
└─┬ @aws-sdk/client-sts 3.511.0
  └─┬ @aws-sdk/credential-provider-node 3.511.0 peer
    ├─┬ @aws-sdk/credential-provider-ini 3.511.0
    │ └─┬ @aws-sdk/credential-provider-sso 3.511.0
    │   ├─┬ @aws-sdk/client-sso 3.511.0
    │   │ ├─┬ @aws-sdk/middleware-user-agent 3.511.0
    │   │ │ └─┬ @aws-sdk/util-endpoints 3.511.0
    │   │ │   └── @smithy/util-endpoints 1.1.2
    │   │ ├─┬ @aws-sdk/util-endpoints 3.511.0
    │   │ │ └── @smithy/util-endpoints 1.1.2
    │   │ └── @smithy/util-endpoints 1.1.2
    │   └─┬ @aws-sdk/token-providers 3.511.0
    │     └─┬ @aws-sdk/client-sso-oidc 3.511.0
    │       ├─┬ @aws-sdk/middleware-user-agent 3.511.0
    │       │ └─┬ @aws-sdk/util-endpoints 3.511.0
    │       │   └── @smithy/util-endpoints 1.1.2
    │       ├─┬ @aws-sdk/util-endpoints 3.511.0
    │       │ └── @smithy/util-endpoints 1.1.2
    │       └── @smithy/util-endpoints 1.1.2
    └─┬ @aws-sdk/credential-provider-sso 3.511.0
      ├─┬ @aws-sdk/client-sso 3.511.0
      │ ├─┬ @aws-sdk/middleware-user-agent 3.511.0
      │ │ └─┬ @aws-sdk/util-endpoints 3.511.0
      │ │   └── @smithy/util-endpoints 1.1.2
      │ ├─┬ @aws-sdk/util-endpoints 3.511.0
      │ │ └── @smithy/util-endpoints 1.1.2
      │ └── @smithy/util-endpoints 1.1.2
      └─┬ @aws-sdk/token-providers 3.511.0
        └─┬ @aws-sdk/client-sso-oidc 3.511.0
          └─┬ @aws-sdk/middleware-user-agent 3.511.0
            └─┬ @aws-sdk/util-endpoints 3.511.0
              └── @smithy/util-endpoints 1.1.2

[janvorwerk]

I have the same issue here... when I upgrade my project from Next.js 13 to Next.js 14. Unfortunately, I am trying to reproduce it on a small project, but there, it works like a charm so far 😭

I have a reproducible test case ✅ 🚀

Same repo as earlier, but I've modified the setup to a pnpm monorepo https://github.com/janvorwerk/aws-s3-demo

• Works like a charm in Next.js 13
• Updating to Next.js 14 produces the same error!

All hints about multiple versions of @smithy/util-endpoints fall short in my case... I suppose the packaging induces multiple instances, but there is only one single version in my depenencies... pinning a version does not help!

[coronapl]

I have the same issue here... when I upgrade my project from Next.js 13 to Next.js 14. Unfortunately, I am trying to reproduce it on a small project, but there, it works like a charm so far 😭

I have a reproducible test case ✅ 🚀

Same repo as earlier, but I've modified the setup to a pnpm monorepo https://github.com/janvorwerk/aws-s3-demo

• Works like a charm in Next.js 13
• Updating to Next.js 14 produces the same error!

All hints about multiple versions of @smithy/util-endpoints fall short in my case... I suppose the packaging induces multiple instances, but there is only one single version in my depenencies... pinning a version does not help!

@janvorwerk We are also facing the same issue. We tried pinning the version, but no success.

[janvorwerk]

@janvorwerk We are also facing the same issue. We tried pinning the version, but no success.

What worked for me @coronapl is to remain on an older AWS SDK version as mentionned in the bug description.

[galer7]

If you use SST: npx sst update and then pnpm install. Docs on sst update: https://docs.sst.dev/packages/sst#sst-update

[RanVaknin]

Hi everyone on the thread.

Thank you for your patience and for participating in the discussion.

The problem that was initially reported was caused because of multiple copies of @smithy/util-endpoints used in different versions by the application. The solution for that was to force the package manager to align the versions by deleting node_modules and the lock files, and then running npm install again.

For some of you using Next 14, I see this is still an issue. We think this is likely a problem with the bundler Next 14 uses or with the build system itself. Issue with Next14 is now tracked here. Based on the investigation done by @changchang, sst is impacted the same way.

Since there are multiple scenarios in which this is manifesting in, some are solved with a simple deduping of versions, some are not I'm going to close this thread, and ask that if none of the solutions on this thread work out for you, please create a new separate issue and include a minimal repro code, preferably in the form of a github repo.

Thanks, Ran~