Pinpoint: Exceeded maximum endpoint per user count: 15

[joebernard]

Describe the bug After upgrading to "@aws-amplify/analytics": "^4.0.0" push notifications have stopped working. I can no longer update endpoints. This was originally solved in #5423 but seems to have reappeared recently. Possibly related to the merge in #7245 .

To Reproduce Install the app on a device more than 10 times.

Expected behavior Amplify should clear old endpoints as mentioned in the docs.

Environment

System:
    OS: macOS 10.15.7
    CPU: (12) x64 Intel(R) Core(TM) i9-8950HK CPU @ 2.90GHz
    Memory: 208.17 MB / 32.00 GB
    Shell: 3.2.57 - /bin/bash
  Binaries:
    Node: 14.2.0 - ~/.nvm/versions/node/v14.2.0/bin/node
    Yarn: 1.22.5 - /usr/local/bin/yarn
    npm: 6.14.8 - ~/.nvm/versions/node/v14.2.0/bin/npm
    Watchman: 4.9.0 - /usr/local/bin/watchman
  Browsers:
    Chrome: 86.0.4240.198
    Firefox: 82.0.3
    Safari: 14.0
  npmPackages:
    @apollo/client: ^3.2.7 => 3.2.7 
    @aws-amplify/analytics: ^4.0.0 => 4.0.0 
    @aws-amplify/auth: ^3.4.12 => 3.4.12 
    @aws-amplify/cache: ^3.1.37 => 3.1.37 
    @aws-amplify/core: ^3.8.4 => 3.8.4 
    @aws-amplify/storage: ^3.3.12 => 3.3.12 
    @babel/core: ^7.8.4 => 7.11.6 
    @babel/runtime: ^7.8.4 => 7.9.2 
    @bugsnag/react-native: ^7.5.2 => 7.5.2 
    @expo/react-native-action-sheet: ^3.8.0 => 3.8.0 
    @react-native-community/art: ^1.2.0 => 1.2.0 
    @react-native-community/async-storage: ^1.12.1 => 1.12.1 
    @react-native-community/eslint-config: ^1.1.0 => 1.1.0 
    @react-native-community/masked-view: ^0.1.10 => 0.1.10 
    @react-native-community/netinfo: ^5.9.7 => 5.9.7 
    @react-native-community/picker: ^1.8.1 => 1.8.1 
    @react-native-community/push-notification-ios: 1.7.3 => 1.7.3 
    @types/react: 16.9.56 => 16.9.56 
    amazon-cognito-identity-js: ^4.5.5 => 4.5.5 
    apollo-link-retry: ^2.2.16 => 2.2.16 
    appcenter: 3.1.2 => 3.1.2 
    appcenter-analytics: 3.1.2 => 3.1.2 
    appcenter-crashes: 3.1.2 => 3.1.2 
    aws-appsync-auth-link: ^3.0.2 => 3.0.2 
    aws-appsync-subscription-link: ^3.0.3 => 3.0.3 
    babel-jest: ^25.1.0 => 25.5.1 
    eslint: ^6.5.1 => 6.8.0 
    exponential-backoff: ^3.1.0 => 3.1.0 
    graphql: 15.4.0 => 15.4.0 
    jest: ^25.1.0 => 25.5.4 
    lodash.debounce: ^4.0.8 => 4.0.8 
    lodash.throttle: ^4.1.1 => 4.1.1 
    metro-react-native-babel-preset: ^0.59.0 => 0.59.0 
    moment: ^2.29.1 => 2.29.1 
    moment-timezone: ^0.5.32 => 0.5.32 
    prop-types: ^15.7.2 => 15.7.2 
    react: 16.13.1 => 16.13.1 
    react-dom: ^16.12.0 => 16.13.1 
    react-native: 0.63.3 => 0.63.3 
    react-native-animatable: ^1.3.3 => 1.3.3 
    react-native-camera: 3.40.0 => 3.40.0 
    react-native-code-push: ^6.4.0 => 6.4.0 
    react-native-config: 1.4.0 => 1.4.0 
    react-native-country-picker-modal: ^2.0.0 => 2.0.0 
    react-native-device-info: ^7.1.0 => 7.1.0 
    react-native-fast-image: ^8.3.4 => 8.3.4 
    react-native-fs: ^2.16.6 => 2.16.6 
    react-native-gesture-handler: ^1.8.0 => 1.8.0 
    react-native-get-random-values: ^1.5.0 => 1.5.0 
    react-native-haptic-feedback: ^1.11.0 => 1.11.0 
    react-native-orientation-locker: ^1.2.0 => 1.2.0 
    react-native-permissions: ^2.2.2 => 2.2.2 
    react-native-progress: ^4.1.2 => 4.1.2 
    react-native-reanimated: ^1.13.2 => 1.13.2 
    react-native-root-toast: ^3.2.1 => 3.2.1 
    react-native-safe-area-context: ^3.1.9 => 3.1.9 
    react-native-safe-area-view: ^2.0.0 => 2.0.0 
    react-native-screens: ^2.15.0 => 2.15.0 
    react-native-share: 4.1.0 => 4.1.0 
    react-native-svg: ^12.1.0 => 12.1.0 => 0.3.4 
    react-native-tab-view: ^2.15.2 => 2.15.2 
    react-native-vector-icons: 7.1.0 => 7.1.0 
    react-native-video: ^5.1.0-alpha8 => 5.1.0-alpha8 
    react-native-webview: ^10.10.2 => 10.10.2 
    react-navigation: ^4.4.3 => 4.4.3 
    react-navigation-props-mapper: ^1.0.0 => 1.0.4 
    react-navigation-stack: ^2.10.1 => 2.10.1 
    react-navigation-tabs: ^2.10.1 => 2.10.1 
    react-test-renderer: 16.13.1 => 16.13.1 
    uuid: ^8.3.1 => 8.3.1 
  npmGlobalPackages:
    @aws-amplify/cli: 4.32.1
    appcenter-cli: 2.7.2
    gatsby-cli: 2.14.0
    lerna: 3.22.1
    npm: 6.14.8
    serverless: 2.11.1
    typescript: 3.9.6

Additional context May be related to #7245

[jpignata]

Thanks @joebernard! Are you seeing this locally in development during test or in production use?

[joebernard]

@jpignata I've only been able to test in development so far but the previous time this happened it was affecting production too. This only started breaking again after upgrading to 4.0.0. I tried rolling back but then started experiencing a different issue (#6145). I can't seem to get around that one either as it seems to persist through re-installs so I'm currently blocked.

[v1pz3n]

Also here

[v1pz3n]

Amplify updated to 3.3.10 and it worked

[joebernard]

Interesting. Maybe this is only broken in the modular packages. Unfortunately I cannot easily convert.

[njdullea]

I am not using modular packages, was on amplify 3.3.9 and encountered this issue. Updating to 3.3.10 did not resolve the issue.

Update: reverting to 3.3.8 resolved it for now.

[joebernard]

Can we get an update on how to proceed now that we are stuck with this artificial endpoint limit? Are we expected to manually manage this limit within Pinpoint somehow? What was the rationale behind removing this functionality? This has become a huge blocker for my team.

[jpignata]

Hi @joebernard - we're working on this with Pinpoint and will have more to share soon. Do you have anytime to chat tomorrow on discord perhaps? I'd like to learn more about the block and see if we can help. Happy Thanksgiving!

[joebernard]

@jpignata That would be helpful. I can meet anytime between 9:00 AM and 2:00 PM ET Nov 27. Happy Thanksgiving to you as well.

[sammartinez]

@joebernard Are you ok if we resolve this issue? Just wanted to validate if there were any other issues you were facing after our conversation last week. Please let us know

[joebernard]

Hi @sammartinez, thanks for asking. I would consider my case temporarily mitigated by rolling back, but the underlying issue re-introduced in @aws-amplify/analytics": "4.0.0" is still a problem. We uninstall / re-install apps so often that we quickly hit that 10 endpoint limit. It could also affect our production users though it would be more of an edge case. With no way for us to manage endpoints, this issue effectively breaks push notifications.

To me, this is a showstopper that precludes us from upgrading to 4.0.0 and beyond. We're locked into specific versions and might not be able to benefit from future features or security patches. I feel we either need some mechanism to manually manage endpoints, automated management of endpoints in Amplify or (preferably) in Pinpoint, or other guidance that would allow us to continue staying current with Amplify packages while avoiding this issue.

[v1pz3n]

I thought it worked but it was only in the web version. For some reason, my react native on android and ios has the pinpoint error.

[dylan-westbury]

Hi @sammartinez, thanks for asking. I would consider my case temporarily mitigated by rolling back, but the underlying issue re-introduced in @aws-amplify/analytics": "4.0.0" is still a problem. We uninstall / re-install apps so often that we quickly hit that 10 endpoint limit. It could also affect our production users though it would be more of an edge case. With no way for us to manage endpoints, this issue effectively breaks push notifications.

I believe new endpoints not only occur when you uninstall apps, but just by clearing local storage.

For us, our big concern is that iOS users will stop receiving push notifications as we can't update the endpoint with the address with the new token.

[blacksheep--]

This is now the 5. incarnation of the very same topic. It seems this problem keeps popping up every other release. I guess it may be due to the fragile nature of the underlying code, where the returned message is string-checked to be starting with 'Exceeded maximum endpoint per user count' in AWSPinpointProvider.ts#461:

!String(message).startsWith('Exceeded maximum endpoint per user count')

A slight change of the message kills it, which already happened in the past.

Also OP seems absolutely right with his assumption that PR #7245 removed the clean-up of unused endpoints which worked perfectly before. How is this supposed to be handled?

[paddlefish]

Here's a manual workaround you can do using the CLI for each affected user:

step 1. Use cognito to search for the affected users sub

in this example i found user 719f397f-c6a6-4598-b603-7cc4d9d17911 also use pinpoint console to look up your analytics application id.

step 2. see how many endpoints are in pinpoint

aws --profile YourProfile pinpoint get-user-endpoints --user-id 719f397f-c6a6-4598-b603-7cc4d9d17911 --application-id daafcc04059a444aa9363e484113c189 { "EndpointsResponse": { "Item": [ { "ApplicationId": "daafcc04059a444aa9363e484113c189", "Attributes": {}, "CohortId ...

count them .. here i'm using jq and wc to count

➜ aws --profile YourProfile pinpoint get-user-endpoints --user-id 719f397f-c6a6-4598-b603-7cc4d9d17911 --application-id daafcc04059a444aa9363e484113c189| jq '.EndpointsResponse.Item[]|.Id' | wc 10 10 390

yup there are 10. so this user can't sign in due to this 10 endpoint restriction

step 3. Clear the oldest endpoints

3a. find the oldest ones. I scanned the list printed in step 2 and had manually found the newest endpoint creation date ('2020-12-18T19:45:48.212Z') to put in this expression:

 aws --profile YourProfile pinpoint get-user-endpoints --user-id 719f397f-c6a6-4598-b603-7cc4d9d17911 --application-id daafcc04059a444aa9363e484113c189| jq '.EndpointsResponse.Item[]|select(.CreationDate | contains("2020-12-18T19:45:48.212Z") | not)|.Id'

"3aa4dc90-2501-11eb-a16e-99d12931a92e" "e5cd9240-276f-11eb-af3c-8130a9b73a99" "3f66e740-2dae-11eb-aae4-153b561a5e3b" "35b34700-2513-11eb-a296-0558d678e564" "7ddf5e70-29cc-11eb-bdbf-671aec7e1f1f" "94427fe0-3038-11eb-bf01-5b5367143dd5" "2cbe3c00-3296-11eb-a617-23713ff6a6ee" "50a2f3a0-357e-11eb-a551-958abb691880" "3928cc60-3964-11eb-9bc3-bdabe9b761f6"

3b. use xargs to delete them

aws --profile YourProfile pinpoint get-user-endpoints --user-id 719f397f-c6a6-4598-b603-7cc4d9d17911 --application-id daafcc04059a444aa9363e484113c189| jq '.EndpointsResponse.Item[]|select(.CreationDate | contains("2020-12-18T19:45:48.212Z") | not)|.Id' -r | xargs -n 1 aws --profile YourProfile pinpoint delete-endpoint --application-id daafcc04059a444aa9363e484113c189 --endpoint-id { "EndpointResponse": { /* nine responses */

[dylan-westbury]

Receiving amplify security notification in AWS console but can't update due to this issue, we must stay on previous version otherwise user will eventually hit 10 endpoints and will no longer receive push notifications.

[joebernard]

@sammartinez Has there been any progress on this "exceeded maximum endpoint per user" issue"?

@dylan-westbury Which version of aws-amplify did you find avoids this issue? I'm using modules now but want to try that package instead.

[dylan-westbury]

Hi @joebernard

We upgraded aws-amplify and noticed this issue, so we downgraded back to the last working version we had within the app, which was:

"aws-amplify": "3.3.8",

We no longer received the "exceeded maximum endpoint per user" once we downgraded.

[maoosi]

I've recently updated to [email protected] (linking to @aws-amplify/[email protected]) and started having a similar errors showing in the console Exceeded maximum endpoint per user count: 15.

Would be nice to get someone from the amplify team commenting on the issue. Is there any config change that we need to apply in order to resolve this? Or is it a bug that need to be reverted as per suggested in https://github.com/aws-amplify/amplify-cli/pull/5918?

[blacksheep--]

Any news on this issue? We still suffer from this issue.

[joebernard]

I've realized that this issue is a critical security vulnerability. The version of aws-amplify that mitigates the Exceed maximum endpoint per user problem discussed in this issue is 3.3.8. That version of aws-amplify exposes two security problems.

1. @aws-amplify/datastore contains a dependency to [email protected]. That package contains an exploit documented in CVE-2020-28477.

2. @aws-amplify/api-rest and @aws-amplify/storage depend on [email protected]. The vulnerability in that package is documented in CVE-2020-28168.

The affected dependencies have been upgraded to patched versions in the latest release of @aws-amplify, however we are stuck on 3.3.8 because of the endpoint issue.

@sammartinez This is a show-stopping issue for us. Can you please provide some update on progress and/or a mitigation strategy we can implement today?

[sammartinez]

I've realized that this issue is a critical security vulnerability. The version of aws-amplify that mitigates the Exceed maximum endpoint per user problem discussed in this issue is 3.3.8. That version of aws-amplify exposes two security problems.

1. @aws-amplify/datastore contains a dependency to [email protected]. That package contains an exploit documented in CVE-2020-28477.
2. @aws-amplify/api-rest and @aws-amplify/storage depend on [email protected]. The vulnerability in that package is documented in CVE-2020-28168.

The affected dependencies have been upgraded to patched versions in the latest release of @aws-amplify, however we are stuck on 3.3.8 because of the endpoint issue.

@sammartinez This is a show-stopping issue for us. Can you please provide some update on progress and/or a mitigation strategy we can implement today?

Hey @joebernard, thanks for this callout. I will work with the team on seeing about getting an update to version 3.3.8 to update these dependencies specifically. Our ETA for this is later today. I will let you know once we update the version with the callouts above. As for the update on the mitigation, we are working with the Pinpoint team on this and will provide a timeline once we have one for you.

[manueliglesias]

Hey @joebernard

We just published a hotfix to npm, you can find it under these versions and the @prev-hotfixes dist-tag:

 - @aws-amplify/[email protected]
 - @aws-amplify/[email protected]
 - @aws-amplify/[email protected]
 - @aws-amplify/[email protected]
 - @aws-amplify/[email protected]
 - @aws-amplify/[email protected]
 - @aws-amplify/[email protected]
 - @aws-amplify/[email protected]
 - @aws-amplify/[email protected]
 - [email protected]
 - [email protected]
 - [email protected]
 - @aws-amplify/[email protected]
 - @aws-amplify/[email protected]
 - @aws-amplify/[email protected]
 - @aws-amplify/[email protected]
 - @aws-amplify/[email protected]
 - @aws-amplify/[email protected]
 - @aws-amplify/[email protected]
 - @aws-amplify/[email protected]
 - @aws-amplify/[email protected]

Please let us know if you have any issues with these versions

Thanks! :D

[joebernard]

Thanks @manueliglesias ! I will test this. I noticed the root version of the hotfix is 3.3.9. I'm on 3.3.8 right now, can you confirm this includes the 3.3.8 endpoint management discussed in this issue?

[manueliglesias]

hey @joebernard

I can confirm this includes the endpoint management code.

You can also check by yourself with, e.g.

diff <(curl -s https://unpkg.com/@aws-amplify/[email protected]/lib/Providers/AWSPinpointProvider.js) <(curl -s https://unpkg.com/@aws-amplify/[email protected]/lib/Providers/AWSPinpointProvider.js)

(This is comparing @aws-amplify/[email protected] and @aws-amplify/analytics@latest)

[nubpro]

Even the official doc has the same issue prompting in the console

Exceeded maximum endpoint per user count

[sammartinez]

All,

Just wanted to provide an update here to this issue. We are working internally with the Amazon Pinpoint team on this. While we are working on a solution, we may reach out to some of you to understand your use case more. Thank you for your patience on this!

[appdatasoft]

Any update ?

[dylan-westbury]

Hi @sammartinez, any update? We are pending release with an app and are just waiting on this to be resolved.

Thanks

[franrios]

Same issue here, waiting for the fix to publish an app.