aws-connected-device-framework icon indicating copy to clipboard operation
aws-connected-device-framework copied to clipboard

Published 20 hours ago • verified publisher icon aws

Fix assetlibrary history batching

Open TonySherman opened this issue 1 year ago • 12 comments

Description

As @aaronatbissell mentioned in #87 - Large amounts of assetlibrary changes can cause hitting lambda concurrency limits. This proposed change utilizes a Kinesis Stream to batch updates to batches of 100 or 30 second window (These can be configured differently if needed)

Type of change

  • [x] Bug fix (non-breaking change which fixes an issue)
  • [ ] New feature (non-breaking change which adds functionality)
  • [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • [ ] Refactor (existing code being refactored)
  • [ ] This change includes a documentation update

Submission Checklist

  • [x] Build Verified
  • [x] Bundle Verified
  • [x] Lint passing
  • [x] Unit tests passing
  • [ ] Integration tests passing
  • [x] Change logs generated
Additional Notes:

TonySherman avatar Oct 24 '22 18:10 TonySherman

Any questions or feedback on this PR that I can help to answer?

aaronatbissell avatar Nov 18 '22 21:11 aaronatbissell

This PR has been sitting for quite a while. Any chance this is even on the radar to get merged in?

@hassankhokhar @boardthatpowder @rrangnekar

aaronatbissell avatar Jan 05 '23 13:01 aaronatbissell

I see there's some movement on this repo again! Is there an appetite to merge in this Pull request (or any other community pull requests)?

@ts-amz @canavandl @BenjiTheC

aaronatbissell avatar May 18 '23 17:05 aaronatbissell

One last effort here to get any kind of feedback from anyone on the team, otherwise we will just fork this and manage it on our own. Is this even worth me going through and fixing all this stuff so that we can get this in?

aaronatbissell avatar Aug 16 '23 13:08 aaronatbissell

@aaronatbissell, as with the other PR, we have resumed reviewing PRs and will be happy to review this once you've updated the conflicts. I am sorry for the delay in response, as we were undergoing a team change.

ts-amz avatar Aug 16 '23 20:08 ts-amz

Hello @aaronatbissell, are you planning to work on this PR? We plan to close this on 9/8/23 for tracking purposes if it's not still active (in that case, please re-open it if/when you have update it). Thank you

ts-amz avatar Sep 07 '23 04:09 ts-amz

Yes - I do plan on working this. I just don't have the time to do it right now. It will probably be in a month or so

aaronatbissell avatar Sep 07 '23 12:09 aaronatbissell

@ts-amz - rebased on main

aaronatbissell avatar Sep 17 '23 14:09 aaronatbissell

@aaronatbissell do you know how we can test this before merging?

canavandl avatar Oct 12 '23 15:10 canavandl

@canavandl This is going back a bit for me and I no longer use cdf but I believe @aaronatbissell and I were able to deploy these changes to a lower environment and make sure that asset library history table was being updated with changes. It might be more difficult to test large batches. Maybe set the BatchSize and MaximumBatchingWindowInSeconds to a very low number to force updates to get batched?

TonySherman avatar Oct 24 '23 18:10 TonySherman

@ts-amz @canavandl - FYI, just made a couple more updates to this. Unit tests are included with #193. We've pushed this out to our system to ensure it's working. I think this is ready for an in-depth review and merge.

aaronatbissell avatar Feb 28 '24 20:02 aaronatbissell

After running for a few days in our production environment, this asset library history batching has reduced our total lambda duration by about 60% and reduced invocations by 99%.

aaronatbissell avatar Mar 20 '24 15:03 aaronatbissell