powertools-lambda-python icon indicating copy to clipboard operation
powertools-lambda-python copied to clipboard

Feature request: Idempotent decorator for asynchronous functions

Open esthermcmahon opened this issue 1 year ago • 2 comments
trafficstars

Use case

Would love to be able to use the AsyncBatchProcessor and async_process_partial_response function along with the idempotency utility.

Solution/User Experience

def async_idempotent_function():
    def wrapper(func):
        @functools.wraps(func)
        async def wrapped(*args):
            return await func(*args)
        return wrapped
    return wrapper

Alternative solutions

No response

Acknowledgment

esthermcmahon avatar Jun 10 '24 21:06 esthermcmahon

Thanks for opening your first issue here! We'll come back to you as soon as we can. In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

boring-cyborg[bot] avatar Jun 10 '24 21:06 boring-cyborg[bot]

hey @esthermcmahon, thank you so much for taking the time to open this feature request. Are you using DynamoDB or Redis as a persistence storage for Idempotency?

The reason we haven't added support for async in idempotency yet is that the AWS SDK is synchronous. This means it'll be blocking and you wouldn't fully benefit from async I/O - e.g., calls to store/retrieve data from Amazon DynamoDB / Redis would block the event loop for a few milliseconds.


for maintainers and anyone looking to help implementing it :)

Off the top of my head, a new async decorator along with these two changes could make it possible (needs to be tested):

  • For DynamoDB. We'd need a new AsyncDynamoDBPersistenceLayer requiring aioboto3 as a dependency.
  • For Redis. We'd need a new AsyncRedisCachePersistenceLayer; Redis client supports asyncio so only syntax changes

PS: Adding help wanted to also signal we need to track customer demand. Personally, this is gonna become a common need in Data Engineering and large data pipelines.

heitorlessa avatar Jun 12 '24 08:06 heitorlessa

Closing this issue as it's been open for a while now, and we didn't have much customer feedback. Please open a new one if it's still an issue.

anafalcao avatar Jan 24 '25 11:01 anafalcao

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

github-actions[bot] avatar Jan 24 '25 11:01 github-actions[bot]