Implement Idempotency in Sendhooks Using `webhookId` and Redis

[koladev32]

Summary

Implement Idempotency in Sendhooks Using webhookIdfield and Redis

Motivation

The current iteration of Sendhooks lacks an idempotency feature, leading to potential duplicate processing of webhook events. This can cause data inconsistencies and redundant operations. Introducing idempotency will ensure that each webhook event is processed only once, enhancing the system's reliability and efficiency.

Proposed Solution

Utilize the webhookId present in incoming data from the streams as an idempotency key, and leverage Redis for tracking these keys:

1. Idempotency Key Utilization: Use the provided webhookId in each webhook as the unique identifier for idempotency purposes.
2. Redis for Tracking: Store and check webhookId in a Redis database to determine whether a webhook has been processed.
3. Processing Checks: Before processing a webhook, query Redis to check if the webhookId exists. If it does, skip processing; otherwise, proceed and mark the ID as processed in Redis.
4. TTL Policy: Implement a Time-To-Live (TTL) for each webhookId in Redis to manage data storage effectively.
5. Handling Failures: Include mechanisms to handle failures and ensure that retries don't lead to duplicate processing.

Alternatives Considered

• Client-Side Duplication Checks: Relying on clients to avoid duplicates. This is less reliable and increases client-side complexity.
• Timestamp-Based Methods: These are less precise and can lead to synchronization issues.

The proposed Redis-based solution with webhookId is more reliable and ensures consistency at the service's core.

Additional Context

Idempotency is a common feature in systems like payment gateways, where it's crucial to avoid duplicate transactions. Its implementation in Sendhooks would align with these industry standards.

Potential Impact

This feature will improve Sendhooks' reliability but will require additional resources for Redis operations. However, these costs are offset by the increased trust and efficiency gained. This change might also necessitate updates to client interactions with the API.

[koladev32]

@asiedu-kev @LewisYann need your help to validate the issue

[asiedu-kev]

Look good to me..we just have to define efficiently the TTL in order to really solve the potential issue.

[asiedu-kev]

I said potential because in staging for example when i testing my app, i don't have time to update payload send to my api or service and every time ...so you in this case you're forcing me do it if i want to receive a response every time on my test.

[koladev32]

@asiedu-kev you are right. For the TTL, I believe that 12 hours should be enough.

[koladev32]

To add to this, we should be able to log when there a duplicate request incoming. Might be a nice idea to find a way to show it on the https://github.com/Transfa/sendhooks-monitoring project.