Feature request: ability to specify JMESPath custom functions for Idempotency

[dreamorosi]

Use case

When working with the JMESPath expressions, the Idempotency utility uses some custom functions (i.e. powertools_json()) that extend the JMESPath built-in functions. This allows customers to work with some complex types that are common when working with AWS payloads.

The underlying JMESPath utility used by Idempotency however allows for further customization by allowing customers to set additional custom functions in addition or instead of the Powertools-provided ones.

It would be great if customers were able to pass their own subclass of Functions or PowertoolsFunctions when using the Idempotency utility.

Solution/User Experience

Currently the Idempotency utility creates its own instance of the PowertoolsFunctions class when instantiating the IdempotencyConfig class. This allows the various components of the utility to reuse it across the implementation.

The setting could be exposed to customers by adding a new option to the IdempotencyConfig class/object:

import { makeIdempotent, IdempotencyConfig } from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';

const persistenceStore = new DynamoDBPersistenceLayer({
  tableName: 'idempotencyTableName',
});

class MyFancyFunctions extends PowertoolsFunctions {
  @Functions.signature({
    argumentsSpecs: [['string']],
  })
  public funcMyFancyFunction(value: string): JSONValue {
    return JSON.parse(value);
  }
}

export const handler = makeIdempotent(async () => true, {
  persistenceStore,
  config: new IdempotencyConfig({
    eventKeyJmespath: 'my_fancy_function(body).["user", "productId"]',
    jmesPathOptions: new MyFancyFunctions(), // passed as part of the idempotency configs
  }),
});

Alternative solutions

No response

Acknowledgment

• [X] This feature request meets Powertools for AWS Lambda (TypeScript) Tenets
• [ ] Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and .NET

Future readers

Please react with 👍 and your use case to help us understand customer demand.

[leandrodamascena]

Hello @dreamorosi! I think this might cover some user cases where the customer might need to decode a compressed string and use as the idempotent key, or want to add some information to the jmespath_key, we never know all the use cases. We support the customer in adding additional functions in Python.

from dataclasses import dataclass

from aws_lambda_powertools.utilities.idempotency import (
    DynamoDBPersistenceLayer,
    IdempotencyConfig,
    idempotent_function,
)
from aws_lambda_powertools.utilities.typing import LambdaContext

from jmespath.functions import signature

from aws_lambda_powertools.utilities.jmespath_utils import (
    PowertoolsFunctions
)

class CustomFunctions(PowertoolsFunctions):
    # only decode if value is a string
    # see supported data types: https://jmespath.org/specification.html#built-in-functions
    @signature({"types": ["string"]})
    def _func_custom_transform(self, payload: str):
        print("Custom function invoked")
        return "customfunction"


custom_jmespath_options = {"custom_functions": CustomFunctions()}

dynamodb = DynamoDBPersistenceLayer(table_name="ddbidempotency")
config = IdempotencyConfig(event_key_jmespath="custom_transform(order_id)", jmespath_options=custom_jmespath_options)  # see Choosing a payload subset section

@dataclass
class OrderItem:
    sku: str
    description: str

@dataclass
class Order:
    item: OrderItem
    order_id: str

@idempotent_function(data_keyword_argument="order", config=config, persistence_store=dynamodb)
def process_order(order: Order):
    return f"processed order {order.order_id}"

def lambda_handler(event: dict, context: LambdaContext):
    config.register_lambda_context(context)  # see Lambda timeouts section
    order_item = OrderItem(sku="fake", description="sample")
    order = Order(item=order_item, order_id="testid")

    # `order` parameter must be called as a keyword argument to work
    process_order(order=order)

Thanks

[dreamorosi]

Thank you Leo, we'll add this to the backlog.

I'm also adding the help-wanted label to signal that this issue is open for contribution. If anyone wants to pick this up please leave a comment and feel free to ask any question.

[arnabrahman]

@dreamorosi please assign me on this.

[pratyaksh123]

Hi, do you have any timeline on when this can be released? I was waiting for such a feature to use custom function for idempotencyKey

[arnabrahman]

@pratyaksh123 There is a planned release today, so it should be part of that release. But not 100% sure, though.

[pratyaksh123]

thanks, for the PR @arnabrahman

[dreamorosi]

Hey hi, yes we're planning on releasing in the next few hours, unless something unexpected comes up during the pre-release checks.

Once it's released there will be a comment under this issue.

[pratyaksh123]

Is there a way I can use two parameters in a function call for idempotency ? Like on my post request I want both url and payload to be considered for idempotency, how can I do that? Because right now I can only see one dataIndexArgument that can be specified

[dreamorosi]

@pratyaksh123 Can you share an example of the payload and one of how the function signature looks like?

[pratyaksh123]

I can't share it fully as its internal company code, but its like this

post(url, payload, config): ...

Here the issue I am facing is that in some of my post calls the payload is exactly the same but url is different, but with powetools idempotency I can only specify one dataIndexArgument , but I need both url and payload to be considered for idempotency key, is there a way to do that ?

[dreamorosi]

import type {
  AxiosInstance,
  AxiosRequestConfig,
  RawAxiosRequestHeaders,
} from 'axios';
import type { Context } from 'aws-lambda';
import {
  makeIdempotent,
  IdempotencyConfig,
} from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import axios from 'axios';
import { Logger } from '@aws-lambda-powertools/logger';

const logger = new Logger({ logLevel: 'debug' });
const persistenceStore = new DynamoDBPersistenceLayer({
  tableName: 'idempotency-table-standard',
});
const IdempConfig = new IdempotencyConfig({});

class MyError extends Error {
  constructor({ name, message }: { name: string; message: string }) {
    super(message);
    this.name = name;
  }
}

class MyFunction {
  private readonly client: AxiosInstance;

  constructor() {
    this.client = axios.create();
  }

  async configureAuthHeader(
    headers: RawAxiosRequestHeaders
  ): Promise<RawAxiosRequestHeaders> {
    // Your implementation goes here
    return headers;
  }

  async post<T, R>(
    url: string,
    payload: T,
    config?: AxiosRequestConfig
  ): Promise<R> {
    const headers = await this.configureAuthHeader(
      config?.headers as RawAxiosRequestHeaders
    );
    const idempotentPost = makeIdempotent(
      async ({ url, payload }) => {
        try {
          const response = await this.client.post(url, payload, {
            ...config,
            headers,
          });
          logger.debug('Post operation completed successfully', {
            data: response.data,
          });
          return await response.data;
        } catch (error) {
          logger.error('Failed to complete post operation', { error });
          throw new MyError({
            name: 'API_ERROR',
            message: 'Failed to complete post operation',
          });
        }
      },
      {
        persistenceStore: persistenceStore,
        config: IdempConfig,
      }
    );

    return await idempotentPost({ url, payload });
  }

  async handler(event: unknown, context: Context) {
    IdempConfig.registerLambdaContext(context); // Important to call this in your handler to avoid timeout issues

    const data = await this.post<Record<string, string>, { id: number }>(
      'https://jsonplaceholder.typicode.com/posts',
      { message: 'Hello, World!' }
    );
    return {
      statusCode: 200,
      body: JSON.stringify(data),
    };
  }
}

const MyFunctionInstance = new MyFunction();
const handler = MyFunctionInstance.handler.bind(MyFunctionInstance);

[github-actions[bot]]

⚠️ COMMENT VISIBILITY WARNING ⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.