[Suggestion] customizable index and type based on record contents

[jeff-k-lee]

I had been using dynamo2es-lambda and saw this module is more lightweight and relies on less dependencies than the other module. However this module routes all records to a fixed index and type in elasticsearch.

The other module dynamod2es-lambda has this concept of indexField and typeField in configuration that allows each incoming record to be routed to different indexes or types depending on the record's contents. I am wondering if this feature will be difficult to add.

Example: DynamoDB source table: "fruit_samples"

Configuration: indexField: ["genus", "species"] typeField: "genus" seperator: "-"

Elasticsearch destinations: index: "citrus-citrus_limon", type: "citrus" index: "citrus-citrus_sinensis", type: "citrus" index: "prunus-prunus_domestica", type: "prunus"

[matrus2]

I will investigate this topic.

[matrus2]

This can be done by transform function. But currently it is applied only to modify/insert events, to make it work entirely it is required to be added to remove event as well. It requires breaking change to be introduced.

[franz101]

How is the investigation going? I modified the package long time ago for custom indices.

Would look into it again if we find a way not to make it a breaking change....