kafka-connect-storage-cloud icon indicating copy to clipboard operation
kafka-connect-storage-cloud copied to clipboard

S3 Object Tagging

Open btnnn opened this issue 7 years ago • 7 comments

Supporting S3 Object Tagging would be helpful for organization, bookkeeping and monitoring.

Adding a configuration option for S3 Object Tags would be preferred. Another option would be to allow additional HTTP header configuration (enabling users to set tags via the x-amz-tagging header).

Thanks, Brandt

S3 Object tagging documentation

btnnn avatar Jul 14 '17 15:07 btnnn

Supporting initially a small set of the configuration options that are available through the S3 SDK was intentional.

This seems useful. I'm marking this issue as "enhancement". Tags are defined per object, at a maximum number of 10 tags. Do you have a specific use case that you'd like to share regarding tagging?

kkonstantine avatar Aug 05 '17 00:08 kkonstantine

@kkonstantine thanks for your reply!

I'm running Kafka Connect in distributed mode against multiple Kafka topics. I'd like to be able to:

  • Tag objects with the name of the topic that they were created from.
  • Tag all objects that were created with Kafka Connect.

This would enable cost analysis and CloudWatch monitoring/alerting by object tag.

btnnn avatar Aug 05 '17 12:08 btnnn

FYI, this was merged: https://github.com/confluentinc/kafka-connect-storage-cloud/issues/68 (tags for start and end offsets as well as record count)

kkonstantine avatar Mar 09 '20 23:03 kkonstantine

Any updates ? We would like to have functionality to add specific tags to be able to setup lifecycle police in S3 based on tags.

mikita-sakalouski avatar Dec 03 '21 13:12 mikita-sakalouski

Like hardcoded key:value pairs from configuration

mikita-sakalouski avatar Dec 03 '21 13:12 mikita-sakalouski

We need tags aswell, preferably customisable from headers.

gotpist1 avatar Apr 25 '23 14:04 gotpist1

This issue should be solved by my PR above. Is there anything else to be done to close it?

dttung2905 avatar Dec 29 '23 18:12 dttung2905