opensearch-migrations
opensearch-migrations copied to clipboard
[FEATURE] Consume S3 persisted objects as an input stream
Is your feature request related to a problem?
It would be nice if the replayer can have a built-in submodule that can read and stream S3 objects (in similar protobuf format) like the messages in Kafka topic and feed it into replay
What solution would you like?
A sub module, like S3InputStream similar to a Kafka stream reader, to feed protobuf messages into main replayer executor.
What alternatives have you considered?
An external Python or other shell scripts to feed S3 into stdin of the replayer pipe
Do you have any additional context?
Add any other context or screenshots about the feature request here.