conduit icon indicating copy to clipboard operation
conduit copied to clipboard

BP: Rewrite built-in processors

Open lovromazgon opened this issue 1 year ago • 1 comments

  • Implement new list of processors defined in the design document
  • Make sure each processor is documented and uses the specification generator to extract the specification from Go docs
  • Full list:
    • [ ] field.set - set a field in the record https://github.com/ConduitIO/conduit/pull/1371
    • [ ] field.subset.exclude - remove fields from the record if they are on the list https://github.com/ConduitIO/conduit/pull/1371
    • [ ] field.subset.include - remove fields from the record if they are not on the list
    • [ ] field.rename - rename fields https://github.com/ConduitIO/conduit/pull/1371
    • [ ] field.convert - convert one type into another in a structured payload https://github.com/ConduitIO/conduit/pull/1371
    • [ ] filter - ack record and remove it from the pipeline https://github.com/ConduitIO/conduit/pull/1371
    • [ ] decode.avro - decode Avro data into structured data
    • [ ] decode.json - decode a JSON string into structured data
    • [ ] encode.avro - encode structured data into Avro data
    • [ ] encode.json - encode structured data into a JSON string
    • [ ] unwrap.kafkaconnect - unwrap a Kafka Connect payload into the current record
    • [ ] unwrap.debezium - unwrap a Debezium payload into the current record #1393
    • [ ] unwrap.opencdc - unwrap a OpenCDC payload into the current record #1387
    • [ ] webhook.http - send a HTTP request for each record #1382
    • [ ] custom.javascript - execute JavaScript code for each record https://github.com/ConduitIO/conduit/pull/1374

lovromazgon avatar Nov 28 '23 19:11 lovromazgon

design doc https://github.com/ConduitIO/conduit/blob/main/docs/design-documents/20231117-better-processors.md

maha-hajja avatar Feb 07 '24 00:02 maha-hajja