Migrate processing elements from three class to once class approach

[bossenti]

Body

Historically, we used a three class based approach to define our pipeline elements (processors and sinks). A while ago we introduced a simplified one class approach to define pipeline elements. Therefore, we need to migrate all existing pipeline elements.

Processors

Old approach A processor is defined by the following three classes:

• StandaloneEventProcessingDeclarer
• EventProcessorBindingParams
• EventProcessor

Examples can be found below.

New approach Only use the following class to define a processing element:

• StreamPipesDataProcessor

See an example: WelfordChangeDetection Processors to migrate

• [x] #1330
• [x] Image Cropper #1330
• [x] #1598
• [x] #1599
• [x] GenericImageClassification #1330
• [x] QrCodeReader #1330
• [x] #1601
• [x] #1603
• [x] #1604
• [x] #1605
• [x] #1606
• [x] #1607
• [x] #1608
• [x] #1609
• [x] #1366
• [x] #1610
• [x] #1387
• [x] #1612
• [x] BooleanTimekeeping #1612
• [x] BooleanTimer #1612
• [x] CSVMetadataEnrichment #1612
• [x] StateBuffer #1612
• [x] StateBufferLabeler #1612
• [x] NumberLabeler #1612
• [x] #1426
• [x] #1434
• [x] #1436
• [x] #1435
• [x] TimestampExtractor #1612
• [x] TransformToBoolean #1612
• [x] ChangedValueDetection #1612
• [x] CalculateDuration #1612
• [x] #1624

Sinks

Old approach A sink is defined by the following three classes:

• StandaloneEventSinkDeclarer
• EventSinkBindingParams
• EventSink

Examples can be found below.

New approach Only use the following class to define a sink:

• StreamPipesDataSink

See an example: RocketMQPublisherSink

Sinks to migrate

• [x] #1614
• [x] JmsPublisher #1614
• [x] #1456
• [x] NatsPublisher #1614
• [x] RabbitMqConsumer (needs to be renamed?!) #1614
• [x] RestPublisher #1614
• [x] #1618
• [x] Ditto #1618
• [x] IotDb #1618
• [x] OpcUa #1618
• [x] PostgreSQL #1618
• [x] #1457
• [x] #1620
• [x] OneSignalProducer #1620
• [x] SlackNotification #1620
• [x] TelegramPublisher #1620

Final tasks

• [x] #1632
• [x] remove deprecated cusum processor https://github.com/apache/streampipes/pull/1586

How to contribute?

Pick one processor or sink, leave a comment below that you want to work on it This ensures that we keep an overview and tasks are not done multiple times When you have done the refactoring of one component, file us a PR This way we approach to tackle this large refactoring effort step by step

Mentoring

As this issue is marked as good first issue: one of @dominikriemer, @tenthe, or @bossenti are happy to provide help for getting started, just tag (one of) them if you want to start working on this issue and need some help. This issue is ideal for people that are interested in getting a first glimpse on how to implement custom pipeline elements.

StreamPipes Committer

I acknowledge that I am a maintainer/committer of the Apache StreamPipes project.

[BudhirajaChinmay]

@bossenti I would like to work on this issue. I am planning to start with the processor Image Enricher.

[bossenti]

@BudhirajaChinmay great to hear 🎉 I have created a sub-task for you: https://github.com/apache/streampipes/issues/1330 If you comment there anything (e.g., "assign me"), I should be able to assign you to the issue.

Please let us know, if you need any help :)

[BudhirajaChinmay]

@bossenti according to my understanding, the processors in the stream-apis-processor-image-jvm (Image Enricher, Image Cropper, GenericImageClassification and QrCodeReader) would be interlinked for depreciating the old flow as they are using a particular set of common helper classes. I can work on these together. If this approach seems ok to you, you can assign these four processors to me.

[bossenti]

Sounds legit 👍🏼

[flaxxen]

Hello! Me and some friends have a school project to work with a refactor issue this week and would like to work on the boolean counter migration.

[tenthe]

Great, thank you! 🚀 Please provide a PR and we can discuss the changes there.

[bossenti]

@flaxxen Great to hear 🤩 I have created an issue for you: https://github.com/apache/streampipes/issues/1366 If you give it a comment, I can assign you to it 🙂 In case you need any help you can ask us there, we are always happy to help!

[flaxxen]

Hi again! Needed another task to fill out the time needed for the week, my group were a bit ahead of me writing so we have already mostly finished the boolean inverter class aswell, could you create another issue for that and we can provide a separate PR?

[bossenti]

@flaxxen awesome 🚀 Here you go: https://github.com/apache/streampipes/issues/1387

[liuxiaocs7]

@bossenti I want to complete this task to be familiar with our pipeline elements, please assign StringCounter to me first.

[bossenti]

@liuxiaocs7 Here you go: https://github.com/apache/streampipes/issues/1426 Please comment there, so I can assign you

[liuxiaocs7]

Hi, @bossenti, StringCounter has completed, i want to work on StringToState and StringTimer this week, could you help me open issues separately, I will take assign in these two issues separately, thanks! :)

[bossenti]

Awesome, here you go :) https://github.com/apache/streampipes/issues/1434 https://github.com/apache/streampipes/issues/1436

[liuxiaocs7]

Hi, @bossenti, StringToState and StringTimer PRs has created, could you assign KafkaPublisher and Redis sink to me, I want to learn more about the sink component, thanks! :)

[tenthe]

Hi @liuxiaocs7, I created #1457 & #1456. Please comment there so I can assign you the issue.

[bossenti]

All pipeline elements are migrated