dlq destination is shown as a connector in the UI

[maha-hajja]

Bug description

stopping and running conduit causes the DLQ destination to be shown as a second destination in the UI

Steps to reproduce

1. run conduit with a pipeline in the pipeline config files
2. check the pipeline from the UI, you will see 1 source and 1 destination
3. stop conduit (ctrl + c) > then run it again (without deleting the db)
4. check the pipeline from the UI, you will see 1 source and 2 destinations (including the dlq)

Version

v0.7.0-nightly.20230422-dirty darwin/amd64

[lovromazgon]

I found the culprit for this error. It's connector lifecycle events. We persist the connector when a lifecycle event is triggered, so that we don't trigger it again next time the connector runs. We do this for DLQ connectors as well, and next time Conduit starts it retrieves the connector from the store.

The issue is that DLQ connectors are meant to be ephemeral and only exist while the pipeline is running. With lifecycle events we can't get away with this anymore, because we have to know if we have triggered the lifecycle events for the DLQ connector or not. Also we have to trigger the on delete event when the DLQ is deleted (i.e. the plugin is updated or the pipeline is deleted).

There are 2 hotfixes I have in mind:

• Do not persist the connector if it's a DLQ connector. The problem will be that every time a pipeline is started we will trigger the "on created" lifecycle event. This is also the only lifecycle event that will ever be triggered on a DLQ connector (no update or delete lifecycle events).
• Skip the lifecycle events for DLQ connectors. The problem here is that connectors might not work correctly if they rely on some initialization logic triggered by the lifecycle events. Connectors don't use this feature yet so currently the impact would be minimal, but we would then have time to create a proper fix.

I'm inclined to do the second hotfix first and then address the bug properly. It won't be an easy fix though, we will probably have to persist DLQ connectors and manage them in the connectors service.

[simonl2002]

closing because of fixes above