🎉 New Destination: SelectDB

[catpineapple]

What

Add destination SelectDB

How

destination-selectdb is a destination implemented based on SelectDB copy into, It works as follows: 1: read stream and write a temp csv file. 2: make a http put request(redirect) to upload this csv file to Object Storage as a selectdb internal stage. 3: make a http request to commit this internal stage with 'copy into' sql .

Recommended reading order

1. SelectdbDestination.java
2. SelectdbConsumer.java
3. SelectdbCopyInto.java

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

• [x] Community member? Grant edit access to maintainers (instructions)
• [x] Secrets in the connector's spec are annotated with airbyte_secret
• [x] Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• [ ] Code reviews completed
• [ ] Documentation updated
  • [x] Connector's README.md
  • [x] Connector's bootstrap.md. See description and examples
  • [x] docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • [x] docs/integrations/README.md
  • [ ] airbyte-integrations/builds.md
• [ ] PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• [ ] Create a non-forked branch based on this PR and test the below items on it
• [ ] Build is successful
• [ ] If new credentials are required for use in CI, add them to GSM. Instructions.
• [ ] /test connector=connectors/<name> command is passing
• [ ] New Connector version released on Dockerhub by running the /publish command described here
• [ ] After the connector is published, connector added to connector index as described here
• [ ] Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

• [ ] Grant edit access to maintainers (instructions)
• [ ] Secrets in the connector's spec are annotated with airbyte_secret
• [ ] Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• [ ] Code reviews completed
• [ ] Documentation updated
  • [ ] Connector's README.md
  • [ ] Connector's bootstrap.md. See description and examples
  • [ ] Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
• [ ] PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• [ ] Create a non-forked branch based on this PR and test the below items on it
• [ ] Build is successful
• [ ] If new credentials are required for use in CI, add them to GSM. Instructions.
• [ ] /test connector=connectors/<name> command is passing
• [ ] New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

• [ ] Issue acceptance criteria met
• [ ] PR name follows PR naming conventions
• [ ] If adding a new generator, add it to the list of scaffold modules being tested
• [ ] The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
• [ ] Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

[itaseskii]

/test connector=connectors/destination-selectdb

:clock2: connectors/destination-selectdb https://github.com/airbytehq/airbyte/actions/runs/3962689266 :x: connectors/destination-selectdb https://github.com/airbytehq/airbyte/actions/runs/3962689266 :bug: https://gradle.com/s/u3ywaqapp3x6u

Build Failed

Test summary info:

Could not find result summary

[itaseskii]

@catpineapple can you provide us with the necessary configuration to run the acceptance tests or instructions on how to create an account for selectdb? The service seems to be in a preview mode and is severely lacking any documentation

[catpineapple]

@catpineapple can you provide us with the necessary configuration to run the acceptance tests or instructions on how to create an account for selectdb? The service seems to be in a preview mode and is severely lacking any documentation

@itaseskii The english documentation for SelectDB will be available soon. We are looking forward to working with the airbyte community to test this pull request. You can contact us through email [email protected], so that we can provide a SelectDB account for testing.

[itaseskii]

@catpineapple can you provide us with the necessary configuration to run the acceptance tests or instructions on how to create an account for selectdb? The service seems to be in a preview mode and is severely lacking any documentation

@itaseskii The english documentation for SelectDB will be available soon. We are looking forward to working with the airbyte community to test this pull request. You can contact us through email [email protected], so that we can provide a SelectDB account for testing.

@catpineapple sure thing! the email has been sent.

[itaseskii]

@catpineapple please check my review comments

[itaseskii]

@catpineapple your tests seem broken. can you rerun them? you have updated the config in spec.json but it seems that you have forgotten in SelectdbDestinationAcceptanceTest.java. There are several other issues as well.

[itaseskii]

@catpineapple The connector is pretty hard to test. This is what I'mg getting on testSync testSyncNotFailsWithNewFields testSyncNotFailsWithNewFields

{"type":"LOG","log":{"level":"INFO","message":"redirect to s3 address:https://selectdb-online-us-east-1-bucket1668080392.s3.amazonaws.com/AWVAOLRZ/stage/admin/admin/_exchange_rate_99f6d9a7-c361-4576-950f-45c7b5449dba1677627426515.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230228T233619Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=AKIA3AUKURBS4Z65WP4C%2F20230228%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=ad78c587f4d616b759044d46091bff8ea50a8dddd97bf2545fc6ffc08a6f715a"}}
{"type":"LOG","log":{"level":"ERROR","message":"Failed to upload selectdb stage in destination: java.lang.RuntimeException: upload file error: HTTP/1.1 501 Not Implemented"}}
{"type":"LOG","log":{"level":"ERROR","message":"Failed to copy into selectdb in destination"}}
{"type":"LOG","log":{"level":"ERROR","message":"Something went wrong in the connector. See the logs for more details.\nStack Trace: java.io.IOException: Failed to copy into selectdb in destination\n\tat io.airbyte.integrations.destination.selectdb.SelectdbConsumer.close(SelectdbConsumer.java:105)\n\tat io.airbyte.integrations.base.FailureTrackingAirbyteMessageConsumer.close(FailureTrackingAirbyteMessageConsumer.java:82)\n\tat io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:149)\n\tat io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:98)\n\tat io.airbyte.integrations.destination.selectdb.SelectdbDestination.main(SelectdbDestination.java:48)\n"}}

btw I'm using a Public Link for the jdbc and http connections

[itaseskii]

@catpineapple friendly reminder that you need to fix the acceptance tests if you want this connector to get merged. https://github.com/airbytehq/airbyte/pull/20881#issuecomment-1448656583 https://github.com/airbytehq/airbyte/pull/20881#issuecomment-1449076685

[itaseskii]

@catpineapple there are other issues to be resolved as well, merging master to you branch won't resolve them.

[catpineapple]

@itaseskii Sorry my friend , something else has been delayed lately and it seems to have fixed the previous issue now. Please help me with the review

[itaseskii]

@itaseskii Sorry my friend , something else has been delayed lately and it seems to have fixed the previous issue now. Please help me with the review

@catpineapple is this a new test run or the old one? please provide me with your email address so I can send you the config that I'm using because I'm getting test failures and this is blocking the release. I suggest you run the tests using my configuration.

[catpineapple]

@itaseskii [email protected]

[itaseskii]

@itaseskii [email protected]

@catpineapple I have sent you the config. Feel free to also login to my acc to see if everything is created as needed.

[itaseskii]

@catpineapple I can confirm that the connector is passing all the acceptance tests :partying_face:

[itaseskii]

@catpineapple one last thing. your secrets/config.json path is wrong. Can you address this issue as suggested here: https://github.com/airbytehq/airbyte/pull/20881#discussion_r1140111330 https://github.com/airbytehq/airbyte/pull/20881#discussion_r1140112334 https://github.com/airbytehq/airbyte/pull/20881#discussion_r1140113041

[catpineapple]

Hello, what else does this pull request need me to do, uh, I want to know about the merge progress of this. @itaseskii

[itaseskii]

Hello, what else does this pull request need me to do, uh, I want to know about the merge progress of this. @itaseskii

No changes needed from your side. I'm waiting for someone to run the tests and publish.

[marcosmarxm]

/test connector=connectors/destination-selectdb

:clock2: connectors/destination-selectdb https://github.com/airbytehq/airbyte/actions/runs/4600653782 :x: connectors/destination-selectdb https://github.com/airbytehq/airbyte/actions/runs/4600653782 :bug: https://gradle.com/s/ul7so4vjlqmaw

Build Failed

Test summary info:

Could not find result summary

[marcosmarxm]

/test connector=connectors/destination-selectdb

:clock2: connectors/destination-selectdb https://github.com/airbytehq/airbyte/actions/runs/4621915257 :x: connectors/destination-selectdb https://github.com/airbytehq/airbyte/actions/runs/4621915257 :bug:

Build Passed

Test summary info:

All Passed

[marcosmarxm]

/publish connector=connectors/destination-selectdb run-tests=false

:clock2: Publishing the following connectors:
connectors/destination-selectdb
https://github.com/airbytehq/airbyte/actions/runs/4622051441

Connector	Did it publish?	Were definitions generated?
connectors/destination-selectdb	:white_check_mark:	:x:

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️