airbyte icon indicating copy to clipboard operation
airbyte copied to clipboard

Published 20 hours ago • verified publisher icon airbytehq

7198: SSH Tunnel: allow using OPENSSH key format

Open kimerinn opened this issue 2 years ago • 5 comments

What

Allows to use nonencrypted OPENSSH keys for SSH tunneling

How

Use sshlib library to deal with OPENSSH keys github.com/connectbot/sshlib

Recommended reading order

  1. airbyte-integrations/bases/base-java/src/main/java/io/airbyte/integrations/base/ssh/SshTunnel.java

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

  • [ ] Community member? Grant edit access to maintainers (instructions)
  • [ ] Secrets in the connector's spec are annotated with airbyte_secret
  • [ ] Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • [ ] Code reviews completed
  • [ ] Documentation updated
    • [ ] Connector's README.md
    • [ ] Connector's bootstrap.md. See description and examples
    • [ ] docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
    • [ ] docs/integrations/README.md
    • [ ] airbyte-integrations/builds.md
  • [ ] PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • [ ] Create a non-forked branch based on this PR and test the below items on it
  • [ ] Build is successful
  • [ ] If new credentials are required for use in CI, add them to GSM. Instructions.
  • [ ] /test connector=connectors/<name> command is passing
  • [ ] New Connector version released on Dockerhub by running the /publish command described here
  • [ ] After the connector is published, connector added to connector index as described here
  • [ ] Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here
Updating a connector

Community member or Airbyter

  • [ ] Grant edit access to maintainers (instructions)
  • [ ] Secrets in the connector's spec are annotated with airbyte_secret
  • [ ] Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • [ ] Code reviews completed
  • [ ] Documentation updated
    • [ ] Connector's README.md
    • [ ] Connector's bootstrap.md. See description and examples
    • [ ] Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • [ ] PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • [ ] Create a non-forked branch based on this PR and test the below items on it
  • [ ] Build is successful
  • [ ] If new credentials are required for use in CI, add them to GSM. Instructions.
  • [ ] /test connector=connectors/<name> command is passing
  • [ ] New Connector version released on Dockerhub and connector version bumped by running the /publish command described here
Connector Generator
  • [ ] Issue acceptance criteria met
  • [ ] PR name follows PR naming conventions
  • [ ] If adding a new generator, add it to the list of scaffold modules being tested
  • [ ] The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
  • [ ] Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

kimerinn avatar Jul 20 '22 11:07 kimerinn

/test connector=connectors/destination-postgres

:clock2: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2711528015 :white_check_mark: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2711528015 Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         162     34    79%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 570    377    34%
-------------------------------------------------------------------------------------
TOTAL                                                              1363    587    57%

Build Passed

Test summary info:

All Passed

kimerinn avatar Jul 21 '22 11:07 kimerinn

/test connector=connectors/destination-postgres

:clock2: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2729103365 :x: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2729103365 :bug: https://gradle.com/s/cl4dly3lksilw

Build Failed

Test summary info:

Could not find result summary

kimerinn avatar Jul 25 '22 00:07 kimerinn

/test connector=connectors/destination-postgres

:clock2: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2729117084 :x: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2729117084 :bug: https://gradle.com/s/24vgvxozty4jk

Build Failed

Test summary info:

Could not find result summary

kimerinn avatar Jul 25 '22 00:07 kimerinn

/test connector=connectors/destination-postgres

:clock2: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2732057538 :x: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2732057538 :bug: https://gradle.com/s/iw7od3p4kc4c2

Build Failed

Test summary info:

Could not find result summary

kimerinn avatar Jul 25 '22 11:07 kimerinn

/test connector=connectors/destination-postgres

:clock2: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2735919521 :x: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2735919521 :bug: https://gradle.com/s/forkhblnwhte6

Build Failed

Test summary info:

Could not find result summary

kimerinn avatar Jul 25 '22 23:07 kimerinn

/test connector=bases/base-normalization

:clock2: bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/2774487624 :x: bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/2774487624 :bug:

kimerinn avatar Aug 01 '22 11:08 kimerinn

/test connector=bases/base-normalization

:clock2: bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/2785729273 :x: bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/2785729273 :bug: https://gradle.com/s/zqfrlomo2lqhg

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 SKIPPED [1] integration_tests/test_ephemeral.py:61: Destinations DestinationType.MYSQL is not in NORMALIZATION_TEST_TARGET env variable (MYSQL is also skipped)
	 SKIPPED [1] integration_tests/test_normalization.py:81: Destinations DestinationType.CLICKHOUSE does not support nested streams
	 SKIPPED [1] integration_tests/test_normalization.py:81: Destinations DestinationType.ORACLE does not support nested streams
	 SKIPPED [1] integration_tests/test_normalization.py:143: DestinationType.SNOWFLAKE is disabled as it doesnt support schema change in incremental yet (column type changes)
	 FAILED integration_tests/test_ephemeral.py::test_destination_supported_limits[DestinationType.CLICKHOUSE-1000]
	 FAILED integration_tests/test_ephemeral.py::test_destination_supported_limits[DestinationType.MSSQL-1000]
	 FAILED integration_tests/test_ephemeral.py::test_destination_supported_limits[DestinationType.ORACLE-1000]
	 FAILED integration_tests/test_ephemeral.py::test_destination_supported_limits[DestinationType.POSTGRES-1000]
	 FAILED integration_tests/test_ephemeral.py::test_destination_failure_over_limits[Postgres-1665-target lists can have at most 1664 entries]
	 FAILED integration_tests/test_ephemeral.py::test_destination_failure_over_limits[MySQL-250-Row size too large]
	 FAILED integration_tests/test_ephemeral.py::test_destination_failure_over_limits[Oracle-1001-ORA-01792: maximum number of columns in a table or view is 1000]
	 FAILED integration_tests/test_ephemeral.py::test_destination_failure_over_limits[MSSQL-1025-exceeds the maximum of 1024 columns.]
	 FAILED integration_tests/test_ephemeral.py::test_empty_streams - AssertionErr...
	 FAILED integration_tests/test_ephemeral.py::test_stream_with_1_airbyte_column
	 FAILED integration_tests/test_normalization.py::test_normalization[DestinationType.CLICKHOUSE-test_simple_streams]
	 FAILED integration_tests/test_normalization.py::test_normalization[DestinationType.MSSQL-test_simple_streams]
	 FAILED integration_tests/test_normalization.py::test_normalization[DestinationType.MSSQL-test_nested_streams]
	 FAILED integration_tests/test_normalization.py::test_normalization[DestinationType.MYSQL-test_simple_streams]
	 FAILED integration_tests/test_normalization.py::test_normalization[DestinationType.MYSQL-test_nested_streams]
	 FAILED integration_tests/test_normalization.py::test_normalization[DestinationType.ORACLE-test_simple_streams]
	 FAILED integration_tests/test_normalization.py::test_normalization[DestinationType.POSTGRES-test_simple_streams]
	 FAILED integration_tests/test_normalization.py::test_normalization[DestinationType.POSTGRES-test_nested_streams]
	 [31m============ [31m[1m18 failed[0m, [32m12 passed[0m, [33m4 skipped[0m[31m in 3319.48s (0:55:19)[0m[31m =============[0m

kimerinn avatar Aug 02 '22 22:08 kimerinn

/test connector=connectors/destination-bigquery

kimerinn avatar Aug 04 '22 10:08 kimerinn

/test connector=connectors/destination-bigquery

:clock2: connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2817132092 :white_check_mark: connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2817132092 Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     157      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         186     49    74%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 589    394    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1421    624    56%

Build Passed

Test summary info:

All Passed

kimerinn avatar Aug 08 '22 10:08 kimerinn

/test connector=connectors/destination-snowflake

:clock2: connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2817378282 :white_check_mark: connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2817378282 Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     157      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         186     49    74%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 589    394    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1421    624    56%

Build Passed

Test summary info:

All Passed

kimerinn avatar Aug 08 '22 11:08 kimerinn

/test connector=connectors/destination-redshift

kimerinn avatar Aug 08 '22 12:08 kimerinn

/test connector=connectors/destination-redshift

:clock2: connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2818734023 :white_check_mark: connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2818734023 Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     157      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         186     49    74%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 589    394    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1421    624    56%

Build Passed

Test summary info:

All Passed

kimerinn avatar Aug 08 '22 14:08 kimerinn

/test connector=connectors/destination-postgres

:clock2: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2819240804 :x: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2819240804 :bug: https://gradle.com/s/b3ydcnzoqcgui

Build Failed

Test summary info:

Could not find result summary

kimerinn avatar Aug 08 '22 15:08 kimerinn

/test connector=connectors/destination-postgres

:clock2: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2820196084 :x: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2820196084 :bug: https://gradle.com/s/cr4a254zexxhs

Build Failed

Test summary info:

Could not find result summary

kimerinn avatar Aug 08 '22 18:08 kimerinn

/test connector=connectors/destination-postgres

:clock2: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2821441098 :x: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2821441098 :bug: https://gradle.com/s/koam2bfsumyre

Build Failed

Test summary info:

Could not find result summary

kimerinn avatar Aug 08 '22 22:08 kimerinn

/test connector=connectors/destination-postgres

:clock2: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2825009219 :x: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2825009219 :bug: https://gradle.com/s/jaq3dnab25p5o

Build Failed

Test summary info:

Could not find result summary

kimerinn avatar Aug 09 '22 11:08 kimerinn

/test connector=connectors/destination-postgres

:clock2: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2828441479 :x: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2828441479 :bug: https://gradle.com/s/kiofn6zclugwu

Build Failed

Test summary info:

Could not find result summary

kimerinn avatar Aug 09 '22 21:08 kimerinn

/test connector=connectors/destination-postgres

kimerinn avatar Aug 09 '22 22:08 kimerinn

/test connector=connectors/destination-postgres

:clock2: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2828965273 :x: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2828965273 :bug: https://gradle.com/s/dldkk4yk6dcxq

Build Failed

Test summary info:

Could not find result summary

kimerinn avatar Aug 09 '22 23:08 kimerinn

/test connector=connectors/destination-postgres

:clock2: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2832014996 :x: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2832014996 :bug: https://gradle.com/s/ffh26baitlr5k

Build Failed

Test summary info:

Could not find result summary

kimerinn avatar Aug 10 '22 10:08 kimerinn

/test connector=connectors/destination-postgres

:clock2: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2832303353 :x: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2832303353 :bug: https://gradle.com/s/n4ihbnje32eqs

Build Failed

Test summary info:

Could not find result summary

kimerinn avatar Aug 10 '22 11:08 kimerinn

/test connector=connectors/destination-postgres

:clock2: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2835948892 :x: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2835948892 :bug: https://gradle.com/s/fhudmvaecikm2

Build Failed

Test summary info:

Could not find result summary

kimerinn avatar Aug 10 '22 21:08 kimerinn

/test connector=connectors/destination-postgres

:clock2: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2842975967 :x: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2842975967 :bug: https://gradle.com/s/hxshfpzzmf2ng

Build Failed

Test summary info:

Could not find result summary

kimerinn avatar Aug 11 '22 21:08 kimerinn

/test connector=connectors/destination-postgres

:clock2: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2846214346 :x: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2846214346 :bug: https://gradle.com/s/q5jzosgex4q64

Build Failed

Test summary info:

Could not find result summary

kimerinn avatar Aug 12 '22 10:08 kimerinn

/test connector=connectors/destination-postgres

:clock2: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2861433576 :x: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2861433576 :bug: https://gradle.com/s/64itvcxkomq2e

Build Failed

Test summary info:

Could not find result summary

kimerinn avatar Aug 15 '22 14:08 kimerinn

/test connector=connectors/destination-postgres

:clock2: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2864002990 :x: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2864002990 :bug: https://gradle.com/s/dy5ctqilzvrlw

Build Failed

Test summary info:

Could not find result summary

kimerinn avatar Aug 15 '22 22:08 kimerinn

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

:white_check_mark: kimerinn
:x: mtuhai
You have signed the CLA already but the status is still pending? Let us recheck it.

CLAassistant avatar Aug 19 '22 00:08 CLAassistant

/test connector=connectors/destination-postgres

:clock2: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2886224334 :x: connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2886224334 :bug: https://gradle.com/s/cxifdm2a3m7xi

Build Failed

Test summary info:

Could not find result summary

kimerinn avatar Aug 19 '22 00:08 kimerinn

@rodireich please review these changes next week

grishick avatar Aug 27 '22 00:08 grishick