airbyte
airbyte copied to clipboard
Published
20 hours ago •
airbytehq
airbytehq
Extend SATs to capture UI limitations - fixed version
New set-up of https://github.com/airbytehq/airbyte/pull/21451
See the second commit (https://github.com/airbytehq/airbyte/pull/21903/commits/e37f4098697d26fb0d5ec8545a65d9a4892f4fbc) for the new changes. In short, I removed the unnecessary fixture and used connector_spec instead.
To make sure it works fine, I tried to run it for an actual connector both via local python ( https://github.com/airbytehq/airbyte/blob/c67bfc50bb72bf2db1ceca127df86a158ccb1941/airbyte-integrations/bases/source-acceptance-test/README.md#L52 ) and via docker image ( acceptance-test-docker.sh in the connector code itself, make sure to use the locally built dev image).
We should definitely make this part of an integration test for the source acceptance tests, I will put up a separate issue for that.
Affected Connector Report
NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:
- Run integration tests
- Bump connector or module version
- Add changelog
- Publish the new version
✅ Sources (63)
| Connector | Version | Changelog | Publish |
|---|---|---|---|
source-airtable |
2.0.0 |
✅ | ✅ |
source-amazon-ads |
0.1.28 |
✅ | ✅ |
source-amazon-seller-partner |
0.2.31 |
✅ | ✅ |
source-amazon-sqs |
0.1.0 |
✅ | ✅ |
source-amplitude |
0.1.20 |
✅ | ✅ |
source-appsflyer |
0.1.0 |
✅ | ✅ |
source-asana |
0.1.5 |
✅ | ✅ |
source-azure-table |
0.1.3 |
✅ | ✅ |
source-braintree |
0.1.3 |
✅ | ✅ |
source-cart |
0.2.0 |
✅ | ✅ |
source-chargebee |
0.1.16 |
✅ | ✅ |
source-commercetools |
0.1.0 |
✅ | ✅ |
source-confluence |
0.1.1 |
✅ | ✅ |
source-datadog |
0.1.0 |
✅ | ✅ |
source-delighted |
0.2.0 |
✅ | ✅ |
source-drift |
0.2.5 |
✅ | ✅ |
source-facebook-marketing |
0.2.84 |
✅ | ✅ |
source-freshcaller |
0.1.0 |
✅ | ✅ |
source-freshsales |
0.1.2 |
✅ | ✅ |
source-freshservice |
0.1.1 |
✅ | ✅ |
source-github |
0.4.1 |
✅ | ✅ |
source-gitlab |
1.0.1 |
✅ | ✅ |
source-google-ads |
0.2.9 |
✅ | ✅ |
source-google-search-console |
0.1.18 |
✅ | ✅ |
source-greenhouse |
0.3.0 |
✅ | ✅ |
source-harvest |
0.1.14 |
✅ | ✅ |
source-instagram |
1.0.1 |
✅ | ✅ |
source-iterable |
0.1.22 |
✅ | ✅ |
source-klaviyo |
0.1.11 |
✅ | ✅ |
source-lemlist |
0.1.1 |
✅ | ✅ |
source-lever-hiring |
0.1.3 |
✅ | ✅ |
source-linnworks |
0.1.5 |
✅ | ✅ |
source-mailchimp |
0.3.1 |
✅ | ✅ |
source-mailgun |
0.1.0 |
✅ | ✅ |
source-monday |
0.2.2 |
✅ | ✅ |
source-notion |
1.0.1 |
✅ | ✅ |
source-okta |
0.1.14 |
✅ | ✅ |
source-onesignal |
0.1.2 |
✅ | ✅ |
source-openweather |
0.1.6 |
✅ | ✅ |
source-outreach |
0.1.2 |
✅ | ✅ |
source-pardot |
0.1.1 |
✅ | ✅ |
source-paystack |
0.1.1 |
✅ | ✅ |
source-pinterest |
0.2.2 |
✅ | ✅ |
source-pipedrive |
0.1.13 |
✅ | ✅ |
source-plaid |
0.3.2 |
✅ | ✅ |
source-posthog |
0.1.8 |
✅ | ✅ |
source-prestashop |
0.3.0 |
✅ | ✅ |
source-quickbooks-singer |
0.1.5 |
✅ | ✅ |
source-recharge |
0.2.5 |
✅ | ✅ |
source-retently |
0.1.3 |
✅ | ✅ |
source-salesforce |
1.0.30 |
✅ | ✅ |
source-salesloft |
0.1.3 |
✅ | ✅ |
source-sendgrid |
0.3.0 |
✅ | ✅ |
source-sentry |
0.1.9 |
✅ | ✅ |
source-strava |
0.1.2 |
✅ | ✅ |
source-surveymonkey |
0.1.13 |
✅ | ✅ |
source-tplcentral |
0.1.1 |
✅ | ✅ |
source-twilio |
0.1.14 |
✅ | ✅ |
source-weatherstack |
0.1.0 |
✅ | ✅ |
source-youtube-analytics |
0.1.3 |
✅ | ✅ |
source-zendesk-sunshine |
0.1.1 |
✅ | ✅ |
source-zendesk-talk |
0.1.5 |
✅ | ✅ |
source-zenloop |
0.1.4 |
✅ | ✅ |
- See "Actionable Items" below for how to resolve warnings and errors.
✅ Destinations (0)
| Connector | Version | Changelog | Publish |
|---|
- See "Actionable Items" below for how to resolve warnings and errors.
✅ Other Modules (0)
Actionable Items
(click to expand)
| Category | Status | Actionable Item |
|---|---|---|
| Version | ❌ mismatch |
The version of the connector is different from its normal variant. Please bump the version of the connector. |
| ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
|
| Changelog | ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
| ❌ changelog missing |
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog. | |
| Publish | ⚠ not in seed |
The connector is not in the seed file (e.g. source_definitions.yaml), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug. |
| ❌ diff seed version |
The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the /publish command to publish the latest version. |
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
/test connector=connectors/source-sendgrid
:clock2: connectors/source-sendgrid https://github.com/airbytehq/airbyte/actions/runs/4014006590 :white_check_mark: connectors/source-sendgrid https://github.com/airbytehq/airbyte/actions/runs/4014006590 Python tests coverage:
Name Stmts Miss Cover
-------------------------------------------------
source_sendgrid/__init__.py 2 0 100%
source_sendgrid/source.py 26 3 88%
source_sendgrid/streams.py 171 21 88%
-------------------------------------------------
TOTAL 199 24 88%
Name Stmts Miss Cover Missing
----------------------------------------------------------------------------------
source_acceptance_test/base.py 12 4 67% 16-19
source_acceptance_test/config.py 141 5 96% 87, 93, 239, 243-244
source_acceptance_test/conftest.py 211 95 55% 36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
source_acceptance_test/plugin.py 69 25 64% 22-23, 31, 36, 120-140, 144-148
source_acceptance_test/tests/test_core.py 476 117 75% 53, 58, 97-108, 113-120, 124-125, 129-130, 380, 400, 438, 476-493, 506-517, 521-526, 532, 565-570, 608-615, 658-660, 663, 728-736, 748-751, 756, 812-813, 819, 822, 858-868, 881-906
source_acceptance_test/tests/test_incremental.py 160 14 91% 58-65, 70-83, 246
source_acceptance_test/utils/asserts.py 39 2 95% 62-63
source_acceptance_test/utils/common.py 94 10 89% 16-17, 32-38, 72, 75
source_acceptance_test/utils/compare.py 62 23 63% 21-51, 68, 97-99
source_acceptance_test/utils/connector_runner.py 133 33 75% 24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
source_acceptance_test/utils/json_schema_helper.py 114 13 89% 31-32, 39, 42, 66-69, 97, 121, 203-205
----------------------------------------------------------------------------------
TOTAL 1690 341 80%
Build Passed
Test summary info:
=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:98: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:507: The previous and actual discovered catalogs are identical.
============ 37 passed, 2 skipped, 37 warnings in 85.94s (0:01:25) =============
/test connector=connectors/source-s3
:clock2: connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/4014007518 :white_check_mark: connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/4014007518 Python tests coverage:
Name Stmts Miss Cover Missing
----------------------------------------------------------------------------------
source_acceptance_test/base.py 12 4 67% 16-19
source_acceptance_test/config.py 141 5 96% 87, 93, 239, 243-244
source_acceptance_test/conftest.py 211 95 55% 36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
source_acceptance_test/plugin.py 69 25 64% 22-23, 31, 36, 120-140, 144-148
source_acceptance_test/tests/test_core.py 476 117 75% 53, 58, 97-108, 113-120, 124-125, 129-130, 380, 400, 438, 476-493, 506-517, 521-526, 532, 565-570, 608-615, 658-660, 663, 728-736, 748-751, 756, 812-813, 819, 822, 858-868, 881-906
source_acceptance_test/tests/test_incremental.py 160 14 91% 58-65, 70-83, 246
source_acceptance_test/utils/asserts.py 39 2 95% 62-63
source_acceptance_test/utils/common.py 94 10 89% 16-17, 32-38, 72, 75
source_acceptance_test/utils/compare.py 62 23 63% 21-51, 68, 97-99
source_acceptance_test/utils/connector_runner.py 133 33 75% 24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
source_acceptance_test/utils/json_schema_helper.py 114 13 89% 31-32, 39, 42, 66-69, 97, 121, 203-205
----------------------------------------------------------------------------------
TOTAL 1690 341 80%
Name Stmts Miss Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/formats/parquet_spec.py 9 0 100%
source_s3/source_files_abstract/formats/jsonl_spec.py 13 0 100%
source_s3/source_files_abstract/formats/csv_spec.py 16 0 100%
source_s3/source_files_abstract/formats/avro_spec.py 5 0 100%
source_s3/s3_utils.py 20 0 100%
source_s3/__init__.py 2 0 100%
source_s3/source_files_abstract/storagefile.py 23 1 96%
source_s3/stream.py 43 3 93%
source_s3/s3file.py 41 3 93%
source_s3/source_files_abstract/stream.py 248 20 92%
source_s3/source_files_abstract/formats/abstract_file_parser.py 41 4 90%
source_s3/source.py 27 4 85%
source_s3/source_files_abstract/formats/csv_parser.py 102 30 71%
source_s3/source_files_abstract/file_info.py 26 8 69%
source_s3/utils.py 31 10 68%
source_s3/exceptions.py 10 4 60%
source_s3/source_files_abstract/source.py 39 16 59%
source_s3/source_files_abstract/spec.py 55 31 44%
source_s3/source_files_abstract/formats/jsonl_parser.py 53 32 40%
source_s3/source_files_abstract/formats/avro_parser.py 39 25 36%
source_s3/source_files_abstract/formats/parquet_parser.py 64 44 31%
-------------------------------------------------------------------------------------
TOTAL 907 235 74%
Name Stmts Miss Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/storagefile.py 23 0 100%
source_s3/source_files_abstract/spec.py 55 0 100%
source_s3/source_files_abstract/formats/parquet_spec.py 9 0 100%
source_s3/source_files_abstract/formats/jsonl_spec.py 13 0 100%
source_s3/source_files_abstract/formats/csv_spec.py 16 0 100%
source_s3/source_files_abstract/formats/avro_spec.py 5 0 100%
source_s3/source.py 27 0 100%
source_s3/exceptions.py 10 0 100%
source_s3/__init__.py 2 0 100%
source_s3/source_files_abstract/formats/parquet_parser.py 64 2 97%
source_s3/source_files_abstract/formats/abstract_file_parser.py 41 2 95%
source_s3/stream.py 43 3 93%
source_s3/s3file.py 41 3 93%
source_s3/source_files_abstract/formats/avro_parser.py 39 3 92%
source_s3/source_files_abstract/formats/jsonl_parser.py 53 5 91%
source_s3/source_files_abstract/file_info.py 26 3 88%
source_s3/source_files_abstract/source.py 39 5 87%
source_s3/source_files_abstract/stream.py 248 45 82%
source_s3/source_files_abstract/formats/csv_parser.py 102 20 80%
source_s3/s3_utils.py 20 4 80%
source_s3/utils.py 31 10 68%
-------------------------------------------------------------------------------------
TOTAL 907 105 88%
Build Passed
Test summary info:
=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:98: The previous and actual specifications are identical.
SKIPPED [6] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:507: The previous and actual discovered catalogs are identical.
================== 108 passed, 7 skipped in 235.55s (0:03:55) ==================
/test connector=connectors/source-airtable
:clock2: connectors/source-airtable https://github.com/airbytehq/airbyte/actions/runs/4019123865 :x: connectors/source-airtable https://github.com/airbytehq/airbyte/actions/runs/4019123865 :bug: https://gradle.com/s/kbi4vq4iotv72
Build Failed
Test summary info:
=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_backward_compatibility[inputs0] - source_...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:507: The previous and actual discovered catalogs are identical.
=================== 1 failed, 32 passed, 2 skipped in 31.28s ===================
/test connector=connectors/source-airtable
:clock2: connectors/source-airtable https://github.com/airbytehq/airbyte/actions/runs/4022868580 :white_check_mark: connectors/source-airtable https://github.com/airbytehq/airbyte/actions/runs/4022868580 Python tests coverage:
Name Stmts Miss Cover
-------------------------------------------------------
source_airtable/schema_helpers.py 19 0 100%
source_airtable/__init__.py 2 0 100%
source_airtable/streams.py 92 1 99%
source_airtable/source.py 38 1 97%
source_airtable/auth.py 17 6 65%
-------------------------------------------------------
TOTAL 168 8 95%
Name Stmts Miss Cover Missing
----------------------------------------------------------------------------------
source_acceptance_test/base.py 12 4 67% 16-19
source_acceptance_test/config.py 141 5 96% 87, 93, 239, 243-244
source_acceptance_test/conftest.py 211 95 55% 36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
source_acceptance_test/plugin.py 69 25 64% 22-23, 31, 36, 120-140, 144-148
source_acceptance_test/tests/test_core.py 476 117 75% 53, 58, 97-108, 113-120, 124-125, 129-130, 380, 400, 438, 476-493, 506-517, 521-526, 532, 565-570, 608-615, 658-660, 663, 728-736, 748-751, 756, 812-813, 819, 822, 858-868, 881-906
source_acceptance_test/tests/test_incremental.py 160 14 91% 58-65, 70-83, 246
source_acceptance_test/utils/asserts.py 39 2 95% 62-63
source_acceptance_test/utils/common.py 94 10 89% 16-17, 32-38, 72, 75
source_acceptance_test/utils/compare.py 62 23 63% 21-51, 68, 97-99
source_acceptance_test/utils/connector_runner.py 133 33 75% 24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
source_acceptance_test/utils/json_schema_helper.py 114 13 89% 31-32, 39, 42, 66-69, 97, 121, 203-205
----------------------------------------------------------------------------------
TOTAL 1690 341 80%
Build Passed
Test summary info:
=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: Incremental syncs are not supported on this connector.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:98: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:507: The previous and actual discovered catalogs are identical.
=================== 36 passed, 3 skipped in 71.59s (0:01:11) ===================
/test connector=connectors/source-airtable
:clock2: connectors/source-airtable https://github.com/airbytehq/airbyte/actions/runs/4022885187 :white_check_mark: connectors/source-airtable https://github.com/airbytehq/airbyte/actions/runs/4022885187 Python tests coverage:
Name Stmts Miss Cover
-------------------------------------------------------
source_airtable/schema_helpers.py 19 0 100%
source_airtable/__init__.py 2 0 100%
source_airtable/streams.py 92 1 99%
source_airtable/source.py 38 1 97%
source_airtable/auth.py 17 6 65%
-------------------------------------------------------
TOTAL 168 8 95%
Name Stmts Miss Cover Missing
----------------------------------------------------------------------------------
source_acceptance_test/base.py 12 4 67% 16-19
source_acceptance_test/config.py 141 5 96% 87, 93, 239, 243-244
source_acceptance_test/conftest.py 211 95 55% 36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
source_acceptance_test/plugin.py 69 25 64% 22-23, 31, 36, 120-140, 144-148
source_acceptance_test/tests/test_core.py 476 117 75% 53, 58, 97-108, 113-120, 124-125, 129-130, 380, 400, 438, 476-493, 506-517, 521-526, 532, 565-570, 608-615, 658-660, 663, 728-736, 748-751, 756, 812-813, 819, 822, 858-868, 881-906
source_acceptance_test/tests/test_incremental.py 160 14 91% 58-65, 70-83, 246
source_acceptance_test/utils/asserts.py 39 2 95% 62-63
source_acceptance_test/utils/common.py 94 10 89% 16-17, 32-38, 72, 75
source_acceptance_test/utils/compare.py 62 23 63% 21-51, 68, 97-99
source_acceptance_test/utils/connector_runner.py 133 33 75% 24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
source_acceptance_test/utils/json_schema_helper.py 114 13 89% 31-32, 39, 42, 66-69, 97, 121, 203-205
----------------------------------------------------------------------------------
TOTAL 1690 341 80%
Build Passed
Test summary info:
=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: Incremental syncs are not supported on this connector.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:98: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:507: The previous and actual discovered catalogs are identical.
=================== 36 passed, 3 skipped in 71.75s (0:01:11) ===================
/test connector=connectors/source-stripe
:clock2: connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/4041797353 :white_check_mark: connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/4041797353 Python tests coverage:
Name Stmts Miss Cover
-----------------------------------------------
source_stripe/source.py 22 0 100%
source_stripe/__init__.py 2 0 100%
source_stripe/streams.py 308 28 91%
-----------------------------------------------
TOTAL 332 28 92%
Name Stmts Miss Cover Missing
----------------------------------------------------------------------------------
source_acceptance_test/base.py 12 4 67% 16-19
source_acceptance_test/config.py 141 5 96% 87, 93, 239, 243-244
source_acceptance_test/conftest.py 211 95 55% 36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
source_acceptance_test/plugin.py 69 25 64% 22-23, 31, 36, 120-140, 144-148
source_acceptance_test/tests/test_core.py 476 117 75% 53, 58, 97-108, 113-120, 124-125, 129-130, 380, 400, 438, 476-493, 506-517, 521-526, 532, 565-570, 608-615, 658-660, 663, 728-736, 748-751, 756, 812-813, 819, 822, 858-868, 881-906
source_acceptance_test/tests/test_incremental.py 160 14 91% 58-65, 70-83, 246
source_acceptance_test/utils/asserts.py 39 2 95% 62-63
source_acceptance_test/utils/common.py 94 10 89% 16-17, 32-38, 72, 75
source_acceptance_test/utils/compare.py 62 23 63% 21-51, 68, 97-99
source_acceptance_test/utils/connector_runner.py 133 33 75% 24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
source_acceptance_test/utils/json_schema_helper.py 114 13 89% 31-32, 39, 42, 66-69, 97, 121, 203-205
----------------------------------------------------------------------------------
TOTAL 1690 341 80%
Build Passed
Test summary info:
=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:98: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:507: The previous and actual discovered catalogs are identical.
============ 53 passed, 3 skipped, 37 warnings in 887.94s (0:14:47) ============
/publish connector=bases/source-acceptance-test auto-bump-version=false
:clock2: Publishing the following connectors:
bases/source-acceptance-test
https://github.com/airbytehq/airbyte/actions/runs/4042003727
| Connector | Did it publish? | Were definitions generated? |
|---|---|---|
| bases/source-acceptance-test | :white_check_mark: | :x: |
if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️