airbyte
airbyte copied to clipboard
🎉 Source Amazon Ads : use optional config report_record_types
What
- Currently, record types are hard coded in respective reports and doesn't provide user a way to selectively fetch specific record type(s)
How
*Adding an optional configuration option to read a list of selective report types from user. When the optional config is empty, it will fetch for all record types ( default behaviour )

Recommended reading order
-
source_amazon_ads/spec.yaml
-
source_amazon_ads/streams/report_streams/report_streams.py
🚨 User Impact 🚨
Not a breaking change
Pre-merge Checklist
Expand the relevant checklist and delete the others.
Updating a connector
Community member or Airbyter
- [x] Grant edit access to maintainers (instructions)
- [ ] Secrets in the connector's spec are annotated with
airbyte_secret
- [x] Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
. - [ ] Code reviews completed
- [ ] Documentation updated
- [ ] Connector's
README.md
- [ ] Connector's
bootstrap.md
. See description and examples - [ ] Changelog updated in
docs/integrations/<source or destination>/<name>.md
including changelog. See changelog example
- [ ] Connector's
- [ ] PR name follows PR naming conventions
Airbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
- [ ] Create a non-forked branch based on this PR and test the below items on it
- [ ] Build is successful
- [ ] If new credentials are required for use in CI, add them to GSM. Instructions.
- [ ]
/test connector=connectors/<name>
command is passing - [ ] New Connector version released on Dockerhub and connector version bumped by running the
/publish
command described here
Tests
Unit
Results (3.80s): 70 passed Name Stmts Miss Cover Missing
source_amazon_ads/init.py 2 0 100% source_amazon_ads/constants.py 1 0 100% source_amazon_ads/schemas/init.py 7 0 100% source_amazon_ads/schemas/attribution_report.py 21 0 100% source_amazon_ads/schemas/common.py 51 1 98% 26 source_amazon_ads/schemas/profile.py 16 0 100% source_amazon_ads/schemas/sponsored_brands.py 22 0 100% source_amazon_ads/schemas/sponsored_display.py 31 0 100% source_amazon_ads/schemas/sponsored_products.py 37 0 100% source_amazon_ads/source.py 44 1 98% 129 source_amazon_ads/streams/init.py 7 0 100% source_amazon_ads/streams/attribution_report.py 81 0 100% source_amazon_ads/streams/common.py 79 1 99% 157 source_amazon_ads/streams/profiles.py 21 0 100% source_amazon_ads/streams/report_streams/init.py 5 0 100% source_amazon_ads/streams/report_streams/brands_report.py 10 0 100% source_amazon_ads/streams/report_streams/brands_video_report.py 10 0 100% source_amazon_ads/streams/report_streams/display_report.py 16 0 100% source_amazon_ads/streams/report_streams/products_report.py 18 0 100% source_amazon_ads/streams/report_streams/report_streams.py 243 19 92% 151, 204-205, 257-258, 380-381, 413-415, 420-421, 430-436 source_amazon_ads/streams/sponsored_brands.py 26 0 100% source_amazon_ads/streams/sponsored_display.py 31 0 100% source_amazon_ads/streams/sponsored_products.py 41 0 100% source_amazon_ads/utils.py 9 0 100% unit_tests/init.py 0 0 100% unit_tests/conftest.py 37 0 100% unit_tests/test_attribution_report.py 66 1 98% 47 unit_tests/test_report_streams.py 315 0 100% unit_tests/test_source.py 47 0 100% unit_tests/test_streams.py 126 1 99% 77 unit_tests/utils.py 32 0 100%
TOTAL 1452 24 98%
Could you please refactor the spec input type to enum? See https://github.com/airbytehq/airbyte/pull/18677#discussion_r1021689703
@monai
- updated spec to be of enum type
- added tests for video report with custom record type
asins
Looks good. I'm going to proceed with publishing the connector.
~~Please do not push anything to this branch anymore. Doing so in the middle of publishing might break the process.~~
/test connector=connectors/source-amazon-ads
:clock2: connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/3513324551 :white_check_mark: connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/3513324551 Python tests coverage:
Name Stmts Miss Cover
-------------------------------------------------------------------------------------
source_amazon_ads/utils.py 22 0 100%
source_amazon_ads/streams/sponsored_products.py 41 0 100%
source_amazon_ads/streams/sponsored_display.py 31 0 100%
source_amazon_ads/streams/sponsored_brands.py 26 0 100%
source_amazon_ads/streams/report_streams/products_report.py 18 0 100%
source_amazon_ads/streams/report_streams/display_report.py 16 0 100%
source_amazon_ads/streams/report_streams/brands_video_report.py 10 0 100%
source_amazon_ads/streams/report_streams/brands_report.py 10 0 100%
source_amazon_ads/streams/report_streams/__init__.py 5 0 100%
source_amazon_ads/streams/profiles.py 21 0 100%
source_amazon_ads/streams/attribution_report.py 81 0 100%
source_amazon_ads/streams/__init__.py 7 0 100%
source_amazon_ads/schemas/sponsored_products.py 37 0 100%
source_amazon_ads/schemas/sponsored_display.py 31 0 100%
source_amazon_ads/schemas/sponsored_brands.py 22 0 100%
source_amazon_ads/schemas/profile.py 16 0 100%
source_amazon_ads/schemas/attribution_report.py 21 0 100%
source_amazon_ads/schemas/__init__.py 7 0 100%
source_amazon_ads/constants.py 1 0 100%
source_amazon_ads/__init__.py 2 0 100%
source_amazon_ads/streams/common.py 79 1 99%
source_amazon_ads/schemas/common.py 51 1 98%
source_amazon_ads/source.py 44 1 98%
source_amazon_ads/streams/report_streams/report_streams.py 243 19 92%
-------------------------------------------------------------------------------------
TOTAL 842 22 97%
Name Stmts Miss Cover Missing
----------------------------------------------------------------------------------
source_acceptance_test/base.py 12 4 67% 16-19
source_acceptance_test/config.py 139 5 96% 87, 93, 235, 239-240
source_acceptance_test/conftest.py 196 92 53% 35, 41-43, 48, 54, 60, 66, 72-74, 93, 98-100, 106-108, 114-115, 120-121, 126, 132, 141-150, 156-161, 176, 200, 231, 237, 243-248, 256-261, 269-282, 287-293, 300-311, 318-334
source_acceptance_test/plugin.py 69 25 64% 22-23, 31, 36, 120-140, 144-148
source_acceptance_test/tests/test_core.py 398 111 72% 53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
source_acceptance_test/tests/test_incremental.py 158 14 91% 52-59, 64-77, 240
source_acceptance_test/utils/asserts.py 37 2 95% 57-58
source_acceptance_test/utils/common.py 94 10 89% 16-17, 32-38, 72, 75
source_acceptance_test/utils/compare.py 62 23 63% 21-51, 68, 97-99
source_acceptance_test/utils/connector_runner.py 112 50 55% 23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
source_acceptance_test/utils/json_schema_helper.py 107 13 88% 30-31, 38, 41, 65-68, 96, 120, 192-194
----------------------------------------------------------------------------------
TOTAL 1563 349 78%
Build Passed
Test summary info:
All Passed
@monai
I have added the requested changes. PTAL
Hello :wave:, first thank you for this amazing contribution.
We really appreciate the effort you've made to improve the project. We ask you patience for the code review. Last month our team was focused on Hacktoberfest event and that probably left some PR without the proper feedback. And this week, due to the Thanksgiving US Holiday, most our team is out of office with their families. Another important piece of information why code won't be merge this week is: as a safety measure the core team has decided to freeze merging code to main branch to keep the release stable. Next week we'll return to you with the proper code review and update the status of your contribution.
If you have any questions feel free to send me a message in Slack! Thanks!
/test connector=connectors/source-amazon-ads
@sajarin @monai @grubberr
PTAL
Kindly inform if we are still considering this PR. Thanks
Hello :wave::skin-tone-2: and thank you for your contribution!
Airbyte has instituted a code freeze between 19 and 30 December, to make sure there are no disruptions during the holidays. Because of this, reviewing and merging your contribution may take longer than usual. We apologize for the delay, but we want everyone to have a quiet and happy holiday.
If you have any questions or need further clarification, please don't hesitate to ping via Slack.
bump 1
Sorry the delay @ganpatagarwal I'll run tests to validate your contribution.
/test connector=connectors/source-amazon-ads
:clock2: connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/3937969464 :x: connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/3937969464 :bug: https://gradle.com/s/ncsuh3lio63wa
Build Failed
Test summary info:
=========================== short test summary info ============================
ERROR test_core.py::TestBasicRead::test_read[inputs0] - Failed: A bypass_reas...
ERROR test_core.py::TestBasicRead::test_read[inputs1] - Failed: High strictne...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:377: The previous and actual discovered catalogs are identical.
============= 31 passed, 1 skipped, 2 errors in 506.42s (0:08:26) ==============
@marcosmarxm please let me know if there are any action points on me for this PR.
Thanks !!
Anyone, please run the tests for this PR
/test connector=connectors/source-amazon-ads
:clock2: connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/4062379788 :x: connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/4062379788 :bug: https://gradle.com/s/2jo4uuim4or6u
Build Failed
Test summary info:
=========================== short test summary info ============================
ERROR test_core.py::TestSpec::test_match_expected[inputs0] - json.decoder.JSO...
ERROR test_core.py::TestSpec::test_property_type_is_not_array[inputs0] - json...
ERROR test_core.py::TestSpec::test_object_not_empty[inputs0] - json.decoder.J...
ERROR test_core.py::TestSpec::test_array_type[inputs0] - json.decoder.JSONDec...
ERROR test_core.py::TestSpec::test_forbidden_complex_types[inputs0] - json.de...
ERROR test_core.py::TestSpec::test_date_pattern[inputs0] - json.decoder.JSOND...
ERROR test_core.py::TestSpec::test_date_format[inputs0] - json.decoder.JSONDe...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:507: The previous and actual discovered catalogs are identical.
============= 30 passed, 1 skipped, 7 errors in 2032.62s (0:33:52) =============
/test connector=connectors/source-amazon-ads
:clock2: connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/4066906258 :white_check_mark: connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/4066906258 Python tests coverage:
Name Stmts Miss Cover
-------------------------------------------------------------------------------------
source_amazon_ads/utils.py 22 0 100%
source_amazon_ads/streams/sponsored_products.py 41 0 100%
source_amazon_ads/streams/sponsored_display.py 31 0 100%
source_amazon_ads/streams/sponsored_brands.py 26 0 100%
source_amazon_ads/streams/report_streams/products_report.py 18 0 100%
source_amazon_ads/streams/report_streams/display_report.py 16 0 100%
source_amazon_ads/streams/report_streams/brands_video_report.py 10 0 100%
source_amazon_ads/streams/report_streams/brands_report.py 10 0 100%
source_amazon_ads/streams/report_streams/__init__.py 5 0 100%
source_amazon_ads/streams/profiles.py 21 0 100%
source_amazon_ads/streams/attribution_report.py 81 0 100%
source_amazon_ads/streams/__init__.py 7 0 100%
source_amazon_ads/schemas/sponsored_products.py 37 0 100%
source_amazon_ads/schemas/sponsored_display.py 31 0 100%
source_amazon_ads/schemas/sponsored_brands.py 22 0 100%
source_amazon_ads/schemas/profile.py 16 0 100%
source_amazon_ads/schemas/attribution_report.py 21 0 100%
source_amazon_ads/schemas/__init__.py 7 0 100%
source_amazon_ads/constants.py 1 0 100%
source_amazon_ads/__init__.py 2 0 100%
source_amazon_ads/schemas/common.py 51 1 98%
source_amazon_ads/source.py 47 1 98%
source_amazon_ads/streams/common.py 83 2 98%
source_amazon_ads/streams/report_streams/report_streams.py 243 19 92%
-------------------------------------------------------------------------------------
TOTAL 849 23 97%
Name Stmts Miss Cover Missing
----------------------------------------------------------------------------------
source_acceptance_test/base.py 12 4 67% 16-19
source_acceptance_test/config.py 141 5 96% 87, 93, 239, 243-244
source_acceptance_test/conftest.py 211 95 55% 36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
source_acceptance_test/plugin.py 69 25 64% 22-23, 31, 36, 120-140, 144-148
source_acceptance_test/tests/test_core.py 476 117 75% 53, 58, 97-108, 113-120, 124-125, 129-130, 380, 400, 438, 476-493, 506-517, 521-526, 532, 565-570, 608-615, 658-660, 663, 728-736, 748-751, 756, 812-813, 819, 822, 858-868, 881-906
source_acceptance_test/tests/test_incremental.py 160 14 91% 58-65, 70-83, 246
source_acceptance_test/utils/asserts.py 39 2 95% 62-63
source_acceptance_test/utils/common.py 94 10 89% 16-17, 32-38, 72, 75
source_acceptance_test/utils/compare.py 62 23 63% 21-51, 68, 97-99
source_acceptance_test/utils/connector_runner.py 133 33 75% 24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
source_acceptance_test/utils/json_schema_helper.py 114 13 89% 31-32, 39, 42, 66-69, 97, 121, 203-205
----------------------------------------------------------------------------------
TOTAL 1690 341 80%
Build Passed
Test summary info:
=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:507: The previous and actual discovered catalogs are identical.
================== 37 passed, 1 skipped in 2197.74s (0:36:37) ==================
@grubberr can you do the final review of this community contribution? The tests are passing now.
@grubberr pinging you - would you have time for a review? :)
/test connector=connectors/source-amazon-ads
:clock2: connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/4312854605 :x: connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/4312854605 :bug: https://gradle.com/s/o35vdud5tvkek
Build Failed
Test summary info:
=========================== short test summary info ============================
FAILED unit_tests/test_report_streams.py::test_products_report_stream_with_custom_record_types[custom_record_types1-expected_record_types1-True]
FAILED unit_tests/test_report_streams.py::test_products_report_stream_with_custom_record_types[custom_record_types2-expected_record_types2-True]
[31m================= [31m[1m2 failed[0m, [32m82 passed[0m, [33m1368 warnings[0m[31m in 6.57s[0m[31m ==================[0m
@grubberr I have updated the tests as per new METRIC_RESPONSE
value.
Please review again
/test connector=connectors/source-amazon-ads
:clock2: connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/4358004236 :white_check_mark: connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/4358004236 Python tests coverage:
Name Stmts Miss Cover
-------------------------------------------------------------------------------------
source_amazon_ads/utils.py 22 0 100%
source_amazon_ads/streams/sponsored_products.py 41 0 100%
source_amazon_ads/streams/sponsored_display.py 31 0 100%
source_amazon_ads/streams/sponsored_brands.py 26 0 100%
source_amazon_ads/streams/report_streams/products_report.py 20 0 100%
source_amazon_ads/streams/report_streams/display_report.py 18 0 100%
source_amazon_ads/streams/report_streams/brands_video_report.py 12 0 100%
source_amazon_ads/streams/report_streams/brands_report.py 12 0 100%
source_amazon_ads/streams/report_streams/__init__.py 5 0 100%
source_amazon_ads/streams/profiles.py 21 0 100%
source_amazon_ads/streams/attribution_report.py 81 0 100%
source_amazon_ads/streams/__init__.py 7 0 100%
source_amazon_ads/schemas/sponsored_products.py 37 0 100%
source_amazon_ads/schemas/sponsored_display.py 31 0 100%
source_amazon_ads/schemas/sponsored_brands.py 22 0 100%
source_amazon_ads/schemas/profile.py 16 0 100%
source_amazon_ads/schemas/attribution_report.py 21 0 100%
source_amazon_ads/schemas/__init__.py 7 0 100%
source_amazon_ads/constants.py 1 0 100%
source_amazon_ads/__init__.py 2 0 100%
source_amazon_ads/schemas/common.py 50 1 98%
source_amazon_ads/source.py 47 1 98%
source_amazon_ads/streams/common.py 83 2 98%
source_amazon_ads/streams/report_streams/report_streams.py 246 19 92%
-------------------------------------------------------------------------------------
TOTAL 859 23 97%
Build Passed
Test summary info:
=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:509: The previous and actual discovered catalogs are identical.
================== 37 passed, 1 skipped in 484.93s (0:08:04) ===================
/test connector=connectors/source-amazon-ads
:clock2: connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/4384003202 :white_check_mark: connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/4384003202 Python tests coverage:
Name Stmts Miss Cover
-------------------------------------------------------------------------------------
source_amazon_ads/utils.py 22 0 100%
source_amazon_ads/streams/sponsored_products.py 41 0 100%
source_amazon_ads/streams/sponsored_display.py 31 0 100%
source_amazon_ads/streams/sponsored_brands.py 26 0 100%
source_amazon_ads/streams/report_streams/products_report.py 20 0 100%
source_amazon_ads/streams/report_streams/display_report.py 18 0 100%
source_amazon_ads/streams/report_streams/brands_video_report.py 12 0 100%
source_amazon_ads/streams/report_streams/brands_report.py 12 0 100%
source_amazon_ads/streams/report_streams/__init__.py 5 0 100%
source_amazon_ads/streams/profiles.py 21 0 100%
source_amazon_ads/streams/attribution_report.py 81 0 100%
source_amazon_ads/streams/__init__.py 7 0 100%
source_amazon_ads/schemas/sponsored_products.py 37 0 100%
source_amazon_ads/schemas/sponsored_display.py 31 0 100%
source_amazon_ads/schemas/sponsored_brands.py 22 0 100%
source_amazon_ads/schemas/profile.py 16 0 100%
source_amazon_ads/schemas/attribution_report.py 21 0 100%
source_amazon_ads/schemas/__init__.py 7 0 100%
source_amazon_ads/constants.py 1 0 100%
source_amazon_ads/__init__.py 2 0 100%
source_amazon_ads/schemas/common.py 50 1 98%
source_amazon_ads/source.py 47 1 98%
source_amazon_ads/streams/common.py 83 2 98%
source_amazon_ads/streams/report_streams/report_streams.py 246 19 92%
-------------------------------------------------------------------------------------
TOTAL 859 23 97%
Build Passed
Test summary info:
=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:509: The previous and actual discovered catalogs are identical.
================== 37 passed, 1 skipped in 473.59s (0:07:53) ===================
/test connector=connectors/source-amazon-ads
:clock2: connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/4384742923 :white_check_mark: connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/4384742923 Python tests coverage:
Name Stmts Miss Cover
-------------------------------------------------------------------------------------
source_amazon_ads/utils.py 22 0 100%
source_amazon_ads/streams/sponsored_products.py 41 0 100%
source_amazon_ads/streams/sponsored_display.py 31 0 100%
source_amazon_ads/streams/sponsored_brands.py 26 0 100%
source_amazon_ads/streams/report_streams/products_report.py 20 0 100%
source_amazon_ads/streams/report_streams/display_report.py 18 0 100%
source_amazon_ads/streams/report_streams/brands_video_report.py 12 0 100%
source_amazon_ads/streams/report_streams/brands_report.py 12 0 100%
source_amazon_ads/streams/report_streams/__init__.py 5 0 100%
source_amazon_ads/streams/profiles.py 21 0 100%
source_amazon_ads/streams/attribution_report.py 81 0 100%
source_amazon_ads/streams/__init__.py 7 0 100%
source_amazon_ads/schemas/sponsored_products.py 37 0 100%
source_amazon_ads/schemas/sponsored_display.py 31 0 100%
source_amazon_ads/schemas/sponsored_brands.py 22 0 100%
source_amazon_ads/schemas/profile.py 16 0 100%
source_amazon_ads/schemas/attribution_report.py 21 0 100%
source_amazon_ads/schemas/__init__.py 7 0 100%
source_amazon_ads/constants.py 1 0 100%
source_amazon_ads/__init__.py 2 0 100%
source_amazon_ads/schemas/common.py 50 1 98%
source_amazon_ads/source.py 48 1 98%
source_amazon_ads/streams/common.py 83 2 98%
source_amazon_ads/streams/report_streams/report_streams.py 246 19 92%
-------------------------------------------------------------------------------------
TOTAL 860 23 97%
Build Passed
Test summary info:
=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:509: The previous and actual discovered catalogs are identical.
================== 37 passed, 1 skipped in 486.33s (0:08:06) ===================
@marcosmarxm @natalyjazzviolin
Please help in merging this PR.
Thanks !!
/publish connector=connectors/source-amazon-ads
:clock2: Publishing the following connectors:
connectors/source-amazon-ads
https://github.com/airbytehq/airbyte/actions/runs/4418720211
Connector | Did it publish? | Were definitions generated? |
---|---|---|
connectors/source-amazon-ads | :white_check_mark: | :white_check_mark: |
if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️