🎉 Source Amazon Ads: filters for state on display campaigns

[sage-watterworth]

What

Adding an option for a filter (by state) in the Amazon Ads connection

How

Adding the field to the spec.yaml, and accessing it by adding to the path to Amazon Ads call as a query parameter.

Recommended reading order

1. spec.yaml
2. sponsored_display.py

🚨 User Impact 🚨

The parameter is optional and will not affect current functionality.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

• [x] Grant edit access to maintainers (instructions)
• [x] Secrets in the connector's spec are annotated with airbyte_secret
• [ ] Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• [ ] Code reviews completed
• [x] Documentation updated
  • [ ] Connector's README.md
  • [ ] Connector's bootstrap.md. See description and examples
  • [ ] Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
• [x] PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• [ ] Create a non-forked branch based on this PR and test the below items on it
• [ ] Build is successful
• [ ] If new credentials are required for use in CI, add them to GSM. Instructions.
• [ ] /test connector=connectors/<name> command is passing
• [ ] New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Tests

Unit

Task :airbyte-integrations:bases:source-acceptance-test:_unitTestCoverage test_backward_compatibility.py::test_spec_backward_compatibility[Making a field nullable and changing type should fail (change list order).] ✓15% █▌
test_backward_compatibility.py::test_spec_backward_compatibility[Nullable field: Changing a field type should fail] ✓15% █▋
test_backward_compatibility.py::test_spec_backward_compatibility[Changing a field type in oneOf should fail.] ✓16% █▋
test_backward_compatibility.py::test_spec_backward_compatibility[Narrowing a field type in oneOf should fail.] ✓16% █▋
test_backward_compatibility.py::test_spec_backward_compatibility[Top level: Narrowing a field enum should fail.] ✓16% █▋
test_backward_compatibility.py::test_spec_backward_compatibility[Nested level: Narrowing a field enum should fail.] ✓17% █▋
test_backward_compatibility.py::test_spec_backward_compatibility[Top level: Declaring a field enum should fail.] ✓17% █▊
test_backward_compatibility.py::test_spec_backward_compatibility[Nested level: Declaring a field enum should fail.] ✓17% █▊
test_backward_compatibility.py::test_spec_backward_compatibility[Changing a 'type' field from a string to something else than a list should fail.] ✓18% █▊
test_backward_compatibility.py::test_spec_backward_compatibility[Changing a 'type' field from a list to something else than a string should fail.] ✓18% █▊
test_backward_compatibility.py::test_spec_backward_compatibility[Not changing a spec should not fail] ✓18% █▉
test_backward_compatibility.py::test_spec_backward_compatibility[Top level: Changing the value of additionalProperties should not fail] ✓19% █▉
test_backward_compatibility.py::test_spec_backward_compatibility[Nested level: Changing the value of additionalProperties should not fail] ✓19% █▉
test_backward_compatibility.py::test_spec_backward_compatibility[Adding an optional field should not fail.] ✓19% █▉
test_backward_compatibility.py::test_spec_backward_compatibility[Adding an optional object with required properties should not fail.] ✓20% ██
test_backward_compatibility.py::test_spec_backward_compatibility[No change should not fail.] ✓20% ██
test_backward_compatibility.py::test_spec_backward_compatibility[Changing a field type from a list to a string with same value should not fail.] ✓20% ██
test_backward_compatibility.py::test_spec_backward_compatibility[Changing a field type from a string to a list should not fail.] ✓21% ██▏
test_backward_compatibility.py::test_spec_backward_compatibility[Adding a field type in list should not fail.] ✓21% ██▏
test_backward_compatibility.py::test_spec_backward_compatibility[Making a field nullable should not fail.] ✓21% ██▎
test_backward_compatibility.py::test_spec_backward_compatibility[Making a field nullable should not fail (change list order).] ✓22% ██▎
test_backward_compatibility.py::test_spec_backward_compatibility[Making a field nullable should not fail (from a list).] ✓22% ██▎
test_backward_compatibility.py::test_spec_backward_compatibility[Making a field nullable should not fail (from a list, changing order).] ✓22% ██▎
test_backward_compatibility.py::test_spec_backward_compatibility[Nullable field: Changing order should not fail] ✓23% ██▍
test_backward_compatibility.py::test_spec_backward_compatibility[Nested level: Expanding a field type should not fail.] ✓23% ██▍
test_backward_compatibility.py::test_spec_backward_compatibility[Changing a order in oneOf should not fail.] ✓23% ██▍
test_backward_compatibility.py::test_spec_backward_compatibility[Top level: Expanding a field enum should not fail.] ✓24% ██▍
test_backward_compatibility.py::test_spec_backward_compatibility[Nested level: Expanding a field enum should not fail.] ✓24% ██▌
test_backward_compatibility.py::test_spec_backward_compatibility[Top level: Adding a new optional field with enum should not fail.] ✓24% ██▌
test_backward_compatibility.py::test_spec_backward_compatibility[Top level: Removing the field enum should not fail.] ✓25% ██▌
test_backward_compatibility.py::test_spec_backward_compatibility[Nested level: Removing the enum field should not fail.] ✓25% ██▌
test_backward_compatibility.py::test_validate_previous_configs[Top level: declaring the required field should fail.] ✓25% ██▌
test_backward_compatibility.py::test_validate_previous_configs[Nested level: adding the required field should fail.] ✓26% ██▋
test_backward_compatibility.py::test_validate_previous_configs[Top level: adding a new required property should fail.] ✓26% ██▋
test_backward_compatibility.py::test_validate_previous_configs[Nested level: adding a new required property should fail.] ✓26% ██▋
test_backward_compatibility.py::test_validate_previous_configs[Nullable: Making a field not nullable should fail (not in a list).] ✓27% ██▋
test_backward_compatibility.py::test_validate_previous_configs[Nested level: Narrowing a field type should fail.] ✓27% ██▊
test_backward_compatibility.py::test_validate_previous_configs[Nullable field: Making a field not nullable should fail] ✓27% ██▊
test_backward_compatibility.py::test_validate_previous_configs[Changing a field type should fail.] ✓28% ██▊
test_backward_compatibility.py::test_validate_previous_configs[Changing a field type from a string to a list with a different type value should fail.] ✓28% ██▊
test_backward_compatibility.py::test_validate_previous_configs[Changing a field type should fail from a list to string with different value should fail.] ✓28% ██▊
test_backward_compatibility.py::test_validate_previous_configs[Changing a field type in list should fail.] ✓29% ██▉
test_backward_compatibility.py::test_validate_previous_configs[Making a field nullable and changing type should fail.] ✓29% ██▉
test_backward_compatibility.py::test_validate_previous_configs[Making a field nullable and changing type should fail (change list order).] ✓29% ██▉
test_backward_compatibility.py::test_validate_previous_configs[Nullable field: Changing a field type should fail] ✓30% ██▉
test_backward_compatibility.py::test_validate_previous_configs[Changing a field type in oneOf should fail.] ✓30% ███
test_backward_compatibility.py::test_validate_previous_configs[Narrowing a field type in oneOf should fail.] ✓30% ███
test_backward_compatibility.py::test_validate_previous_configs[Top level: Narrowing a field enum should fail.] ✓30% ███▏
test_backward_compatibility.py::test_validate_previous_configs[Nested level: Narrowing a field enum should fail.] ✓31% ███▏
test_backward_compatibility.py::test_validate_previous_configs[Top level: Declaring a field enum should fail.] ✓31% ███▎
test_backward_compatibility.py::test_validate_previous_configs[Nested level: Declaring a field enum should fail.] ✓31% ███▎
test_backward_compatibility.py::test_validate_previous_configs[Not changing a spec should not fail] ✓32% ███▎
test_backward_compatibility.py::test_validate_previous_configs[Top level: Changing the value of additionalProperties should not fail] ✓32% ███▎
test_backward_compatibility.py::test_validate_previous_configs[Nested level: Changing the value of additionalProperties should not fail] ✓32% ███▎
test_backward_compatibility.py::test_validate_previous_configs[Adding an optional field should not fail.] ✓33% ███▍
test_backward_compatibility.py::test_validate_previous_configs[Adding an optional object with required properties should not fail.] ✓33% ███▍
test_backward_compatibility.py::test_validate_previous_configs[No change should not fail.] ✓33% ███▍
test_backward_compatibility.py::test_validate_previous_configs[Changing a field type from a list to a string with same value should not fail.] ✓34% ███▍
test_backward_compatibility.py::test_validate_previous_configs[Changing a field type from a string to a list should not fail.] ✓34% ███▌
test_backward_compatibility.py::test_validate_previous_configs[Adding a field type in list should not fail.] ✓34% ███▌
test_backward_compatibility.py::test_validate_previous_configs[Making a field nullable should not fail.] ✓35% ███▌
test_backward_compatibility.py::test_validate_previous_configs[Making a field nullable should not fail (change list order).] ✓35% ███▌
test_backward_compatibility.py::test_validate_previous_configs[Making a field nullable should not fail (from a list).] ✓35% ███▋
test_backward_compatibility.py::test_validate_previous_configs[Making a field nullable should not fail (from a list, changing order).] ✓36% ███▋
test_backward_compatibility.py::test_validate_previous_configs[Nullable field: Changing order should not fail] ✓36% ███▋
test_backward_compatibility.py::test_validate_previous_configs[Nested level: Expanding a field type should not fail.] ✓36% ███▋
test_backward_compatibility.py::test_validate_previous_configs[Changing a order in oneOf should not fail.] ✓37% ███▋
test_backward_compatibility.py::test_validate_previous_configs[Top level: Expanding a field enum should not fail.] ✓37% ███▊
test_backward_compatibility.py::test_validate_previous_configs[Nested level: Expanding a field enum should not fail.] ✓37% ███▊
test_backward_compatibility.py::test_validate_previous_configs[Top level: Adding a new optional field with enum should not fail.] ✓38% ███▊
test_backward_compatibility.py::test_validate_previous_configs[Top level: Removing the field enum should not fail.] ✓38% ███▊
test_backward_compatibility.py::test_validate_previous_configs[Nested level: Removing the enum field should not fail.] ✓38% ███▉
test_backward_compatibility.py::test_catalog_backward_compatibility[Removing a stream from a catalog should fail.] ✓39% ███▉
test_backward_compatibility.py::test_catalog_backward_compatibility[Changing a field type should fail.] ✓39% ███▉
test_backward_compatibility.py::test_catalog_backward_compatibility[Renaming a stream should fail.] ✓39% ███▉
test_backward_compatibility.py::test_catalog_backward_compatibility[Changing a cursor in a stream should fail.] ✓40% ████
test_backward_compatibility.py::test_catalog_backward_compatibility[Changing a cursor in a stream should fail (nested cursors).] ✓40% ████
test_backward_compatibility.py::test_catalog_backward_compatibility[Changing a cursor in a stream should fail (nested cursors removal).] ✓40% ████
test_backward_compatibility.py::test_catalog_backward_compatibility[Making a field nullable should not fail.] ✓41% ████▏
test_backward_compatibility.py::test_catalog_backward_compatibility[Changing 'type' field to list should not fail.] ✓41% ████▏
test_backward_compatibility.py::test_catalog_backward_compatibility[Removing a field should not fail.] ✓41% ████▎
test_backward_compatibility.py::test_catalog_backward_compatibility[Not changing a cursor in a stream should not fail.] ✓42% ████▎
test_core.py::test_discovery[schema0-cursors0-True] ✓ 42% ████▎
test_core.py::test_discovery[schema1-cursors1-False] ✓ 42% ████▎
test_core.py::test_discovery[schema2-cursors2-True] ✓ 43% ████▍
test_core.py::test_discovery[schema3-cursors3-True] ✓ 43% ████▍
test_core.py::test_discovery[schema4-cursors4-False] ✓ 43% ████▍
test_core.py::test_discovery[schema5-cursors5-True] ✓ 44% ████▍
test_core.py::test_ref_in_discovery_schemas[schema0-False] ✓ 44% ████▌
test_core.py::test_ref_in_discovery_schemas[schema1-True] ✓ 44% ████▌
test_core.py::test_ref_in_discovery_schemas[schema2-True] ✓ 45% ████▌
test_core.py::test_ref_in_discovery_schemas[schema3-True] ✓ 45% ████▌
test_core.py::test_ref_in_discovery_schemas[schema4-False] ✓ 45% ████▌
test_core.py::test_ref_in_discovery_schemas[schema5-True] ✓ 46% ████▋
test_core.py::test_ref_in_discovery_schemas[schema6-True] ✓ 46% ████▋
test_core.py::test_keyword_in_discovery_schemas[schema0-allOf-False] ✓46% ████▋
test_core.py::test_keyword_in_discovery_schemas[schema1-allOf-True] ✓47% ████▋
test_core.py::test_keyword_in_discovery_schemas[schema2-allOf-False] ✓47% ████▊
test_core.py::test_keyword_in_discovery_schemas[schema3-allOf-True] ✓47% ████▊
test_core.py::test_keyword_in_discovery_schemas[schema4-allOf-True] ✓48% ████▊
test_core.py::test_keyword_in_discovery_schemas[schema5-allOf-True] ✓48% ████▊
test_core.py::test_keyword_in_discovery_schemas[schema6-not-True] ✓48% ████▊
test_core.py::test_keyword_in_discovery_schemas[schema7-not-False] ✓49% ████▉
test_core.py::test_keyword_in_discovery_schemas[schema8-not-True] ✓49% ████▉
test_core.py::test_supported_sync_modes_in_stream[discovered_catalog0-expectation0] ✓49% ████▉
test_core.py::test_supported_sync_modes_in_stream[discovered_catalog1-expectation1] ✓50% ████▉
test_core.py::test_supported_sync_modes_in_stream[discovered_catalog2-expectation2] ✓50% █████
test_core.py::test_supported_sync_modes_in_stream[discovered_catalog3-expectation3] ✓50% █████
test_core.py::test_supported_sync_modes_in_stream[discovered_catalog4-expectation4] ✓50% █████▏
test_core.py::test_additional_properties_is_true[discovered_catalog0-expectation0] ✓51% █████▏
test_core.py::test_additional_properties_is_true[discovered_catalog1-expectation1] ✓51% █████▎
test_core.py::test_additional_properties_is_true[discovered_catalog2-expectation2] ✓51% █████▎
test_core.py::test_additional_properties_is_true[discovered_catalog3-expectation3] ✓52% █████▎
test_core.py::test_additional_properties_is_true[discovered_catalog4-expectation4] ✓52% █████▎
test_core.py::test_additional_properties_is_true[discovered_catalog5-expectation5] ✓52% █████▎
test_core.py::test_read[schema0-record0-False] ✓ 53% █████▍
test_core.py::test_read[schema1-record1-False] ✓ 53% █████▍
test_core.py::test_read[schema2-record2-True] ✓ 53% █████▍
test_core.py::test_read[schema3-record3-False] ✓ 54% █████▍
test_core.py::test_read[schema4-record4-True] ✓ 54% █████▌
test_core.py::test_read[schema5-record5-False] ✓ 54% █████▌
test_core.py::test_airbyte_trace_message_on_failure[output0-True-False] ✓55% █████▌
test_core.py::test_airbyte_trace_message_on_failure[output1-True-False] ✓55% █████▌
test_core.py::test_airbyte_trace_message_on_failure[output2-True-False] ✓55% █████▋
test_core.py::test_airbyte_trace_message_on_failure[output3-True-True] ✓56% █████▋
test_core.py::test_airbyte_trace_message_on_failure[output4-True-True] ✓56% █████▋
test_core.py::test_airbyte_trace_message_on_failure[output5-False-False] ✓56% █████▋
test_core.py::test_airbyte_trace_message_on_failure[output6-False-False] ✓57% █████▋
test_core.py::test_airbyte_trace_message_on_failure[output7-False-False] ✓57% █████▊
test_core.py::test_validate_field_appears_at_least_once[records0-configured_catalog0-] ✓57% █████▊
test_core.py::test_validate_field_appears_at_least_once[records1-configured_catalog1-test1 stream has \\['/f2'\\]] ✓58% █████▊
test_core.py::test_validate_field_appears_at_least_once[records2-configured_catalog2-] ✓58% █████▊
test_core.py::test_validate_field_appears_at_least_once[records3-configured_catalog3-] ✓58% █████▉
test_core.py::test_validate_field_appears_at_least_once[records4-configured_catalog4-test1 stream has \\['/f3/\\[\\]'\\]] ✓59% █████▉
test_core.py::test_validate_field_appears_at_least_once[records5-configured_catalog5-] ✓59% █████▉
test_core.py::test_validate_field_appears_at_least_once[records6-configured_catalog6-test1 stream has \\['/f3/f5/\\[\\]'\\]] ✓59% █████▉
test_core.py::test_validate_field_appears_at_least_once[records7-configured_catalog7-test1 stream has \\['/f3/f5/\\[\\]/f6', '/f3/f5/\\[\\]/f7/\\[\\]'\\]] ✓60% ██████
test_core.py::test_validate_field_appears_at_least_once[records8-configured_catalog8-] ✓60% ██████
test_core.py::test_validate_field_appears_at_least_once[records9-configured_catalog9-(test1 stream has \\['/f3/f5/\\[\\]/f7/\\[\\]'])|(test2 \\['/f8'\\])] ✓60% ██████
test_core.py::test_validate_field_appears_at_least_once[records10-configured_catalog10-] ✓61% ██████▏
test_core.py::test_validate_field_appears_at_least_once[records11-configured_catalog11-] ✓61% ██████▏
test_core.py::test_validate_field_appears_at_least_once[records12-configured_catalog12-test1 stream has \\['/f3\\(0\\)/f4', '/f3\\(1\\)/f5\\(0\\)/f6', '/f3\\(1\\)/f5\\(1\\)/f7'\\]] ✓61% ██████▎ IDLE test_incremental.py::test_compare_cursor_with_threshold[record_value0-state_value0-0-True] ✓62% ██████▎
test_incremental.py::test_compare_cursor_with_threshold[record_value1-state_value1-0-False] ✓62% ██████▎
test_incremental.py::test_compare_cursor_with_threshold[record_value2-state_value2-1-True] ✓62% ██████▎
test_incremental.py::test_compare_cursor_with_threshold[record_value3-state_value3-0-True] ✓63% ██████▍
test_incremental.py::test_compare_cursor_with_threshold[record_value4-state_value4-0-False] ✓63% ██████▍
test_incremental.py::test_compare_cursor_with_threshold[record_value5-state_value5-1-True] ✓63% ██████▍
test_incremental.py::test_compare_cursor_with_threshold[2020-10-10-2020-10-09-0-True] ✓64% ██████▍
test_incremental.py::test_compare_cursor_with_threshold[2020-10-10-2020-10-11-0-False] ✓64% ██████▌
test_incremental.py::test_compare_cursor_with_threshold[2020-10-10-2020-10-11-1-True] ✓64% ██████▌
test_incremental.py::test_compare_cursor_with_threshold[1602288000000-1602201600000-0-True] ✓65% ██████▌
test_incremental.py::test_compare_cursor_with_threshold[1602288000000-1602374400000-0-False] ✓65% ██████▌
test_incremental.py::test_compare_cursor_with_threshold[1602288000000-1602374400000-1-True] ✓65% ██████▌
test_incremental.py::test_compare_cursor_with_threshold[1602288000-1602201600-0-True] ✓66% ██████▋
test_incremental.py::test_compare_cursor_with_threshold[1602288000-1602374400-0-False] ✓66% ██████▋
test_incremental.py::test_compare_cursor_with_threshold[1602288000-1602374400-1-True] ✓66% ██████▋
test_incremental.py::test_compare_cursor_with_threshold[aaa-bbb-0-False] ✓67% ██████▋
test_incremental.py::test_compare_cursor_with_threshold[bbb-aaa-0-True] ✓67% ██████▊
test_incremental.py::test_incremental_two_sequential_reads[records10-records20-2020-01-02-0-None-date] ✓67% ██████▊
test_incremental.py::test_incremental_two_sequential_reads[records10-records20-2020-01-02-0-None-string] ✓68% ██████▊
test_incremental.py::test_incremental_two_sequential_reads[records11-records21-2020-01-02-0-First incremental sync should produce records younger-date] ✓68% ██████▊
test_incremental.py::test_incremental_two_sequential_reads[records11-records21-2020-01-02-0-First incremental sync should produce records younger-string] ✓68% ██████▊
test_incremental.py::test_incremental_two_sequential_reads[records12-records22-2020-01-02-0-None-date] ✓69% ██████▉
test_incremental.py::test_incremental_two_sequential_reads[records12-records22-2020-01-02-0-None-string] ✓69% ██████▉
test_incremental.py::test_incremental_two_sequential_reads[records13-records23-2020-01-02-0-Second incremental sync should produce records older-date] ✓69% ██████▉
test_incremental.py::test_incremental_two_sequential_reads[records13-records23-2020-01-02-0-Second incremental sync should produce records older-string] ✓70% ██████▉
test_incremental.py::test_incremental_two_sequential_reads[records14-records24-2020-01-03-2-None-date] ✓70% ███████
test_incremental.py::test_incremental_two_sequential_reads[records14-records24-2020-01-03-2-None-string] ✓70% ███████
test_incremental.py::test_incremental_two_sequential_reads[records15-records25-2020-01-02-2-First incremental sync should produce records younger-date] ✓70% ███████▏
test_incremental.py::test_incremental_two_sequential_reads[records15-records25-2020-01-02-2-First incremental sync should produce records younger-string] ✓71% ███████▏
test_incremental.py::test_incremental_two_sequential_reads[records16-records26-2020-01-06-3-Second incremental sync should produce records older-date] ✓71% ███████▎
test_incremental.py::test_incremental_two_sequential_reads[records16-records26-2020-01-06-3-Second incremental sync should produce records older-string] ✓71% ███████▎
test_incremental.py::test_read_with_multiple_states[test_incremental_with_2_states-records0-state_records0-0-None] ✓72% ███████▎
test_incremental.py::test_read_with_multiple_states[test_first_incremental_only_younger_records-records1-state_records1-0-AssertionError] ✓72% ███████▎
test_incremental.py::test_read_with_multiple_states[test_incremental_with_threshold-records2-state_records2-3-None] ✓72% ███████▎
test_incremental.py::test_read_with_multiple_states[test_incremental_with_incorrect_messages-records3-state_records3-0-AssertionError] ✓73% ███████▍
test_incremental.py::test_read_with_multiple_states[test_incremental_with_multiple_streams-records4-state_records4-0-None] ✓73% ███████▍
test_incremental.py::test_read_with_multiple_states[test_incremental_with_none_state-records5-state_records5-0-None] ✓73% ███████▍
test_incremental.py::test_config_skip_test ✓ 74% ███████▍
test_json_schema_helper.py::test_simple_path ✓ 74% ███████▌
test_json_schema_helper.py::test_nested_path ✓ 74% ███████▌
test_json_schema_helper.py::test_absolute_path ✓ 75% ███████▌
test_json_schema_helper.py::test_none_state ✓ 75% ███████▌
test_json_schema_helper.py::test_json_schema_helper_pydantic_generated ✓75% ███████▋
test_json_schema_helper.py::test_get_object_strucutre[object0-pathes0] ✓76% ███████▋
test_json_schema_helper.py::test_get_object_strucutre[object1-pathes1] ✓76% ███████▋
test_json_schema_helper.py::test_get_object_strucutre[object2-pathes2] ✓76% ███████▋
test_json_schema_helper.py::test_get_object_strucutre[object3-pathes3] ✓77% ███████▋
test_json_schema_helper.py::test_get_object_strucutre[object4-pathes4] ✓77% ███████▊
test_json_schema_helper.py::test_get_object_strucutre[object5-pathes5] ✓77% ███████▊
test_json_schema_helper.py::test_get_object_strucutre[object6-pathes6] ✓78% ███████▊
test_json_schema_helper.py::test_get_expected_schema_structure[schema0-pathes0] ✓78% ███████▊
test_json_schema_helper.py::test_get_expected_schema_structure[schema1-pathes1] ✓78% ███████▉
test_json_schema_helper.py::test_get_expected_schema_structure[schema2-pathes2] ✓79% ███████▉
test_json_schema_helper.py::test_get_expected_schema_structure[schema3-pathes3] ✓79% ███████▉
test_json_schema_helper.py::test_get_expected_schema_structure[schema4-pathes4] ✓79% ███████▉
test_json_schema_helper.py::test_get_expected_schema_structure[schema5-pathes5] ✓80% ████████
test_json_schema_helper.py::test_get_expected_schema_structure[schema6-pathes6] ✓80% ████████
test_json_schema_helper.py::test_get_expected_schema_structure[schema7-pathes7] ✓80% ████████
test_json_schema_helper.py::test_get_expected_schema_structure[schema8-pathes8] ✓81% ████████▏ test_json_schema_helper.py::test_get_expected_schema_structure[schema9-pathes9] ✓81% ████████▏ test_spec.py::test_ref_in_spec_schemas[connector_spec0-True] ✓ 81% ████████▎ test_spec.py::test_ref_in_spec_schemas[connector_spec1-True] ✓ 82% ████████▎ test_spec.py::test_ref_in_spec_schemas[connector_spec2-True] ✓ 82% ████████▎ test_spec.py::test_ref_in_spec_schemas[connector_spec3-True] ✓ 82% ████████▎ test_spec.py::test_ref_in_spec_schemas[connector_spec4-False] ✓ 83% ████████▍ test_spec.py::test_ref_in_spec_schemas[connector_spec5-False] ✓ 83% ████████▍ test_spec.py::test_ref_in_spec_schemas[connector_spec6-True] ✓ 83% ████████▍ test_spec.py::test_ref_in_spec_schemas[connector_spec7-True] ✓ 84% ████████▍ test_spec.py::test_ref_in_spec_schemas[connector_spec8-True] ✓ 84% ████████▌ test_spec.py::test_ref_in_spec_schemas[connector_spec9-False] ✓ 84% ████████▌ test_spec.py::test_ref_in_spec_schemas[connector_spec10-True] ✓ 85% ████████▌ test_spec.py::test_oneof_usage[all_good] ✓ 85% ████████▌ test_spec.py::test_oneof_usage[top_level_node_is_not_of_object_type] ✓85% ████████▌ test_spec.py::test_oneof_usage[all_oneof_options_should_have_same_constant_attribute] ✓86% ████████▋ test_spec.py::test_oneof_usage[one_of_item_is_not_of_type_object] ✓86% ████████▋ test_spec.py::test_oneof_usage[no_common_property_for_all_oneof_subobjects] ✓86% ████████▋ test_spec.py::test_oneof_usage[two_common_properties_with_const_keyword] ✓87% ████████▋ test_spec.py::test_oneof_usage[default_keyword_in_common_property] ✓87% ████████▊ test_spec.py::test_validate_oauth_flow[connector_spec0-] ✓ 87% ████████▊ test_spec.py::test_validate_oauth_flow[connector_spec1-Specified oauth fields are missed from spec schema:] ✓88% ████████▊ test_spec.py::test_validate_oauth_flow[connector_spec2-] ✓ 88% ████████▊ test_spec.py::test_validate_oauth_flow[connector_spec3-Specified oauth fields are missed from spec schema:] ✓88% ████████▊ test_spec.py::test_validate_oauth_flow[connector_spec4-] ✓ 89% ████████▉ test_spec.py::test_validate_oauth_flow[connector_spec5-] ✓ 89% ████████▉ test_spec.py::test_validate_oauth_flow[connector_spec6-Specified oauth fields are missed from spec schema:] ✓89% ████████▉ test_spec.py::test_validate_oauth_flow[connector_spec7-] ✓ 90% ████████▉ test_spec.py::test_additional_properties_is_true[connector_spec0-expectation0] ✓90% █████████ test_spec.py::test_additional_properties_is_true[connector_spec1-expectation1] ✓90% █████████ test_spec.py::test_additional_properties_is_true[connector_spec2-expectation2] ✓90% █████████▏ test_spec.py::test_additional_properties_is_true[connector_spec3-expectation3] ✓91% █████████▏ test_spec.py::test_additional_properties_is_true[connector_spec4-expectation4] ✓91% █████████▎ test_spec_unit.py::TestEnvAttributes.test_correct_connector_image ✓91% █████████▎ test_spec_unit.py::TestEnvAttributes.test_connector_image_without_env ✓92% █████████▎ test_spec_unit.py::TestEnvAttributes.test_docker_image_env_ne_entrypoint ✓92% █████████▎ test_test_full_refresh.py::test_read_with_ignore_fields[no_ignored_fields_present] ✓92% █████████▎ test_test_full_refresh.py::test_read_with_ignore_fields[with_ignored_field] ✓93% █████████▍ test_test_full_refresh.py::test_read_with_ignore_fields[ignore_field_present_but_a_required_is_not] ✓93% █████████▍ test_utils.py::test_compare_two_records_nested_with_different_orders[obj10-obj20-True] ✓93% █████████▍ test_utils.py::test_compare_two_records_nested_with_different_orders[obj11-obj21-True] ✓94% █████████▍ test_utils.py::test_compare_two_records_nested_with_different_orders[obj12-obj22-False] ✓94% █████████▌ test_utils.py::test_compare_two_records_nested_with_different_orders[obj13-obj23-False] ✓94% █████████▌ test_utils.py::test_compare_two_records_nested_with_different_orders[obj14-obj24-True] ✓95% █████████▌ test_utils.py::test_compare_two_records_nested_with_different_orders[obj15-obj25-True] ✓95% █████████▌ test_utils.py::test_compare_two_records_nested_with_different_orders[obj16-obj26-False] ✓95% █████████▋ test_utils.py::test_exclude_fields ✓ 96% █████████▋ test_utils.py::test_successful_logs_reading ✓ 96% █████████▋{"type": "LOG", "log": {"level": "ERROR", "message": "Docker container was failed, code 1, error:\nSome Container Error"}}

      test_utils.py::test_failed_reading[interal_error] ✓              96% █████████▋{"type": "LOG", "log": {"level": "ERROR", "message": "Docker container was failed, code 1, error:\nTraceback (most recent call last):\n  File \"<stdin>\", line 1, in <module>\nKeyError: 'bbbb'"}}
     
      test_utils.py::test_failed_reading[traceback] ✓                  97% █████████▋{"type": "LOG", "log": {"level": "ERROR", "message": "Docker container was failed, code 1, error:\nLast Container Logs Line"}}yte-integrations:bases:source-acceptance-test:_unitTestCoverage
     
      test_utils.py::test_failed_reading[last_line] ✓                  97% █████████▊
      test_utils.py::test_docker_runner[standard] ✓                    97% █████████▊
      test_utils.py::test_docker_runner[waiting] ✓                     98% █████████▊
      test_utils.py::test_not_found_container ✓                        98% █████████▊
      test_utils.py::TestLoadYamlOrJsonPath.test_load_json ✓           98% █████████▉
      test_utils.py::TestLoadYamlOrJsonPath.test_load_yaml ✓           99% █████████▉
      test_utils.py::TestLoadYamlOrJsonPath.test_load_other ✓          99% █████████▉
      test_utils.py::test_find_all_values_for_key_in_schema[schema0-looking_for_this_key-expected_values0] ✓99% █████████▉
      test_utils.py::test_find_all_values_for_key_in_schema[schema1-looking_for_this_key-expected_values1] ✓99% ██████████
      test_utils.py::test_find_all_values_for_key_in_schema[schema2-looking_for_this_key-expected_values2] ✓100% ██████████=============================== warnings summary ===============================
     source_acceptance_test/tests/test_core.py:170
       /Users/sagewatterworth/airbyte-amazon-ads/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py:170: PytestUnknownMarkWarning: Unknown pytest.mark.backward_compatibility - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/mark.html
         @pytest.mark.backward_compatibility
     
     source_acceptance_test/tests/test_core.py:311
       /Users/sagewatterworth/airbyte-amazon-ads/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py:311: PytestUnknownMarkWarning: Unknown pytest.mark.backward_compatibility - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/mark.html
         @pytest.mark.backward_compatibility
     
     -- Docs: https://docs.pytest.org/en/stable/warnings.html
     
     ---------- coverage: platform darwin, python 3.9.0-final-0 -----------
     Name                                                 Stmts   Miss  Cover   Missing
     ----------------------------------------------------------------------------------
     source_acceptance_test/base.py                          10      4    60%   15-18
     source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
     source_acceptance_test/conftest.py                     164    164     0%   6-282
     source_acceptance_test/plugin.py                        48     48     0%   6-104
     source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
     source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
     source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
     source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
     source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
     source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
     source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
     source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
     ----------------------------------------------------------------------------------
     TOTAL                                                 1321    463    65%
     
     4 files skipped due to complete coverage.
     
     Required test coverage of 64.0% reached. Total coverage: 64.95%
     
     Results (194.65s):
          305 passed

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

[CLAassistant]

All committers have signed the CLA.

[sajarin]

assigned to @monai

[sage-watterworth]

Hey @monai! Thanks for the edits. I've updated the requested changes, let me know if you see anything else I can do.

[monai]

@sajarin the PR is ready to merge.

[sajarin]

/test connector=connectors/source-amazon-ads

:clock2: connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/3046393857 :x: connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/3046393857 :bug: https://gradle.com/s/sgp2hdiruym7s

Build Failed

Test summary info:

Could not find result summary

[sagew4]

Hi @sajarin - is the Build Fail: 'could not find result summary' preventing this PR from being merged? Is there something on my end I can fix? Or if not, how might this be addressed?

[monai]

2022-09-13T15:44:55.3993207Z > Task :airbyte-integrations:connectors:source-amazon-ads:sourceAcceptanceTest
2022-09-13T15:44:55.3993803Z WARNING: Error loading config file: .dockercfg: $HOME is not defined
2022-09-13T15:44:55.3995006Z ============================= test session starts ==============================
2022-09-13T15:44:55.3996134Z platform linux -- Python 3.9.11, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
2022-09-13T15:44:55.3996786Z rootdir: /test_input
2022-09-13T15:44:55.3997721Z plugins: sugar-0.9.5, mock-3.6.1, timeout-1.4.2, cov-3.0.0, requests-mock-1.9.3, hypothesis-6.54.5
2022-09-13T15:44:55.3998160Z collected 32 items
2022-09-13T15:44:55.3998362Z 
2022-09-13T15:47:27.6992928Z test_core.py ...........................                                 [ 84%]
2022-09-13T15:48:29.2992607Z time="2022-09-13T15:48:29Z" level=error msg="error waiting for container: unexpected EOF"
2022-09-13T15:48:29.3994335Z test_full_refresh.py .
2022-09-13T15:48:29.3994770Z 
2022-09-13T15:48:29.3995093Z FAILURE: Build failed with an exception.
2022-09-13T15:48:29.3996074Z 
2022-09-13T15:48:29.3996368Z * What went wrong:
2022-09-13T15:48:29.3997480Z Execution failed for task ':airbyte-integrations:connectors:source-amazon-ads:sourceAcceptanceTest'.
2022-09-13T15:48:29.3998718Z > Process 'command 'docker'' finished with non-zero exit value 125
2022-09-13T15:48:29.3999129Z 
2022-09-13T15:48:29.3999301Z * Try:
2022-09-13T15:48:29.3999929Z > Run with --stacktrace option to get the stack trace.
2022-09-13T15:48:29.4000691Z > Run with --info or --debug option to get more log output.
2022-09-13T15:48:29.4001389Z > Run with --scan to get full insights.
2022-09-13T15:48:29.4001707Z 
2022-09-13T15:48:29.4001985Z * Get more help at https://help.gradle.org
2022-09-13T15:48:29.4002340Z 
2022-09-13T15:48:29.4002555Z BUILD FAILED in 9m 29s
2022-09-13T15:48:29.4003360Z > Task :airbyte-integrations:connectors:source-amazon-ads:sourceAcceptanceTest FAILED
2022-09-13T15:48:29.4003813Z 
2022-09-13T15:48:29.4004216Z Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
2022-09-13T15:48:29.4004672Z 
2022-09-13T15:48:29.4017201Z You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
2022-09-13T15:48:29.4017780Z 
2022-09-13T15:48:29.4018253Z See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
2022-09-13T15:48:29.4018587Z 
2022-09-13T15:48:29.4018885Z Execution optimizations have been disabled for 1 invalid unit(s) of work during this build to ensure correctness.
2022-09-13T15:48:29.4019392Z Please consult deprecation warnings for more details.
2022-09-13T15:48:29.4019973Z 48 actionable tasks: 35 executed, 13 up-to-date
2022-09-13T15:48:29.4991930Z 
2022-09-13T15:48:29.4993010Z Publishing build scan...
2022-09-13T15:48:29.7993599Z https://gradle.com/s/sgp2hdiruym7s
2022-09-13T15:48:29.7994601Z 
2022-09-13T15:48:30.2084307Z success: false
2022-09-13T15:48:30.2084898Z python_info: 
2022-09-13T15:48:30.2085401Z java_info: 
2022-09-13T15:48:30.2159925Z ##[error]Process completed with exit code 1.

Some acceptance test failed by the log doesn't show which and why. The relevant log line:

2022-09-13T15:47:27.6992928Z test_core.py ...........................                                 [ 84%]

At the end is a link https://gradle.com/s/sgp2hdiruym7s but it's unaccessible without authentication. Maybe there's some information?

[sagew4]

@monai what are your suggestions for next steps? thanks!

[sajarin]

@sagew4 @monai I was running the tests locally using the same credentials and the CI and the tests were hanging. It's possible there's an issue with our current config or with our sandbox account. I'll take some time today to debug and report back. Thanks for the patience.

[sagew4]

@sajarin thanks for the update!

[sage-watterworth]

@monai @sajarin I've created a new PR (here) to add request params to Brands and Product campaigns for state filters as well.

[monai]

@sagew4 @monai I was running the tests locally using the same credentials and the CI and the tests were hanging. It's possible there's an issue with our current config or with our sandbox account. I'll take some time today to debug and report back. Thanks for the patience.

~~See my comment and the following discussion here https://github.com/airbytehq/airbyte/pull/15843#issuecomment-1243845628. I've even pointed Amazon Ads as an example~~

~~The issue is twofold: 1) one big configured_catalog.json is a bad pattern and doesn't scale; 2) Amazon Ads report generation is inherently slow and sometimes timeouts. However, solving 1) would reduce the chances of timeout.~~

Update:

The acceptance tests run for only one stream, sponsored_products_report_stream. So the only viable solution is to update start_date in config.json to the latest possible date, today, and hope for the best.

[sage-watterworth]

@monai I broke up the config catalogue file but if that is not necessary anymore and needs to be undone let me know. Also added a start_date of today. Tests are still passing on my end ✔️

[harshithmullapudi]

/test connector=connectors/source-amazon-ads

:clock2: connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/3100112294 :x: connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/3100112294 :bug: https://gradle.com/s/35abe5qdhuhns

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contai...
================== 1 failed, 45 passed in 3777.18s (1:02:57) ===================

[sage-watterworth]

hi harshithmullapudi is this error an issue with credentials on your end? or something in my code I'm missing or has errored

[harshithmullapudi]

Hey I think this is something related to permission on our end.

[sage-watterworth]

Hi harshithmullapudi, thanks for the update. let me know if theres anything I can do on my end to help out.

[grubberr]

@sage-watterworth I see you provided 2 very similar PR

https://github.com/airbytehq/airbyte/pull/15985 https://github.com/airbytehq/airbyte/pull/16829

Can you do next things:

1. Can you merge 2 PR to 1 PR? For example you can merge this PR 15985 -> 16829

2. Please don't touch source-acceptance-test config We tuned it optimal way to pass tests every day .

[sagew4]

Here is the new merged PR https://github.com/airbytehq/airbyte/pull/17475/

On Fri, Sep 30, 2022 at 3:18 AM Serhii Chvaliuk @.***> wrote:

@sage-watterworth https://github.com/sage-watterworth I see you provided 2 very similar PR

#15985 https://github.com/airbytehq/airbyte/pull/15985 #16829 https://github.com/airbytehq/airbyte/pull/16829

Can you do next things:

Can you merge 2 PR to 1 PR? For example you can merge this PR 15985 -> 16829 2.

Please don't touch source-acceptance-test config We tuned it optimal way to pass tests every day .

— Reply to this email directly, view it on GitHub https://github.com/airbytehq/airbyte/pull/15985#issuecomment-1263200282, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZGE7UA6VSNFNNCILL7Z72LWA2H4HANCNFSM57UE34WA . You are receiving this because you were mentioned.Message ID: @.***>

--

https://claravine.com/

Sage Watterworth Software Engineer P. (203) 910-7774 E. @.*** 7774