:bug:[source-google-ads] Fixing an error in code that checks custom GAQL queries

[dmykhailov]

What

The issue here is with check for incremental queries. Some tables have a limited lookback time frame to be valid. For instance, click_view should not be asked for data older than 90 days. Hardcoded 1980-01-01 is much older than 90 days.

How

As the fix for the check procedure I've added a much more recent time frame: last month. The period is calculated based on today's date during the check.

Also it would be nice to see the exact query being executed during the check in the logs.

User Impact

People will be able to use "click_view" in custom GAQL queries.

Can this PR be safely reverted and rolled back?

• [X] YES 💚

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 27, 2024 3:27pm

[CLAassistant]

All committers have signed the CLA.

[dmykhailov]

@marcosmarxm sure. Here is the error log from the server:

airbyte-server                    | 2024-05-07 08:59:31 INFO i.a.p.j.e.LoggingJobErrorReportingClient(reportJobFailureReason):28 - Report Job Error -> workspaceId: aae92b4d-8d4b-4596-9cd0-f0f3cd65019c, dockerImage: airbyte/source-google-ads:3.4.1, failureReason: io.airbyte.config.FailureReason@865d3b[failureOrigin=source,failureType=system_error,internalMessage=(<_InactiveRpcError of RPC that terminated with:
airbyte-server                    |     status = StatusCode.INVALID_ARGUMENT
airbyte-server                    |     details = "Request contains an invalid argument."
airbyte-server                    |     debug_error_string = "UNKNOWN:Error received from peer ipv4:142.251.31.95:443 {grpc_message:"Request contains an invalid argument.", grpc_status:3, created_time:"2024-05-07T08:59:30.865496668+00:00"}"
airbyte-server                    | >, <_InactiveRpcError of RPC that terminated with:
airbyte-server                    |     status = StatusCode.INVALID_ARGUMENT
airbyte-server                    |     details = "Request contains an invalid argument."
airbyte-server                    |     debug_error_string = "UNKNOWN:Error received from peer ipv4:142.251.31.95:443 {grpc_message:"Request contains an invalid argument.", grpc_status:3, created_time:"2024-05-07T08:59:30.865496668+00:00"}"
airbyte-server                    | >, errors {
airbyte-server                    |   error_code {
airbyte-server                    |     request_error: UNKNOWN
airbyte-server                    |   }
airbyte-server                    |   message: "The requested date is too old. It cannot be older than 90 days."
airbyte-server                    | }
airbyte-server                    | request_id: "apM-g78Jb6ua6ND4n-sMYg"
airbyte-server                    | , 'apM-g78Jb6ua6ND4n-sMYg'),externalMessage=The requested date is too old. It cannot be older than 90 days.,metadata=io.airbyte.config.Metadata@23b19bbb[additionalProperties={attemptNumber=null, jobId=null, from_trace_message=true, connector_command=check}],stacktrace=Traceback (most recent call last):
airbyte-server                    |   File "/airbyte/integration_code/source_google_ads/source.py", line 208, in check_connection
airbyte-server                    |     response = google_api.send_request(
airbyte-server                    |   File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry
airbyte-server                    |     ret = target(*args, **kwargs)
airbyte-server                    |   File "/airbyte/integration_code/source_google_ads/google_ads.py", line 88, in send_request
airbyte-server                    |     return [self.ga_service(login_customer_id).search(search_request)]
airbyte-server                    |   File "/usr/local/lib/python3.9/site-packages/google/ads/googleads/v15/services/services/google_ads_service/client.py", line 4170, in search
airbyte-server                    |     response = rpc(
airbyte-server                    |   File "/usr/local/lib/python3.9/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
airbyte-server                    |     return wrapped_func(*args, **kwargs)
airbyte-server                    |   File "/usr/local/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 76, in error_remapped_callable
airbyte-server                    |     return callable_(*args, **kwargs)
airbyte-server                    |   File "/usr/local/lib/python3.9/site-packages/grpc/_interceptor.py", line 277, in __call__
airbyte-server                    |     response, ignored_call = self._with_call(
airbyte-server                    |   File "/usr/local/lib/python3.9/site-packages/grpc/_interceptor.py", line 332, in _with_call
airbyte-server                    |     return call.result(), call
airbyte-server                    |   File "/usr/local/lib/python3.9/site-packages/grpc/_interceptor.py", line 171, in result
airbyte-server                    |     raise self._exception
airbyte-server                    |   File "/usr/local/lib/python3.9/site-packages/grpc/_interceptor.py", line 315, in continuation
airbyte-server                    |     response, call = self._thunk(new_method).with_call(
airbyte-server                    |   File "/usr/local/lib/python3.9/site-packages/grpc/_interceptor.py", line 343, in with_call
airbyte-server                    |     return self._with_call(
airbyte-server                    |   File "/usr/local/lib/python3.9/site-packages/grpc/_interceptor.py", line 332, in _with_call
airbyte-server                    |     return call.result(), call
airbyte-server                    |   File "/usr/local/lib/python3.9/site-packages/grpc/_interceptor.py", line 171, in result
airbyte-server                    |     raise self._exception
airbyte-server                    |   File "/usr/local/lib/python3.9/site-packages/grpc/_interceptor.py", line 315, in continuation
airbyte-server                    |     response, call = self._thunk(new_method).with_call(
airbyte-server                    |   File "/usr/local/lib/python3.9/site-packages/grpc/_interceptor.py", line 343, in with_call
airbyte-server                    |     return self._with_call(
airbyte-server                    |   File "/usr/local/lib/python3.9/site-packages/grpc/_interceptor.py", line 329, in _with_call
airbyte-server                    |     call = self._interceptor.intercept_unary_unary(
airbyte-server                    |   File "/usr/local/lib/python3.9/site-packages/google/ads/googleads/interceptors/exception_interceptor.py", line 99, in intercept_unary_unary
airbyte-server                    |     self._handle_grpc_failure(response)
airbyte-server                    |   File "/usr/local/lib/python3.9/site-packages/google/ads/googleads/interceptors/exception_interceptor.py", line 71, in _handle_grpc_failure
airbyte-server                    |     raise self._get_error_from_response(response)
airbyte-server                    | google.ads.googleads.errors.GoogleAdsException: (<_InactiveRpcError of RPC that terminated with:
airbyte-server                    |     status = StatusCode.INVALID_ARGUMENT
airbyte-server                    |     details = "Request contains an invalid argument."
airbyte-server                    |     debug_error_string = "UNKNOWN:Error received from peer ipv4:142.251.31.95:443 {grpc_message:"Request contains an invalid argument.", grpc_status:3, created_time:"2024-05-07T08:59:30.865496668+00:00"}"
airbyte-server                    | >, <_InactiveRpcError of RPC that terminated with:
airbyte-server                    |     status = StatusCode.INVALID_ARGUMENT
airbyte-server                    |     details = "Request contains an invalid argument."
airbyte-server                    |     debug_error_string = "UNKNOWN:Error received from peer ipv4:142.251.31.95:443 {grpc_message:"Request contains an invalid argument.", grpc_status:3, created_time:"2024-05-07T08:59:30.865496668+00:00"}"
airbyte-server                    | >, errors {
airbyte-server                    |   error_code {
airbyte-server                    |     request_error: UNKNOWN
airbyte-server                    |   }
airbyte-server                    |   message: "The requested date is too old. It cannot be older than 90 days."
airbyte-server                    | }
airbyte-server                    | request_id: "apM-g78Jb6ua6ND4n-sMYg"
airbyte-server                    | , 'apM-g78Jb6ua6ND4n-sMYg')

GAQL Sample:

SELECT 
customer.id, 
customer.descriptive_name, 
customer.resource_name, 
metrics.clicks, 
ad_group.id, 
ad_group.name, 
ad_group.resource_name, 
ad_group.status, 
campaign.id, 
campaign.name, 
campaign.resource_name, 
campaign.status, 
segments.ad_network_type, 
segments.click_type, 
segments.date, 
segments.device, 
segments.slot, 
segments.month_of_year, 
click_view.gclid, 
click_view.keyword, 
click_view.keyword_info.match_type, 
click_view.keyword_info.text, 
click_view.ad_group_ad, 
click_view.campaign_location_target, 
click_view.area_of_interest.region, 
click_view.area_of_interest.most_specific, 
click_view.area_of_interest.metro, 
click_view.area_of_interest.country, 
click_view.area_of_interest.city, 
click_view.location_of_presence.city, 
click_view.location_of_presence.country, 
click_view.location_of_presence.metro, 
click_view.location_of_presence.most_specific, 
click_view.location_of_presence.region, 
click_view.resource_name, 
click_view.page_number, 
click_view.user_list 
FROM click_view

[dmykhailov]

@marcosmarxm

I've added the sample GAQL and the exact error message. Regarding the rest of the requested changes:

• I don't see a suitable unit test to modify, the one that is there tests that provided date is appended to an incremental GAQL query, and in that case it does not matter what is the source of the date string.
• is 'dockerImageTag' property in 'metadata.yaml' file the one where I should bump the connector version?
• where the change log record should be added?

Kind regards, Dmytro

[riiji]

Got the same issue. It is possible to create this source through API and even create a connection. However, during the sync process it throws the same error (The requested date is too old. It cannot be older than 90 days.)

[dmykhailov]

@marcosmarxm please let me know what else should be done to make this fix merged?

[lazebnyi]

PR added date range query validation for click_view. Additionally, an issue with check connection for custom queries was fixed. This PR will be merged on Monday, May 27th.