googleads
Consistent 503 when trying to list batch job results
Describe the bug: Hi team,
I was referred to you by the Google Ads API support team, because they couldn't provide any guidance regarding this error. We're an integrator who serve a Shopping Ads integration for numerous advertisers. For one of them we started seeing this error on each batch job that we try to get results from, since at least August 19th (maybe longer):
google.api_core.exceptions.ServiceUnavailable: 503 The service is currently unavailable.
This hasn't always been the case, and we can't find what triggers it. I've tried creating new batch jobs, even with different operations, but it keeps happening. Also, we integrate with multiple campaigns of theirs, but this campaign is the only one with the issue. Is it possible to get your assistance?
Best, Ciel
Steps to Reproduce:
- Create batch job via
mutate_batch_jobofBatchJobServiceClient - Upload
listing_group(ad group criterion)createandremoveoperations viaadd_batch_job_operations - List results once status is done, via:
client: GoogleAdsClient = get_google_ads_client(client_customer_id=client_customer_id)
batch_job_service = client.get_service('BatchJobService')
resource_name = batch_job_service.batch_job_path(
customer_id=client_customer_id, batch_job_id=batch_job_id
)
list_results_request = client.get_type('ListBatchJobResultsRequest')
list_results_request.resource_name = resource_name
list_results_request.page_size = download_page_size
list_results_request.response_content_type = (
client.enums.ResponseContentTypeEnum.MUTABLE_RESOURCE
)
results = batch_job_service.list_batch_job_results(request=list_results_request)
Expected behavior:
A successful response.
Client library version and API version: Client library version: 17.0.0 Google Ads API version: v11
Request/Response Logs:
Click for logs
-------
Method: /google.ads.googleads.v11.services.GoogleAdsService/SearchStream
Host: googleads.googleapis.com
Headers: {
"developer-token": "REDACTED",
"login-customer-id": "REDACTED",
"x-goog-api-client": "gl-python/3.8.13 grpc/1.46.0 gax/2.8.1 gccl/17.0.0",
"x-goog-request-params": "customer_id=REDACTED"
}
Request: customer_id: "REDACTED"
query: "\n SELECT\n batch_job.id,\n batch_job.resource_name,\n batch_job.status,\n batch_job.metadata.operation_count\n FROM batch_job\n WHERE batch_job.id = 10314070081\n "
Response
-------
Headers: {
"content-disposition": "attachment",
"request-id": "YohZrWs16vM2-ET7YfYF9g"
}
Response: results {
batch_job {
resource_name: "customers/REDACTED/batchJobs/10314070081"
metadata {
operation_count: 7358
}
status: DONE
id: 10314070081
}
}
field_mask {
paths: "batch_job.id"
paths: "batch_job.resource_name"
paths: "batch_job.status"
paths: "batch_job.metadata.operation_count"
}
request_id: "YohZrWs16vM2-ET7YfYF9g"
-------
Method: /google.ads.googleads.v11.services.BatchJobService/ListBatchJobResults
Host: googleads.googleapis.com
Headers: {
"developer-token": "REDACTED",
"login-customer-id": "REDACTED",
"x-goog-api-client": "gl-python/3.8.13 grpc/1.46.0 gax/2.8.1 gccl/17.0.0",
"x-goog-request-params": "resource_name=customers/REDACTED/batchJobs/10314070081"
}
Request: resource_name: "customers/REDACTED/batchJobs/10314070081"
page_size: 1000
response_content_type: MUTABLE_RESOURCE
Response
-------
Headers: {}
Fault: {
"created": "@1661426012.154116431",
"description": "Error received from peer ipv4:REDACTED",
"file": "src/core/lib/surface/call.cc",
"file_line": 952,
"grpc_message": "The service is currently unavailable.",
"grpc_status": 14
}
[2022-08-25 13:13:32,155 - WARNING] Request made: ClientCustomerId: REDACTED, Host: googleads.googleapis.com, Method: /google.ads.googleads.v11.services.BatchJobService/ListBatchJobResults, RequestId: None, IsFault: True, FaultMessage: The service is currently unavailable.
[2022-08-25 13:13:32,170 - DEBUG] STORE took 0:01:00.081909 (wall), 0:00:00.811298 (CPU).
[2022-08-25 13:13:32,170 - ERROR] Got an error when retrieving the BJ, rescheduling store
Traceback (most recent call last):
File "/nix/store/z7ka6np07mx5p7s0jwd3hh297jbn6m89-python3-3.8.13-env/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
return callable_(*args, **kwargs)
File "/nix/store/z7ka6np07mx5p7s0jwd3hh297jbn6m89-python3-3.8.13-env/lib/python3.8/site-packages/grpc/_interceptor.py", line 216, in __call__
response, ignored_call = self._with_call(request,
File "/nix/store/z7ka6np07mx5p7s0jwd3hh297jbn6m89-python3-3.8.13-env/lib/python3.8/site-packages/grpc/_interceptor.py", line 257, in _with_call
return call.result(), call
File "/nix/store/z7ka6np07mx5p7s0jwd3hh297jbn6m89-python3-3.8.13-env/lib/python3.8/site-packages/grpc/_channel.py", line 343, in result
raise self
File "/nix/store/z7ka6np07mx5p7s0jwd3hh297jbn6m89-python3-3.8.13-env/lib/python3.8/site-packages/grpc/_interceptor.py", line 241, in continuation
response, call = self._thunk(new_method).with_call(
File "/nix/store/z7ka6np07mx5p7s0jwd3hh297jbn6m89-python3-3.8.13-env/lib/python3.8/site-packages/grpc/_interceptor.py", line 266, in with_call
return self._with_call(request,
File "/nix/store/z7ka6np07mx5p7s0jwd3hh297jbn6m89-python3-3.8.13-env/lib/python3.8/site-packages/grpc/_interceptor.py", line 257, in _with_call
return call.result(), call
File "/nix/store/z7ka6np07mx5p7s0jwd3hh297jbn6m89-python3-3.8.13-env/lib/python3.8/site-packages/grpc/_channel.py", line 343, in result
raise self
File "/nix/store/z7ka6np07mx5p7s0jwd3hh297jbn6m89-python3-3.8.13-env/lib/python3.8/site-packages/grpc/_interceptor.py", line 241, in continuation
response, call = self._thunk(new_method).with_call(
File "/nix/store/z7ka6np07mx5p7s0jwd3hh297jbn6m89-python3-3.8.13-env/lib/python3.8/site-packages/grpc/_interceptor.py", line 266, in with_call
return self._with_call(request,
File "/nix/store/z7ka6np07mx5p7s0jwd3hh297jbn6m89-python3-3.8.13-env/lib/python3.8/site-packages/grpc/_interceptor.py", line 254, in _with_call
call = self._interceptor.intercept_unary_unary(continuation,
File "/nix/store/z7ka6np07mx5p7s0jwd3hh297jbn6m89-python3-3.8.13-env/lib/python3.8/site-packages/google/ads/googleads/interceptors/exception_interceptor.py", line 99, in intercept_unary_unary
self._handle_grpc_failure(response)
File "/nix/store/z7ka6np07mx5p7s0jwd3hh297jbn6m89-python3-3.8.13-env/lib/python3.8/site-packages/google/ads/googleads/interceptors/exception_interceptor.py", line 71, in _handle_grpc_failure
raise self._get_error_from_response(response)
File "/nix/store/z7ka6np07mx5p7s0jwd3hh297jbn6m89-python3-3.8.13-env/lib/python3.8/site-packages/grpc/_interceptor.py", line 241, in continuation
response, call = self._thunk(new_method).with_call(
File "/nix/store/z7ka6np07mx5p7s0jwd3hh297jbn6m89-python3-3.8.13-env/lib/python3.8/site-packages/grpc/_channel.py", line 957, in with_call
return _end_unary_response_blocking(state, call, True, None)
File "/nix/store/z7ka6np07mx5p7s0jwd3hh297jbn6m89-python3-3.8.13-env/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "The service is currently unavailable."
debug_error_string = "{"created":"@1661426012.154116431","description":"Error received from peer ipv4:REDACTED","file":"src/core/lib/surface/call.cc","file_line":952,"grpc_message":"The service is currently unavailable.","grpc_status":14}"
>
Anything else we should know about your project / environment: Google Support issue ref:_00D1U1174p._5004Q2dn4In:ref
@ceijssen thanks for sharing these details. At first glance this looks like an issue with you connection with the API servers, but it's unclear what is causing the problem. Do you see this same error when using other services, or is it just with the batch job service?
Hi @BenRKarl , we're only seeing it for the batch job service and only for this one campaign in particular.
Hi @BenRKarl is there anything else we can do to investigate what's causing this issue?
@BenRKarl We are also facing this issue with GoogleAdsService when we try to get some reports for a number of manager clients (~12) having many accounts under them (~30 under each) concurrently. We are using SearchStream to get the data.
Hi All, sorry for the delayed response here, I was just checking whether you're still encountering these issues or not.
@BenRKarl we stopped trying to interface with this particular campaign, and haven't seen this issue with other campaigns, so this issue can be closed if you like.
@ceijssen thank you, closing this for now but please add more details if this happens again.