google-api-python-client icon indicating copy to clipboard operation
google-api-python-client copied to clipboard

HttpError 404 when requesting https://indexing.googleapis.com/v3/urlNotifications/metadata?url=

Open MumukoQAQ opened this issue 1 year ago • 8 comments

I submit my url on the urlNotifications endpoint to notify Google of removal, and when I view the submission with urlNotifications().getMetadat(url), he still says "HttpError 404 Details:" Requested entity was not found."

MumukoQAQ avatar Sep 12 '24 11:09 MumukoQAQ

@MumukoQAQ Can you please provide more information related to the issue that you're encountering? Probably a code snippet to replicate the issue, hiding any confidential information, and the error that you're receiving with a stack trace.

ohmayr avatar Sep 18 '24 03:09 ohmayr

from googleapiclient.discovery import build
from oauth2client.service_account import ServiceAccountCredentials
from datetime import datetime

class GooleIndexingApi:
    URL_UPDATE = "URL_UPDATED"
    URL_DELETED = "URL_DELETED"
def __init__(self, json_key_file: str, url_list: list, req_type: int):
    scopes = ["https://www.googleapis.com/auth/indexing", "https://www.googleapis.com/auth/webmasters.readonly"]
    credentials = ServiceAccountCredentials.from_json_keyfile_name(json_key_file, scopes=scopes)
    self.service = build('indexing', 'v3', credentials=credentials)
    self.search_console = build('searchconsole', 'v1', credentials=credentials)
    self.url_list = url_list
    self.req_type = req_type
    self.sucess_count = 0
    self.fail_count = 0

def structure_urls(self) -> list:
    req_body = []
    body_type = self.URL_UPDATE if self.req_type == 1 else self.URL_DELETED
    for url in self.url_list:
        req_body.append({"url": url, "type": body_type})
    return req_body

def batch_publish(self):
    url_list = self.structure_urls()
    batch = self.service.new_batch_http_request()
    for i, url_data in enumerate(url_list):
        batch.add(self.service.urlNotifications().publish(body=url_data), request_id=str(i),
                  callback=self.batch_callback)
    batch.execute()

def check_url(self, url):
    response = self.service.urlNotifications().getMetadata(url=url).execute()
    print(response)

def get_indexing_list(self, row_limit: int, expression: str):

    site_url = 'my_domain'
    start_date = '2020-01-01'
    end_date = datetime.today().strftime('%Y-%m-%d')

    request = {
        'startDate': start_date,
        'endDate': end_date,
        'dimensions': ['page'],
        'dimensionFilterGroups': [{
            'filters': [{
                'dimension': 'page',
                'operator': 'contains',
                'expression': expression
            }]
        }],
        'rowLimit': row_limit
    }
    response = self.search_console.searchanalytics().query(siteUrl=site_url, body=request).execute()
    index_urls = [row['keys'][0] for row in response['rows']]
    print(index_urls)

def batch_callback(self, request_id, response, exception):
    if exception is not None:
        self.fail_count += 1
    else:
        url = response["urlNotificationMetadata"]["url"]
        self.sucess_count += 1
if __name__ == '__main__':
    urls = ["my_url"]
    json_key_file = "my_json_key.json"
    req_type = 0
    google_api = GooleIndexingApi(json_key_file, urls, req_type)
    google_api.batch_publish()
    for url in urls:
        google_api.check_url(url)

This is my code, and after I notified Google to remove my url, it prompts when I check it

HttpError: <HttpError 404 when requesting https://indexing.googleapis.com/v3/urlNotifications/metadata?url=my_url&alt=json returned "Requested entity was not found.". Details: "Requested entity was not found.">

I thought it had a delay at first, then a few days later I checked the same url again and it was still the same.

MumukoQAQ avatar Sep 19 '24 02:09 MumukoQAQ

I'm running currently in a similar issue. Did you have already some new insights?

I'm a little bit confused about the error, in the documentation this error can result in 2 different issues.

Could it be, that this is the issue?

A resource associated with the request could not be found. If you have not used this API in the last two weeks, please re-deploy the App Engine app and try calling it again.

The post works for me on the other hand. I get the expected result.

xo12 avatar Oct 26 '24 00:10 xo12

I'm running currently in a similar issue. Did you have already some new insights?

I'm a little bit confused about the error, in the documentation this error can result in 2 different issues.

Could it be, that this is the issue?

A resource associated with the request could not be found. If you have not used this API in the last two weeks, please re-deploy the App Engine app and try calling it again.

The post works for me on the other hand. I get the expected result.

I redeployed it, and still have the same problem

MumukoQAQ avatar Nov 19 '24 11:11 MumukoQAQ

Any updates, I am having a similar issue

aitdahane avatar Dec 04 '24 13:12 aitdahane

Any updates, I am having a similar issue

no

MumukoQAQ avatar Dec 05 '24 00:12 MumukoQAQ

same issue here. I get a 200 response when I submit but when I check already submitted urls I get 404 NOT_FOUND

ATX92614 avatar Feb 11 '25 00:02 ATX92614

Same issue. I get a 200 response when I submit an URL. If I check the submitted URL I get a 404.

What irritates me is that I get the following response body when submitting the URL.

{                                                                                                                                                                                            
  "urlNotificationMetadata": {                                                                                                                                                               
    "url": "https://www.exp.de/"
  }
}

According to the documentation, I would have expected to receive a UrlNotificationMetadata object.

Google responds to successful Indexing API calls with the an HTTP 200. An HTTP 200 response means that Google may remove this URL from the index. The body of the response contains a UrlNotificationMetadata object, whose fields correspond to those returned by a notification status request.

For example:

{
  'urlNotificationMetadata': {
    'url': 'https: //www.exp.de/', 
    'latestUpdate': { 
      'url': 'https: //www.exp.de/', 
      'type': 'URL_UPDATED', 
      'notifyTime': '2025-03-03T09:00:08.031188922Z'
    }
  }
}

netzstreuner avatar Mar 05 '25 08:03 netzstreuner

Same...

rickd3ckard avatar May 30 '25 09:05 rickd3ckard

I got to the bottom of it. We can not use it unless.. :

Image

rickd3ckard avatar May 31 '25 13:05 rickd3ckard