talk icon indicating copy to clipboard operation
talk copied to clipboard

Scraper failing, unsure why

Open WyattJChalifoux opened this issue 3 years ago • 2 comments
trafficstars

I am trying to use the scraper to generate titles for stories, here is the error it gives me:

{
    "errors": [
        {
            "message": "Could not scrape the URL http://localhost:8080/my/test/url/.",
            "locations": [
                {
                    "line": 1,
                    "column": 3
                }
            ],
            "path": [
                "debugScrapeStoryMetadata"
            ],
            "extensions": {
                "code": "SCRAPE_FAILED",
                "id": "1d8ed340-dfef-11ec-8f87-2500d726475a",
                "traceID": "1d8e0ff0-dfef-11ec-8f87-2500d726475a",
                "type": "INVALID_REQUEST_ERROR",
                "message": "Could not scrape the URL http://localhost:8080/my/test/url/."
            }
        }
    ],
    "data": {
        "debugScrapeStoryMetadata": null
    }
}

As for metadata, I have tried various formats without success: <title>My Title</title> <meta property="og:title" content="My Title"> <meta property="title" content="My Title">

Rescraping the metadata from the admin UI fails as well.

Any ideas as to what might be causing this?

WyattJChalifoux avatar May 30 '22 08:05 WyattJChalifoux

Why is the '.' behind the last slash in the url? Try to do a 'curl' to the url to see if that works

remko79 avatar May 30 '22 08:05 remko79

Yeah, that scared me at first too, but I think the period is just because it's part of the message. It's saying "could not scrap the url {the url}." I tried going to the url anyways, and it just loads the webpage normally with a status code of 200.

WyattJChalifoux avatar Jun 01 '22 19:06 WyattJChalifoux