actor-page-analyzer icon indicating copy to clipboard operation
actor-page-analyzer copied to clipboard

Timeout seems to be needed on some things that have cloudfront and or cloudflare BUG

Open devcoinfet opened this issue 5 years ago • 1 comments

also this error is encountered not sure if that has anything to do with it or if it is 2 sep issues

================================ 2020-08-16T09:06:37.279Z '0.01s' 'analysisStarted' 2020-08-16T09:06:37.438Z '0.16s' 'scrapping started' 2020-08-16T09:06:37.645Z '0.21s' 'initial response' (node:17514) UnhandledPromiseRejectionWarning: ApifyClientError: The "value" parameter must be a String or Buffer when "options.contentType" is specified. at exports.checkParamOrThrow (/root/Desktop/backupscanner/actor-page-analyzer/node_modules/apify-client/build/utils.js:222:15) at validateSetValueParams (/root/Desktop/backupscanner/actor-page-analyzer/node_modules/apify/build/key_value_store.js:79:34) at KeyValueStoreLocal.setValue (/root/Desktop/backupscanner/actor-page-analyzer/node_modules/apify/build/key_value_store.js:434:5) at Object.setValue (/root/Desktop/backupscanner/actor-page-analyzer/node_modules/apify/build/key_value_store.js:732:16) at process._tickCallback (internal/process/next_tick.js:68:7) (node:17514) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2) (node:17514) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

devcoinfet avatar Aug 16 '20 09:08 devcoinfet

I had to build a module in python for my framework to generate the input.js file i also looked into the package.json and found how you are running it and it works for some but has issues with the multi json format

{ "pages": [

    {
        "url": "https://community.example.com",
        "searchFor": [
            ""
        ]
    },
    {
        "url": "http://apiguide.example.com",
        "searchFor": [
            ""
        ]
    },
    {
        "url": "http://community.example.com",
        "searchFor": [
            ""
        ]
    },
    {
        "url": "https://docs.example.com",
        "searchFor": [
            ""
        ]
    },
    {
        "url": "https://dev-hackathon.example.com",
        "searchFor": [
            ""
        ]
    },
    {
        "url": "http://docs.example.com",
        "searchFor": [
            ""
        ]
    },
    {
        "url": "https://example.com",
        "searchFor": [
            ""
        ]
    },
    {
        "url": "https://www.example.com",
        "searchFor": [
            ""
        ]
    }
],
"tests": [
    "XHR"
]

}

devcoinfet avatar Aug 16 '20 09:08 devcoinfet