actor-page-analyzer
actor-page-analyzer copied to clipboard
Timeout seems to be needed on some things that have cloudfront and or cloudflare BUG
also this error is encountered not sure if that has anything to do with it or if it is 2 sep issues
================================ 2020-08-16T09:06:37.279Z '0.01s' 'analysisStarted' 2020-08-16T09:06:37.438Z '0.16s' 'scrapping started' 2020-08-16T09:06:37.645Z '0.21s' 'initial response' (node:17514) UnhandledPromiseRejectionWarning: ApifyClientError: The "value" parameter must be a String or Buffer when "options.contentType" is specified. at exports.checkParamOrThrow (/root/Desktop/backupscanner/actor-page-analyzer/node_modules/apify-client/build/utils.js:222:15) at validateSetValueParams (/root/Desktop/backupscanner/actor-page-analyzer/node_modules/apify/build/key_value_store.js:79:34) at KeyValueStoreLocal.setValue (/root/Desktop/backupscanner/actor-page-analyzer/node_modules/apify/build/key_value_store.js:434:5) at Object.setValue (/root/Desktop/backupscanner/actor-page-analyzer/node_modules/apify/build/key_value_store.js:732:16) at process._tickCallback (internal/process/next_tick.js:68:7) (node:17514) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2) (node:17514) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
I had to build a module in python for my framework to generate the input.js file i also looked into the package.json and found how you are running it and it works for some but has issues with the multi json format
{ "pages": [
{
"url": "https://community.example.com",
"searchFor": [
""
]
},
{
"url": "http://apiguide.example.com",
"searchFor": [
""
]
},
{
"url": "http://community.example.com",
"searchFor": [
""
]
},
{
"url": "https://docs.example.com",
"searchFor": [
""
]
},
{
"url": "https://dev-hackathon.example.com",
"searchFor": [
""
]
},
{
"url": "http://docs.example.com",
"searchFor": [
""
]
},
{
"url": "https://example.com",
"searchFor": [
""
]
},
{
"url": "https://www.example.com",
"searchFor": [
""
]
}
],
"tests": [
"XHR"
]
}