app-store-scraper
app-store-scraper copied to clipboard
facundoolano
Error 404 when running store.app
- Operating System: Ubuntu 18.04 x64
- Node version: v8.10.0
- app-play-scraper version: v0.16.3
Description:
I am using this Library on node-js with a discord bot. It's hosted on a VPS. The bot checks for details of various apps regularly, with a gap of 10 seconds between each time store.app is ran. Recently, (starting yesterday), Error 404 responses are randomly returned when the iTunes Lookup API is used. Around 5 - 10% of the requests fail. I know that there is nothing the dev can do about it, I'm just looking to learn what could be the possible cause and if there is any fix.
Example code:
store.app({ appId: "com.miniclip.agar.io", country: "us" });
Error message:
function vap{ response:
IncomingMessage {
_readableState:
ReadableState {
objectMode: false,
highWaterMark: 16384,
buffer: [Object],
length: 0,
pipes: null,
pipesCount: 0,
flowing: true,
ended: true,
endEmitted: true,
reading: false,
sync: true,
needReadable: false,
emittedReadable: false,
readableListening: false,
resumeScheduled: false,
destroyed: false,
defaultEncoding: 'utf8',
awaitDrain: 0,
readingMore: false,
decoder: null,
encoding: null },
readable: false,
domain: null,
_events:
{ end: [Array],
close: [Array],
data: [Function],
error: [Function] },
_eventsCount: 4,
_maxListeners: undefined,
socket:
TLSSocket {
_tlsOptions: [Object],
_secureEstablished: true,
_securePending: false,
_newSessionPending: false,
_controlReleased: true,
_SNICallback: null,
servername: null,
npnProtocol: false,
alpnProtocol: false,
authorized: true,
authorizationError: null,
encrypted: true,
_events: [Object],
_eventsCount: 9,
connecting: false,
_hadError: false,
_handle: null,
_parent: null,
_host: 'itunes.apple.com',
_readableState: [Object],
readable: false,
domain: null,
_maxListeners: undefined,
_writableState: [Object],
writable: false,
allowHalfOpen: false,
_bytesDispatched: 124,
_sockname: null,
_pendingData: null,
_pendingEncoding: '',
server: undefined,
_server: null,
ssl: null,
_requestCert: true,
_rejectUnauthorized: true,
parser: null,
_httpMessage: [Object],
read: [Function],
_consuming: true,
_idleNext: null,
_idlePrev: null,
_idleTimeout: -1,
[Symbol(asyncId)]: 1758,
[Symbol(bytesRead)]: 1523 },
connection:
TLSSocket {
_tlsOptions: [Object],
_secureEstablished: true,
_securePending: false,
_newSessionPending: false,
_controlReleased: true,
_SNICallback: null,
servername: null,
npnProtocol: false,
alpnProtocol: false,
authorized: true,
authorizationError: null,
encrypted: true,
_events: [Object],
_eventsCount: 9,
connecting: false,
_hadError: false,
_handle: null,
_parent: null,
_host: 'itunes.apple.com',
_readableState: [Object],
readable: false,
domain: null,
_maxListeners: undefined,
_writableState: [Object],
writable: false,
allowHalfOpen: false,
_bytesDispatched: 124,
_sockname: null,
_pendingData: null,
_pendingEncoding: '',
server: undefined,
_server: null,
ssl: null,
_requestCert: true,
_rejectUnauthorized: true,
parser: null,
_httpMessage: [Object],
read: [Function],
_consuming: true,
_idleNext: null,
_idlePrev: null,
_idleTimeout: -1,
[Symbol(asyncId)]: 1758,
[Symbol(bytesRead)]: 1523 },
httpVersionMajor: 1,
httpVersionMinor: 1,
httpVersion: '1.1',
complete: true,
headers:
{ 'x-apple-jingle-correlation-key': 'Z5SDCVDZO2TY6S57NMYO4U3MDA',
'x-apple-request-uuid': 'cf643154-7976-a78f-4bbf-6b30ee536c18',
'x-apple-translated-wo-url': '/WebObjects/MZStoreServices.woa/ws/wsLookup?bundleId=com.miniclip.agar.io&country=us&entity=software&urlDesc=',
'apple-tk': 'false',
'x-b3-spanid': '9c95d6f7edbef463',
'content-type': 'text/plain; charset=UTF-8',
b3: 'cf6431547976a78f4bbf6b30ee536c18-9c95d6f7edbef463',
'x-b3-traceid': 'cf6431547976a78f4bbf6b30ee536c18',
'x-webobjects-loadaverage': '0',
'last-modified': 'Thu, 27 Aug 2020 15:48:01 GMT',
'apple-seq': '0',
'access-control-allow-origin': '*',
'apple-originating-system': 'MZStore',
'strict-transport-security': 'max-age=31536000',
'x-frame-options': 'SAMEORIGIN',
'x-apple-orig-url': 'https://itunes.apple.com/lookup?bundleId=com.miniclip.agar.io&country=us&entity=software',
'x-apple-application-site': 'MR22',
'apple-timing-app': '2 ms',
'x-apple-application-instance': '3004002',
'cache-control': 'max-age=3600',
date: 'Thu, 27 Aug 2020 15:48:01 GMT',
'content-length': '134',
'x-cache': 'TCP_REFRESH_MISS from a23-218-93-133.deploy.akamaitechnologies.com (AkamaiGHost/10.1.2-30481071) (S)',
'x-true-cache-key': '/L/itunes.apple.com/lookup vcd=2897 ci2=bundleId=com.miniclip.agar.io&country=us&entity=software__',
connection: 'close',
'x-cache-remote': 'TCP_REFRESH_MISS from a23-218-93-164.deploy.akamaitechnologies.com (AkamaiGHost/10.1.2-30481071) (S)',
'x-apple-partner': 'origin.0' },
rawHeaders:
[ 'x-apple-jingle-correlation-key',
'Z5SDCVDZO2TY6S57NMYO4U3MDA',
'x-apple-request-uuid',
'cf643154-7976-a78f-4bbf-6b30ee536c18',
'x-apple-translated-wo-url',
'/WebObjects/MZStoreServices.woa/ws/wsLookup?bundleId=com.miniclip.agar.io&country=us&entity=software&urlDesc=',
'apple-tk',
'false',
'x-b3-spanid',
'9c95d6f7edbef463',
'Content-Type',
'text/plain; charset=UTF-8',
'b3',
'cf6431547976a78f4bbf6b30ee536c18-9c95d6f7edbef463',
'x-b3-traceid',
'cf6431547976a78f4bbf6b30ee536c18',
'x-webobjects-loadaverage',
'0',
'Last-Modified',
'Thu, 27 Aug 2020 15:48:01 GMT',
'apple-seq',
'0',
'Access-Control-Allow-Origin',
'*',
'apple-originating-system',
'MZStore',
'strict-transport-security',
'max-age=31536000',
'x-frame-options',
'SAMEORIGIN',
'x-apple-orig-url',
'https://itunes.apple.com/lookup?bundleId=com.miniclip.agar.io&country=us&entity=software',
'x-apple-application-site',
'MR22',
'apple-timing-app',
'2 ms',
'x-apple-application-instance',
'3004002',
'Cache-Control',
'max-age=3600',
'Date',
'Thu, 27 Aug 2020 15:48:01 GMT',
'Content-Length',
'134',
'X-Cache',
'TCP_REFRESH_MISS from a23-218-93-133.deploy.akamaitechnologies.com (AkamaiGHost/10.1.2-30481071) (S)',
'X-True-Cache-Key',
'/L/itunes.apple.com/lookup vcd=2897 ci2=bundleId=com.miniclip.agar.io&country=us&entity=software__',
'Connection',
'close',
'X-Cache-Remote',
'TCP_REFRESH_MISS from a23-218-93-164.deploy.akamaitechnologies.com (AkamaiGHost/10.1.2-30481071) (S)',
'X-Apple-Partner',
'origin.0' ],
trailers: {},
rawTrailers: [],
upgrade: false,
url: '',
method: null,
statusCode: 404,
statusMessage: 'Not Found',
client:
TLSSocket {
_tlsOptions: [Object],
_secureEstablished: true,
_securePending: false,
_newSessionPending: false,
_controlReleased: true,
_SNICallback: null,
servername: null,
npnProtocol: false,
alpnProtocol: false,
authorized: true,
authorizationError: null,
encrypted: true,
_events: [Object],
_eventsCount: 9,
connecting: false,
_hadError: false,
_handle: null,
_parent: null,
_host: 'itunes.apple.com',
_readableState: [Object],
readable: false,
domain: null,
_maxListeners: undefined,
_writableState: [Object],
writable: false,
allowHalfOpen: false,
_bytesDispatched: 124,
_sockname: null,
_pendingData: null,
_pendingEncoding: '',
server: undefined,
_server: null,
ssl: null,
_requestCert: true,
_rejectUnauthorized: true,
parser: null,
_httpMessage: [Object],
read: [Function],
_consuming: true,
_idleNext: null,
_idlePrev: null,
_idleTimeout: -1,
[Symbol(asyncId)]: 1758,
[Symbol(bytesRead)]: 1523 },
_consuming: true,
_dumped: false,
req:
ClientRequest {
domain: null,
_events: [Object],
_eventsCount: 5,
_maxListeners: undefined,
output: [],
outputEncodings: [],
outputCallbacks: [],
outputSize: 0,
writable: true,
_last: true,
upgrading: false,
chunkedEncoding: false,
shouldKeepAlive: false,
useChunkedEncodingByDefault: false,
sendDate: false,
_removedConnection: false,
_removedContLen: false,
_removedTE: false,
_contentLength: 0,
_hasBody: true,
_trailer: '',
finished: true,
_headerSent: true,
socket: [Object],
connection: [Object],
_header: 'GET /lookup?bundleId=com.miniclip.agar.io&country=us&entity=software HTTP/1.1\r\nhost: itunes.apple.com\r\nConnection: close\r\n\r\n',
_onPendingData: [Function: noopPendingOutput],
agent: [Object],
socketPath: undefined,
timeout: undefined,
method: 'GET',
path: '/lookup?bundleId=com.miniclip.agar.io&country=us&entity=software',
_ended: true,
res: [Circular],
aborted: undefined,
timeoutCb: null,
upgradeOrConnect: false,
parser: null,
maxHeadersCount: null,
[Symbol(outHeadersKey)]: [Object] },
request:
Request {
domain: null,
_events: [Object],
_eventsCount: 5,
_maxListeners: undefined,
headers: {},
method: 'GET',
callback: [Function],
readable: true,
writable: true,
explicitMethod: true,
_qs: [Object],
_auth: [Object],
_oauth: [Object],
_multipart: [Object],
_redirect: [Object],
_tunnel: [Object],
setHeader: [Function],
hasHeader: [Function],
getHeader: [Function],
removeHeader: [Function],
localAddress: undefined,
pool: {},
dests: [],
__isRequestRequest: true,
_callback: [Function],
uri: [Object],
proxy: null,
tunnel: true,
setHost: true,
originalCookieHeader: undefined,
_disableCookies: true,
_jar: undefined,
port: 443,
host: 'itunes.apple.com',
path: '/lookup?bundleId=com.miniclip.agar.io&country=us&entity=software',
httpModule: [Object],
agentClass: [Object],
agent: [Object],
_started: true,
href: 'https://itunes.apple.com/lookup?bundleId=com.miniclip.agar.io&country=us&entity=software',
req: [Object],
ntick: true,
response: [Circular],
originalHost: 'itunes.apple.com',
originalHostHeaderName: 'host',
responseContent: [Circular],
_destdata: true,
_ended: true,
_callbackCalled: true },
toJSON: [Function: responseToJSON],
caseless: Caseless { dict: [Object] },
read: [Function],
body: 'Resource not found: GET /WebObjects/MZStoreServices.woa/ws/wsLookup?bundleId=com.miniclip.agar.io&country=us&entity=software&urlDesc=\n' } }
@TheArmKing is API responding to your query always with 404? My experience is that if fails quite randomly so I was thinking about add max_attempts=1 parameter to be able to do it multiple times if 404 is encountered (so some timing will be required as well).
@TheArmKing is API responding to your query always with 404? My experience is that if fails quite randomly so I was thinking about add max_attempts=1 parameter to be able to do it multiple times if 404 is encountered (so some timing will be required as well).
Oops, seems like I forgot to mention its quite randomly returning 404 messages. I have a loop going through the same app list, and each time there are different apps that return 404. (Around 10 to 12 out of 140 returned 404). What is the default value of max_attempts? I think it would be better to try a run a request 2 more times if it returns 404, because in some cases the app is deleted and the 404 is actually true.
@TheArmKing I have very similar issue, it is independent of the specific API call (hopefully, it means that we will have proper API one day :)). The idea with max_attempts (with default value set to 1) is mine and it should be able to fix these kind of bugs. @facundoolano what do you think about such additional parameter?
I was able to emulate something like 'max_attempts', retrying search for an app thrice with 5 second gaps (so as to not trigger the API Rate Limit). It seems to be working well, the error rate has reduced to < 1%
@TheArmKing are you going to write a PR or should I? imho the right place might be doRequest() function so it is common functionality for all interaction but I can take a deeper look at it (hopefully) in the night.
I won't be making a PR, I didn't modify any files of the library, I just made some changes in my bot's js file.
#144 has many merge conflicts and there is no reply here from the maintainer. What is the status? Will this get merged? It's been more than a year ...
