amazon-ynab-sync TimeoutError after 30-50 seconds

I used to get this occasionally, but it would work properly when I tried again, so I assumed it was just amazon servers being slow.

Now it errors out every time (maybe 10 tries), and in less than 60 seconds (usually about 35), so I'm thinking it's a bug.

Or maybe there's just a 30 second timeout that needs to be higher with current server load? [ ETA: YES. See comments two weeks later. Don't bother reading more of this entry. ]

DEBUG=* shows:

------- lots of bursts of puppeteer debugging info cut --------
...
  puppeteer:protocol:RECV ◀ {"method":"Network.dataReceived","params":{"requestId":"85537.975","timestamp":197556.927511,"dataLength":89,"encodedDataLength":0},"sessionId":"65696E4FF54A1FE685C48DB4"} +0ms
  puppeteer:protocol:RECV ◀ {"method":"Network.dataReceived","params":{"requestId":"85537.975","timestamp":197556.928222,"dataLength":0,"encodedDataLength":131},"sessionId":"65696E4FF54A1FE685C48DB4"} +1ms
  puppeteer:protocol:RECV ◀ {"method":"Network.loadingFinished","params":{"requestId":"85537.975","timestamp":197556.925519,"encodedDataLength":446,"shouldReportCorbBlocking":false},"sessionId":"65696E4FF54A1FE685C48DB4"} +0ms
------- this is the last pause before termination, for 1-2 seconds --------
  puppeteer:protocol:SEND ► {"method":"Browser.close","id":301} +28s
  puppeteer:protocol:RECV ◀ {"id":301,"result":{}} +2s
  puppeteer:protocol:RECV ◀ {"method":"Target.detachedFromTarget","params":{"sessionId":"65696E4FF54A1FE685C48DB4","targetId":"A5D2F3249B49396AA7921E4D"}} +0ms
  puppeteer-extra-plugin:user-data-dir onClose +49s
  puppeteer-extra-plugin:user-data-dir removeUserDataDir +0ms
(node:85487) [DEP0147] DeprecationWarning: In future versions of Node.js, fs.rmdir(path, { recursive: true }) will be removed. Use fs.rm(path, { recursive: true }) instead
(Use `node --trace-deprecation ...` to show where the warning was created)
  puppeteer-extra-plugin:user-data-dir onClose +13ms
  puppeteer-extra-plugin:user-data-dir removeUserDataDir +0ms
  puppeteer-extra-plugin:user-data-dir Error: ENOENT: no such file or directory, stat '/tmp/puppeteer_dev_profile-kFpxUA'
  puppeteer-extra-plugin:user-data-dir     at Object.statSync (node:fs:1526:3)
  puppeteer-extra-plugin:user-data-dir     at __node_internal_ (node:internal/fs/utils:767:8)
  puppeteer-extra-plugin:user-data-dir     at Object.rmdirSync (node:fs:1146:15)
  puppeteer-extra-plugin:user-data-dir     at Plugin.deleteUserDataDir (/home/sandro/.npm-global/lib/node_modules/amazon-ynab-sync/node_modules/puppeteer-extra-plugin-user-data-dir/index.js:72:11)
  puppeteer-extra-plugin:user-data-dir     at Plugin.onClose (/home/sandro/.npm-global/lib/node_modules/amazon-ynab-sync/node_modules/puppeteer-extra-plugin-user-data-dir/index.js:114:12)
  puppeteer-extra-plugin:user-data-dir     at process.emit (node:events:406:35)
  puppeteer-extra-plugin:user-data-dir     at process._fatalException (node:internal/process/execution:177:19)
  puppeteer-extra-plugin:user-data-dir     at processPromiseRejections (node:internal/process/promises:246:11)
  puppeteer-extra-plugin:user-data-dir     at processTicksAndRejections (node:internal/process/task_queues:97:32) +0ms
/home/sandro/.npm-global/lib/node_modules/amazon-ynab-sync/node_modules/puppeteer/lib/cjs/puppeteer/common/helper.js:120
            rejectCallback(new Errors_js_1.TimeoutError('Timeout exceeded while waiting for event'));
                           ^

TimeoutError: Timeout exceeded while waiting for event
    at Timeout.<anonymous> (/home/sandro/.npm-global/lib/node_modules/amazon-ynab-sync/node_modules/puppeteer/lib/cjs/puppeteer/common/helper.js:120:28)
    at listOnTimeout (node:internal/timers:557:17)
    at processTimers (node:internal/timers:500:7)
Command exited with non-zero status 1
3.45user 0.98system 0:49.33elapsed 8%CPU (0avgtext+0avgdata 108328maxresident)k

Aug 24 '21 14:08 sandhawke

FWIW trying it again a week later, it worked fine.

Aug 30 '21 15:08 sandhawke

Another week later it's broken again.

Looks like it's just Puppeteer defaults to a 30 second timeout and Amazon's systems can be slower than that, maybe because I have a lot of transactions on my account or something.

Fixed with calling page.setDefaultTimeout and page.setDefaultNavigationTimeout with 5 * 60 * 1000 in _getPage in amazon-order-reports-api. I've never used TS, so I just patched the JS for now.

Sep 06 '21 14:09 sandhawke