github-csv-tools

github-csv-tools copied to clipboard

Hi there! Me again...

So the throttling is better but I was apparently still too liberal. I'm still hitting rate limit issues with the current configuration (my test case is about 78 issues that I am importing, FYI).

Also, I'm checking for items in the queue before exiting the program but if I have requests that lag, it aborts early. So that needs adjusting too.

I will work on it and send you another PR; just wanted to let you know in case you get to it first.

Now using octokit and https://github.com/octokit/plugin-throttling.js

Importing 80 issues. I got an 403 error at number 22 with the following message:

'You have exceeded a secondary rate limit and have been temporarily blocked from content creation. Please retry your request again later.'

Abuse detected for request POST /repos/{owner}/{repo}/issues
Error
{ HttpError: You have exceeded a secondary rate limit and have been temporarily blocked from content creation. Please retry your request again later.
    at fetch.then (/usr/local/lib/node_modules/github-csv-tools/node_modules/@octokit/request/dist-node/index.js:86:21)
    at process._tickCallback (internal/process/next_tick.js:68:7)
  name: 'HttpError',
  status: 403,
  response:
   { url: 'https://api.github.com/repos/convergence-mag/web/issues',
     status: 403,
     headers:
      { 'access-control-allow-origin': '*',
        'access-control-expose-headers':
         'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset',
        connection: 'close',
        'content-encoding': 'gzip',
        'content-security-policy': 'default-src \'none\'',
        'content-type': 'application/json; charset=utf-8',
        date: 'Thu, 28 Oct 2021 23:59:40 GMT',
        'github-authentication-token-expiration': '2021-11-28 00:50:03 UTC',
        'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
        server: 'GitHub.com',
        'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
        'transfer-encoding': 'chunked',
        vary: 'Accept-Encoding, Accept, X-Requested-With',
        'x-accepted-oauth-scopes': '',
        'x-content-type-options': 'nosniff',
        'x-frame-options': 'deny',
        'x-github-media-type': 'github.v3; format=json',
        'x-github-request-id': '9636:05EC:1A87BC:26DCD0:617B396B',
        'x-oauth-scopes': 'repo',
        'x-ratelimit-limit': '5000',
        'x-ratelimit-remaining': '4972',
        'x-ratelimit-reset': '1635465581',
        'x-ratelimit-resource': 'core',
        'x-ratelimit-used': '28',
        'x-xss-protection': '0' },
     data:
      { message:
         'You have exceeded a secondary rate limit and have been temporarily blocked from content creation. Please retry your request again later.',

File: CM-github-issues.csv

Thanks for the report @campaignupgrade. We should probably start by upgrading plugin-throttling.js to the latest version (3.5.2). Could you do that and send a PR and test if that helps? If not we can look into setting some better defaults for the plugin-throttling.js settings.

I worked around the issue by deleting lines from the csv and restarting the import. It took three passes to complete.

I'm not sure if it's related, but none of the milestones in this csv were applied. I'm wondering if a milestones error causes the rate limit overflow.

I don't actually have time to troubleshoot this issue right now but I'll check again on my next project.

Apparently there is a different endpoint for issue creation at https://api.github.com/repos/${GITHUB_USERNAME}/${GIHUB_REPONAME}/import/issues, see this gist. Since it does not send notifications, it is not subject to the secondary rate limit. I found this by asking github support. xref https://github.com/piceaTech/node-gitlab-2-github/issues/28.

Thanks for the input. PRs welcome to switch it to use that other API endpoint.

:tada: This issue has been resolved in version 3.1.4 :tada:

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot :package::rocket: