GitHub commenting with many issues triggers abuse filtering

[ragesoss]

I tried out pronto from my command line, and it crashed in the middle of leaving comments on a GitHub PR:

$ PULL_REQUEST_ID=712 pronto run -f github_pr -c origin/master
/home/sage/.rvm/gems/ruby-2.1.5/gems/octokit-4.3.0/lib/octokit/response/raise_error.rb:16:in `on_complete': POST https://api.github.com/repos/WikiEducationFoundation/WikiEduDashboard/pulls/712/comments: 403 - You have triggered an abuse detection mechanism and have been temporarily blocked from content creation. Please retry your request again later. // See: https://developer.github.com/v3#abuse-rate-limits (Octokit::AbuseDetected)

I think was making comments at a little under one per second, and it made 36 comments before triggering the filter: https://github.com/WikiEducationFoundation/WikiEduDashboard/pull/712

[mur-wtag]

I'm having same issue.

[doomspork]

@ragesoss have you tried Pronto with the consolidated_comments option set? It should significantly reduce the number of messages.

[mikegee]

Also consider using github_pr_review rather than github_pr. That should make all the comments with one HTTP call.

[mmozuras]

@ragesoss @mur-wtag did @doomspork and @mikegee suggestions help? 🙂

[ragesoss]

I haven't tried them out yet, but plan to soon. Will report back once I do. Thanks!

[ragesoss]

I tried out github_pr_review and it failed on a PR with lots of new issues:

Failed to post: POST https://api.github.com/repos/WikiEducationFoundation/WikiEduDashboard/pulls/1418/reviews: 422 - Validation Failed
Error summary:
  was submitted too quickly after a previous comment // See: https://developer.github.com/v3/pulls/reviews/#create-a-pull-request-review
38 Pronto messages posted to GitHub

None of those 38 messages actually appear on Github.

I tried it with PRONTO_CONSOLIDATE_COMMENTS=true (with both github_pr and github_pr_review) and it posts 0 messages.

[drewish]

I've also been hitting this even using the github_pr_review formatter. Is there a comment count that triggers that? Could we stop below that threshold and just indicate that there are more errors remaining?

[drewish]

I reached out to GitHub to see if there's a PR comment limit we could use to stay under the abuse threshold. If they get back to me I'll open up a PR to cap the number of comments.

[drewish]

I guess https://github.com/prontolabs/pronto/pull/304 would help address this too.