git-xargs icon indicating copy to clipboard operation
git-xargs copied to clipboard

Published 20 hours ago • verified publisher icon gruntwork-io

Rate limit github clone calls

Open cbrand opened this issue 1 year ago • 3 comments

Description

Fixes #139.

Add rate limiting to github clone calls, avoiding issues with github "permission denied" and EOF errors when cloning too many repositories in parallel.

TODOs

Read the Gruntwork contribution guidelines.

  • [x] Update the docs.
  • [x] Run the relevant tests successfully, including pre-commit checks.
  • [x] Ensure any 3rd party code adheres with our license policy or delete this line if its not applicable.
  • [x] Include release notes. If this PR is backward incompatible, include a migration guide.

Release Notes

Added

  • Rate limiting Github Clone calls to 4 per default, allows being parametrized by the --max-concurrent-clones flag

Migration Guide

None needed

cbrand avatar Jan 08 '24 22:01 cbrand

For the reviewers, I was able to build git-xargs and used this build to clone ~80 repos mostly successfully, while the current release was only cloning 1 repo.

XavierChanth avatar Feb 23 '24 20:02 XavierChanth

@brikis98 @hongil0316 What all is necessary in order to help get these changes merged/published? Asking as we've started to use git-xargs recently and without this feature we'll be very unlikely to proceed in using this tool as we're attempting to use it across ~300+ repositories.

flinn avatar Mar 19 '24 22:03 flinn

We have 2000+ repos to update and I had to use a script to execute git-xargs in batches of 50 repos to go around repo clone rate-limiting, which somewhat defeats the purpose of batching offered by this tool!

I compiled the changes in this PR and used it. It solved our issue! Would be great if it gets merged soon.

yogur avatar Aug 23 '24 11:08 yogur

Sorry for the delay. It seems like some of the tests for repository is failing. Can you make sure those tests pass?

FAIL github.com/gruntwork-io/git-xargs/repository 300.028s

james00012 avatar Nov 02 '24 01:11 james00012

Friendly ping on this PR.

james00012 avatar Nov 11 '24 00:11 james00012

@james03160927 Hey I had a chance to take a look and seem to have fixed the unit tests. Please take another look:

?   	github.com/gruntwork-io/git-xargs/common	[no test files]
?   	github.com/gruntwork-io/git-xargs/config	[no test files]
?   	github.com/gruntwork-io/git-xargs/local	[no test files]
?   	github.com/gruntwork-io/git-xargs/mocks	[no test files]
?   	github.com/gruntwork-io/git-xargs/printer	[no test files]
?   	github.com/gruntwork-io/git-xargs/stats	[no test files]
?   	github.com/gruntwork-io/git-xargs/util	[no test files]
ok  	github.com/gruntwork-io/git-xargs	0.488s
ok  	github.com/gruntwork-io/git-xargs/auth	0.223s
ok  	github.com/gruntwork-io/git-xargs/cmd	1.893s
ok  	github.com/gruntwork-io/git-xargs/io	0.680s
ok  	github.com/gruntwork-io/git-xargs/repository	2.154s
ok  	github.com/gruntwork-io/git-xargs/types	1.158s

cbrand avatar Nov 14 '24 21:11 cbrand