git-xargs
git-xargs copied to clipboard
gruntwork-io
Does git-xargs support only 10 PRs?
Describe the bug Using git-xargs with 10+ repositories, I end up with only 10 PRs (consistently between runs, I tried running the command multiple times)
To Reproduce Steps to reproduce the behavior including the relevant Terraform/Terragrunt/Packer version number and any code snippets and module inputs you used. Running the git-xargs command with default settings produces only 10 PRs. For example I ran the following:
git-xargs \
--repos $PWD/tools/git-xargs/dbt_v1/repos \
--branch-name branch-name-here \
--loglevel DEBUG \
--commit-message "My Commit Message" \
--pull-request-description "DESCRIPTION HERE" \
--draft \
--dry-run \
-- $PWD/tools/git-xargs/dbt_v1/script.sh
The output looks like this:
REPOS AGAINST WHICH PULL REQUESTS FAILED TO BE OPENED │──────────────────────────────│────────────────────────────────────────────────────────────│ │ REPO NAME │ REPO URL │ │──────────────────────────────│────────────────────────────────────────────────────────────│ │ my-11th-repo │ my-11th-repo-url │ │──────────────────────────────│────────────────────────────────────────────────────────────│
REPOS WHOSE SPECIFIED BRANCHES DID NOT EXIST ON THE REMOTE, AND SO WERE FIRST CREATED LOCALLY │──────────────────────────────────────│────────────────────────────────────────────────────────────────────│ │ REPO NAME (11) │ REPO URL │ │──────────────────────────────────────│────────────────────────────────────────────────────────────────────│ ...I omitted content here... │──────────────────────────────────────│────────────────────────────────────────────────────────────────────│
PULL REQUESTS OPENED
│──────────────────────────────────────│────────────────────────────────────────────────────────────────────────────│ │ REPO NAME (10) │ PR URL │ │──────────────────────────────────────│────────────────────────────────────────────────────────────────────────────│ ...I omitted content here... │──────────────────────────────────────│────────────────────────────────────────────────────────────────────────────│
// paste code snippets here
Expected behavior I'm expecting 10+ PRs to be created, however only 10 were created. I can see that on the 11th repository a branch was created with the right content, however no PR.
Can it be related to the tool itself or a policy I have in my organization?
Nice to have
- [ ] Terminal output
- [ ] Screenshots
Additional context Add any other context about the problem here.
THANKS!!! Eyal
Probably because of rate limiting see #59 and open PR #67 which seems to fix this
git-xargs does support opening more than 10 prs concurrently. As @Flydiverny points out, you're likely hitting rate limits, depending on what you're trying to run and how, and we've got a fix in place I hope to land shortly.
You could try cloning and building from #67 and see if it addresses your issue, but in any event, we won't know for sure what you're encountering without the actual errors you're hitting.
Can you find the actual errors in your logs since you're running with log level debug? They would help us determine what's actually going on.
From what I remember, I didn't see any ERROR in the log. However, I'm going to run a similar command in the next few days, so:
- First, I can try having 11 repositories using the old command, and share the output here.
- Then, I can use your version (including #67) and check if the issue is fixed.
Thanks for being so responsive. I'll keep you updated.
Excellent. Your feedback on the pr branch would be very valuable. Let me know what you find.
Hi Zack, I used your version and I got the same issue.
I attach the following debug message (I couldn't include the full log, cause it includes some repo and the org name, so I tried to just include any log entry that repo without PR (let's call it boo) was mentioned in...). I hope you will find it helpful.
In the meantime, my plan to move forward with making 10 changes at a time to avoid this limitation.
The interesting errors are:
[git-xargs] INFO[2022-05-16T12:48:18-06:00] Retrying PR for repo: boo again later with 60 second delay due to secondary rate limiting.
...
repos-file
[git-xargs] DEBU[2022-05-16T12:48:18-06:00] pullRequestWorker received pull request job. Delay: 60 seconds. Retries: 1 for repo: boo on branch: refs/heads/add-some-env
More details below:
...
[git-xargs] DEBU[2022-05-16T12:48:05-06:00] Looking up filename provided repo Name=boo Organization=my-org
[git-xargs] DEBU[2022-05-16T12:48:05-06:00] Successfully fetched repo Name=boo Organization=my-org
...
[git-xargs] DEBU[2022-05-16T12:48:05-06:00] Repo will have all targeted scripts run against it Repository=boo
...
[git-xargs] DEBU[2022-05-16T12:48:05-06:00] Attempting to clone repository using GITHUB_OAUTH_TOKEN Repo=boo
...
Total 17710 (delta 1067), reused 1376 (delta 931), pack-reused 16141 Repo=boo
...
[git-xargs] DEBU[2022-05-16T12:48:08-06:00] Created branch Branch Name=refs/heads/add-some-env Repo=boo
...
[git-xargs] DEBU[2022-05-16T12:48:08-06:00] pullRequestWorker received pull request job. Delay: 0 seconds. Retries: 0 for repo: csp-cohero-reg-config on branch: refs/heads/add-some-env
[git-xargs] DEBU[2022-05-16T12:48:08-06:00] Repo=boo
...
[git-xargs] DEBU[2022-05-16T12:48:08-06:00] Executing command against local clone of repo... Command="[/Users/peleyal/github/my-repo-that-contains-the-script/tools/git-xargs/foo/script.py]" Directory=/var/folders/yt/tclcbvkn1dl92w37kfn7v_v8005lgd/T/git-xargs-boo380861117 Repo=boo
...
[git-xargs] DEBU[2022-05-16T12:48:08-06:00] Output of command [/Users/peleyal/github/my-repo-that-contains-the-script/tools/git-xargs/foo/script.py] for repo boo in directory /var/folders/yt/tclcbvkn1dl92w37kfn7v_v8005lgd/T/git-xargs-boo380861117:
...
[git-xargs] DEBU[2022-05-16T12:48:08-06:00] Local repository worktree no longer clean, will stage and add new files and commit changes Repo=boo
...
[git-xargs] DEBU[2022-05-16T12:48:10-06:00] Successfully pushed local branch to remote origin Repo=boo
[git-xargs] INFO[2022-05-16T12:48:10-06:00] Repository successfully processed Repo name=boo
[git-xargs] DEBU[2022-05-16T12:48:10-06:00] pullRequestWorker received pull request job. Delay: 0 seconds. Retries: 0 for repo: boo on branch: refs/heads/add-some-env
...
[git-xargs] INFO[2022-05-16T12:48:18-06:00] Retrying PR for repo: boo again later with 60 second delay due to secondary rate limiting.
...
REPO SELECTION METHOD USED FOR THIS RUN - (see README.md for more information)
...
*****************************************************************
GIT-XARGS RUN SUMMARY @ 2022-05-16 18:48:18.619488 +0000 UTC
Runtime in seconds: 15
*****************************************************************
...
repos-file
[git-xargs] DEBU[2022-05-16T12:48:18-06:00] pullRequestWorker received pull request job. Delay: 60 seconds. Retries: 1 for repo: boo on branch: refs/heads/add-some-env
REPOS SUPPLIED VIA --repos FILE FLAG
│─────────────────────────│────────────────────────────│
│ ORGANIZATION NAME (12) │ URL │
│─────────────────────────│────────────────────────────│
< OTHER REPOS HERE >
│ boo │ my-org/boo |
│─────────────────────────│────────────────────────────│
ALL REPOS THAT WERE TARGETED FOR PROCESSING AFTER FILTERING MISSING / MALFORMED REPOS
│────────────────────────────│──────────────────────────────────────────────────────────│
│ REPO NAME (12) │ REPO URL │
│────────────────────────────│──────────────────────────────────────────────────────────│
< OTHER REPOS HERE >
│ boo │ https://github.com/my-org/boo |
│────────────────────────────│──────────────────────────────────────────────────────────│
REPOS THAT WERE SUCCESSFULLY CLONED TO THE LOCAL FILESYSTEM
│────────────────────────────│──────────────────────────────────────────────────────────│
│ REPO NAME (12) │ REPO URL │
│────────────────────────────│──────────────────────────────────────────────────────────│
< OTHER REPOS HERE >
│ boo │ https://github.com/my-org/boo |
│────────────────────────────│──────────────────────────────────────────────────────────│
REPOS THAT SHOWED FILE CHANGES TO THEIR WORKING DIRECTORY FOLLOWING COMMAND EXECUTION
│────────────────────────────│──────────────────────────────────────────────────────────│
│ REPO NAME (11) │ REPO URL │
│────────────────────────────│──────────────────────────────────────────────────────────│
< OTHER REPOS HERE >
│ boo │ https://github.com/my-org/boo |
│────────────────────────────│──────────────────────────────────────────────────────────│
REPOS THAT SHOWED NO FILE CHANGES TO THEIR WORKING DIRECTORY FOLLOWING COMMAND EXECUTION
│────────────────────│──────────────────────────────────────────────────│
│ REPO NAME │ REPO URL │
│────────────────────│──────────────────────────────────────────────────│
< OTHER REPOS HERE >
│────────────────────│──────────────────────────────────────────────────│
REPOS WHOSE SPECIFIED BRANCHES DID NOT EXIST ON THE REMOTE, AND SO WERE FIRST CREATED LOCALLY
│────────────────────────────│──────────────────────────────────────────────────────────│
│ REPO NAME (12) │ REPO URL │
│────────────────────────────│──────────────────────────────────────────────────────────│
< OTHER REPOS HERE >
│────────────────────────────│──────────────────────────────────────────────────────────│
REPOS WHOSE INITIAL PULL REQUEST FAILED TO BE CREATED DUE TO GITHUB RATE LIMITS
│─────────────────│───────────────────────────────────────────────│
│ REPO NAME │ REPO URL │
│─────────────────│───────────────────────────────────────────────│
│ boo │ https://github.com/my-org/boo │
│─────────────────│───────────────────────────────────────────────│
*****************************************************
DRAFT PULL REQUESTS OPENED
*****************************************************
│────────────────────────────│──────────────────────────────────────────────────────────────────│
│ REPO NAME (10) │ PR URL │
│────────────────────────────│──────────────────────────────────────────────────────────────────│
< OTHER REPOS HERE>
│────────────────────────────│──────────────────────────────────────────────────────────────────│
Hi Eyal,
Thanks for the update! Sorry you're still encountering issues.
It's a little difficult to determine for sure what's happening without the full debug log output. I understand that your logs may contain repo and org names that you don't wish to publish.
However, from the last output you shared, I can see that you did encounter secondary rate limiting when running your git-xargs job. That could mean you're attempting to open pull requests more quickly than GitHub's API will tolerate. The changes in https://github.com/gruntwork-io/git-xargs/pull/67 will address this issue directly - and it provides some sane defaults, plus the ability to fine-tune your git-xargs run's behavior when encountering rate limits.
For example, you can increase the delay between opening pull requests, plus increase the amount of extra time to wait whenever git-xargs detects it has been rate-limited. As mentioned in the updated docs in https://github.com/gruntwork-io/git-xargs/pull/67 - you can tweak and experiment with these new flags and various values until your job is no longer being rate-limited.
Here's a sample command I've run on the rate-limiting branch successfully:
./git-xargs \
--loglevel debug \
--github-org zack-test-org \
--branch-name tt-2 \
--seconds-between-prs 2 \
# This is a dummy command - but it could be a script or any other command
touch wakka.txt
In my case, I asked git-xargs to run against a GitHub organization with 35+ repositories. Many of those repos are intentionally empty - so git-xargs ignores them, but ultimately, 14 repos were in the correct state (not empty, didn't have a conflicting file on that branch, etc) for pull requests to be opened, and git-xargs handled the rate limiting that GitHub applied, by waiting 2 seconds between PRs and retrying PRs later that failed the first time due to rate limiting.
I'm relaying these values to you just by way of example and so that it can hopefully help you figure out what exactly you're encountering on your end.
It could also be the case that you're encountering an error case unique to your script's logic and/or unique to the state of your target repositories (for example, if one were missing an expected file).
Would you be willing to try re-producing your issue on a test or public GitHub org with test repos and a simple script that perhaps adds a file to each repo? That would also help to narrow down if the issue is specific to git-xargs or some combination of your script logic and repo state.
Thanks!
I'm going to close this ticket out due to inactivity and because the underlying issue should have been resolved by recent changes.
Zack, do you think that this issue was already resolved? Can you link to the relevant PRs?
Thanks!