git-plugin icon indicating copy to clipboard operation
git-plugin copied to clipboard

[JENKINS-75526] Large repository clone on controller crash

Open jenkins-infra-bot opened this issue 9 months ago • 2 comments

When doing large (4,7GB) repository clone on the controller side we get the exception visible on the code pane. The clone on the controller side is needed for the Pipeline job type (Jenkins file). The issue is only happening on the controller side. Agents clones are working without issues on other job types (e.g. Frestyle). 

We use reference repository feature, and do not expect that we are facing any timeout related issue here. 

The error does not occur once we limit refspec to one or few branches. We have lots of branches (little over 2k).  Strangely, when doing the same operations manually on the server (same server, same user account), git is not returning any errors and the `git fetch` operation is successful. 

Checked the release notes of the plugin, but I have not found anything related to the issue described. 

The recommended git tool is: git.exeusing credential ****Wiping out workspace first.Cloning the remote Git repositoryAvoid fetching tagsCloning repository  > git.exe init C:\Sources\Job@​script\8ea19a52dbc789f10f7cf0b663b949ce2b7f4032ecfa961ea88f6bdda9dbb89f # timeout=10Using reference repository: C:\reference-sources\repo.gitFetching upstream changes from  > git.exe --version # timeout=10 > git --version # 'git version 2.49.0.windows.1'using GIT_ASKPASS to set credentials  > git.exe fetch --no-tags --force --progress --  +refs/heads/*:refs/remotes/origin/* # timeout=30ERROR: Error cloning remote repo 'origin'hudson.plugins.git.GitException: Command "git.exe fetch --no-tags --force --progress --  +refs/heads/*:refs/remotes/origin/*" returned status code 1:stdout: stderr: remote: Azure Repos        remote: remote: Found 757 objects to send. (157 ms)        Receiving objects:   0% (1/757)Receiving objects:   1% (8/757)Receiving objects:   2% (16/757)Receiving objects:   3% (23/757)Receiving objects:   4% (31/757)
*** REDACTED FOR BREVITY ***
 * [new branch]              ver/24.1.1              -> origin/ver/24.1.1 * [new branch]              ver/25.1.1              -> origin/ver/25.1.1 * [new branch]              ver/25.2                -> origin/ver/25.2
  at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2846)  at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2185)  at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:635)  at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:871)  at PluginClassLoader for git//hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1224)  at PluginClassLoader for git//hudson.plugins.git.GitSCM.checkout(GitSCM.java:1302)  at PluginClassLoader for workflow-scm-step//org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:136)  at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:167)  at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:73)  at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:319)  at hudson.model.ResourceController.execute(ResourceController.java:101)  at hudson.model.Executor.run(Executor.java:445)ERROR: Error cloning remote repo 'origin'ERROR: Maximum checkout retry attempts reached, abortingERROR: Error cloning remote repo 'origin'ERROR: Maximum checkout retry attempts reached, abortingFinished: FAILURE 

Originally reported by j_k_webc, imported from: Large repository clone on controller crash
  • status: Open
  • priority: Major
  • component(s): git-plugin
  • label(s): Git-plugin, git, plugin
  • resolution: Unresolved
  • votes: 0
  • watchers: 2
  • imported: 2025-12-02
Raw content of original issue

When doing large (4,7GB) repository clone on the controller side we get the exception visible on the code pane. The clone on the controller side is needed for the Pipeline job type (Jenkins file). The issue is only happening on the controller side. Agents clones are working without issues on other job types (e.g. Frestyle). 

We use reference repository feature, and do not expect that we are facing any timeout related issue here. 

The error does not occur once we limit refspec to one or few branches. We have lots of branches (little over 2k).  Strangely, when doing the same operations manually on the server (same server, same user account), git is not returning any errors and the `git fetch` operation is successful. 

Checked the release notes of the plugin, but I have not found anything related to the issue described. 

The recommended git tool is: git.exeusing credential ****Wiping out workspace first.Cloning the remote Git repositoryAvoid fetching tagsCloning repository <REPOURL> > git.exe init C:\Sources\Job@script\8ea19a52dbc789f10f7cf0b663b949ce2b7f4032ecfa961ea88f6bdda9dbb89f # timeout=10Using reference repository: C:\reference-sources\repo.gitFetching upstream changes from <REPOURL> > git.exe --version # timeout=10 > git --version # 'git version 2.49.0.windows.1'using GIT_ASKPASS to set credentials  > git.exe fetch --no-tags --force --progress -- <REPOURL> +refs/heads/*:refs/remotes/origin/* # timeout=30ERROR: Error cloning remote repo 'origin'hudson.plugins.git.GitException: Command "git.exe fetch --no-tags --force --progress -- <REPOURL> +refs/heads/*:refs/remotes/origin/*" returned status code 1:stdout: stderr: remote: Azure Repos        remote: remote: Found 757 objects to send. (157 ms)        Receiving objects:   0% (1/757)Receiving objects:   1% (8/757)Receiving objects:   2% (16/757)Receiving objects:   3% (23/757)Receiving objects:   4% (31/757)
*** REDACTED FOR BREVITY ***
 * [new branch]              ver/24.1.1              -> origin/ver/24.1.1 * [new branch]              ver/25.1.1              -> origin/ver/25.1.1 * [new branch]              ver/25.2                -> origin/ver/25.2
  at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2846)  at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2185)  at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:635)  at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:871)  at PluginClassLoader for git//hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1224)  at PluginClassLoader for git//hudson.plugins.git.GitSCM.checkout(GitSCM.java:1302)  at PluginClassLoader for workflow-scm-step//org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:136)  at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:167)  at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:73)  at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:319)  at hudson.model.ResourceController.execute(ResourceController.java:101)  at hudson.model.Executor.run(Executor.java:445)ERROR: Error cloning remote repo 'origin'ERROR: Maximum checkout retry attempts reached, abortingERROR: Error cloning remote repo 'origin'ERROR: Maximum checkout retry attempts reached, abortingFinished: FAILURE 
environment
Controller:<br/>
Windows Server 2019 (1809) <br/>
Java: 17.0.14+7-LTS, Microsoft-10800295<br/>
Jenkins Version 2.479.3<br/>
Git Plugin: 4.10.3<br/>
Git client plugin: 4.7.0<br/>
git version 2.49.0.windows.1<br/>

jenkins-infra-bot avatar Apr 08 '25 08:04 jenkins-infra-bot

markewaite:
  • Original comment link
  • Raw content of original comment:

    We use reference repository feature, and do not expect that we are facing any timeout related issue here.

    When command line git reports many lines of "receiving objects", that usually indicates that it is copying data from the remote repository to the local repository. The reference repository may be outdated or not being used. Check the size of the .git/objects directory to confirm that the reference repository is being used.

    The git plugin uses two command line git operations, fetch and checkout, while git clone inside command line git may be a single operation with some optimizations that are not available with the pair of operations. I don't expect to ever change the use of fetch and checkout because of the compatibility risks involved in that change.

    I don't see anything that I can do as git plugin maintainer to resolve this. You'll need to either increase your local timeout settings or narrow the refspecs or use shallow clone.

We use reference repository feature, and do not expect that we are facing any timeout related issue here.

When command line git reports many lines of "receiving objects", that usually indicates that it is copying data from the remote repository to the local repository. The reference repository may be outdated or not being used. Check the size of the .git/objects directory to confirm that the reference repository is being used.

The git plugin uses two command line git operations, fetch and checkout, while git clone inside command line git may be a single operation with some optimizations that are not available with the pair of operations. I don't expect to ever change the use of fetch and checkout because of the compatibility risks involved in that change.

I don't see anything that I can do as git plugin maintainer to resolve this. You'll need to either increase your local timeout settings or narrow the refspecs or use shallow clone.

jenkins-infra-bot avatar Apr 08 '25 13:04 jenkins-infra-bot

j_k_webc:
  • Original comment link
  • Raw content of original comment:

    We have checked the .git/objects and the reference repository is for sure used (1MB used, as opposed to 4,7GB full repo). The fetch operation returns almost instantaneously when done on by operator on the controller node,  similarly when executed by Jenkins plugin. However, for some reason git plugin for fetch operation is returning an error, whereas when done "by hand" no error occurs. That is why we issued the issue for the git plugin. We do not expect to change the base operations that git plugin is using (init, fetch etc.). 

We have checked the .git/objects and the reference repository is for sure used (1MB used, as opposed to 4,7GB full repo). The fetch operation returns almost instantaneously when done on by operator on the controller node,  similarly when executed by Jenkins plugin. However, for some reason git plugin for fetch operation is returning an error, whereas when done "by hand" no error occurs. That is why we issued the issue for the git plugin. We do not expect to change the base operations that git plugin is using (init, fetch etc.). 

jenkins-infra-bot avatar Apr 09 '25 08:04 jenkins-infra-bot