git-plugin icon indicating copy to clipboard operation
git-plugin copied to clipboard

[JENKINS-71349] git plugin left a lock file in controller cache folder for pipeline repos, which block all flows

Open jenkins-infra-bot opened this issue 2 years ago • 3 comments

git plugin left a lock file in controller cache folder for pipeline repos, which block all flows.

It happens after we uplift git plugin to 4.12.1.

I think it may related to the feature disable hooks or something else.

we do not enable hooks in controller.

 

It happens three times in one week.

I think sometime it could not release lock file success or it happens that multiple job try to lock same file.

When it happens, all job fails to load pipeline repos until we delete the lock file munally.

Could you please help to check it? Could we ignore this error when fail to change hooks configuration?

 

error log:
ERROR: Checkout failed
org.eclipse.jgit.errors.LockFailedException: Cannot lock /proj/XXX/workspace@​libs/72398dd74922d3156ed234ba0cbb097decd3eeabff521f0f59e1fa740478dc95/.git/config. Ensure that no other process has an open file handle on the lock file /proj/XXX/workspace@​libs/72398dd74922d3156ed234ba0cbb097decd3eeabff521f0f59e1fa740478dc95/.git/config.lock, then you may delete the lock file and retry.
at org.eclipse.jgit.storage.file.FileBasedConfig.save(FileBasedConfig.java:221)
at jenkins.plugins.git.GitHooksConfiguration.disable(GitHooksConfiguration.java:139)
at jenkins.plugins.git.GitHooksConfiguration.lambda$configure$2f1f6402$1(GitHooksConfiguration.java:113)
at org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:29)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:87)
at jenkins.plugins.git.GitHooksConfiguration.configure(GitHooksConfiguration.java:112)
at jenkins.plugins.git.GitHooksConfiguration.configure(GitHooksConfiguration.java:99)
at jenkins.plugins.git.GitHooksConfiguration.configure(GitHooksConfiguration.java:91)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1240)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1312)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)
at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.lambda$doRetrieve$1(SCMSourceRetriever.java:201)
at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrySCMOperation(SCMSourceRetriever.java:148)
at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.doRetrieve(SCMSourceRetriever.java:200)
at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:137)
at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:260)
at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:150)
at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125)
at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1087)
at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:624)
at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:602)
at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:579)
at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:323)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:293)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox$Scope.parse(GroovySandbox.java:163)
at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142)
at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:561)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:513)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:335)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:442)


Originally reported by li445861132, imported from: git plugin left a lock file in controller cache folder for pipeline repos, which block all flows
  • status: Open
  • priority: Major
  • component(s): git-plugin
  • resolution: Unresolved
  • votes: 0
  • watchers: 2
  • imported: 2025-12-02
Raw content of original issue

git plugin left a lock file in controller cache folder for pipeline repos, which block all flows.

It happens after we uplift git plugin to 4.12.1.

I think it may related to the feature disable hooks or something else.

we do not enable hooks in controller.

 

It happens three times in one week.

I think sometime it could not release lock file success or it happens that multiple job try to lock same file.

When it happens, all job fails to load pipeline repos until we delete the lock file munally.

Could you please help to check it? Could we ignore this error when fail to change hooks configuration?

 

error log: ERROR: Checkout failed org.eclipse.jgit.errors.LockFailedException: Cannot lock /proj/XXX/workspace@libs/72398dd74922d3156ed234ba0cbb097decd3eeabff521f0f59e1fa740478dc95/.git/config. Ensure that no other process has an open file handle on the lock file /proj/XXX/workspace@libs/72398dd74922d3156ed234ba0cbb097decd3eeabff521f0f59e1fa740478dc95/.git/config.lock, then you may delete the lock file and retry. at org.eclipse.jgit.storage.file.FileBasedConfig.save(FileBasedConfig.java:221) at jenkins.plugins.git.GitHooksConfiguration.disable(GitHooksConfiguration.java:139) at jenkins.plugins.git.GitHooksConfiguration.lambda$configure$2f1f6402$1(GitHooksConfiguration.java:113) at org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:29) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:87) at jenkins.plugins.git.GitHooksConfiguration.configure(GitHooksConfiguration.java:112) at jenkins.plugins.git.GitHooksConfiguration.configure(GitHooksConfiguration.java:99) at jenkins.plugins.git.GitHooksConfiguration.configure(GitHooksConfiguration.java:91) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1240) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1312) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129) at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.lambda$doRetrieve$1(SCMSourceRetriever.java:201) at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrySCMOperation(SCMSourceRetriever.java:148) at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.doRetrieve(SCMSourceRetriever.java:200) at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:137) at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:260) at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:150) at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125) at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1087) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:624) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:602) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:579) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:323) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:293) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox$Scope.parse(GroovySandbox.java:163) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:561) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:513) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:335) at hudson.model.ResourceController.execute(ResourceController.java:101) at hudson.model.Executor.run(Executor.java:442)

  • environment: Jenkins 2.346.1 with git plugin 4.12.1

jenkins-infra-bot avatar May 26 '23 08:05 jenkins-infra-bot

markewaite:
  • Original comment link
  • Raw content of original comment:

    There are at least two options available to you to reduce the chances of a collision on a Pipeline shared library. They include:

There are at least two options available to you to reduce the chances of a collision on a Pipeline shared library. They include:

jenkins-infra-bot avatar May 26 '23 10:05 jenkins-infra-bot

li445861132:
  • Original comment link
  • Raw content of original comment:

    Thanks markewaite .

    I already allow git hooks on the controller.

    Will try library caching.

     

    And is there any plan to improve git plugin for this error?

     

Thanks markewaite .

I already allow git hooks on the controller.

Will try library caching.

 

And is there any plan to improve git plugin for this error?

 

jenkins-infra-bot avatar May 29 '23 00:05 jenkins-infra-bot

markewaite:
  • Original comment link
  • Raw content of original comment:

    And is there any plan to improve git plugin for this error?

    I don't plan to implement any changes in that area. If you'd like to improve it, you're welcome to propose a pull request that either adds a conditional to avoid saving the configuration if it is not changed or that adds a try/catch to detect the exception and ignore it. The lines of code that need the implementation are in GitHooksConfiguration.

And is there any plan to improve git plugin for this error?

I don't plan to implement any changes in that area. If you'd like to improve it, you're welcome to propose a pull request that either adds a conditional to avoid saving the configuration if it is not changed or that adds a try/catch to detect the exception and ignore it. The lines of code that need the implementation are in GitHooksConfiguration.

jenkins-infra-bot avatar May 29 '23 01:05 jenkins-infra-bot