Bug: "another git process seems to be running..." in sync mode

[edbzn]

I'm still trying to reproduce the issue myself, since it pops up occasionally in my CI even with it inline.

Right now I'm having doubts as to whether or not "parallel" changes the behavior at all, or with what runners does it change the behavior, or if it only affects certain commands, such as nx run-many.

One possible guess is here:

https://github.com/jscutlery/semver/blob/2d72467f542dc026241114bf3a36e54b234010d5/packages/semver/src/executors/version/version.ts#L86

There might be a race condition with performing concurrent addToStage() since we're using concatMap, so each project is still being updated in parallel. It's hard to reproduce though, but if the changes at addToStage were performed sequentially it might help in my case.

In your case, my guess is that the addToStage() from updating the changeLogs and updating the packageJson hit at the same time, since those are also done in parallel:

https://github.com/jscutlery/semver/blob/2d72467f542dc026241114bf3a36e54b234010d5/packages/semver/src/executors/version/version.ts#L54

Originally posted by @dereekb in https://github.com/jscutlery/semver/issues/500#issuecomment-1129200958

[dereekb]

I created a quick pull request that merges together all the paths then stages them one time instead of in parallel.

#516

[dereekb]

This bug is preventing me from running a release right now actually, so I'm working on testing these changes with this branch. I've run it like 7-8 times and it fails with this issue each time.

https://app.circleci.com/pipelines/github/dereekb/dbx-components/552/workflows/99174b61-6122-46b6-9f3e-1bbb694eacba/jobs/1203

Of course when I run locally the current branch seems to not choke up, so working on running a release on CircleCI with the fix and see if that helps.

Edit:

Seems to have worked:

https://app.circleci.com/pipelines/github/dereekb/dbx-components/556/workflows/e61cd5cf-55f6-4815-bfd9-ccf7022067c8/jobs/1214

[DanielNetzer]

I would like to upvote this as it seems that the process is running faster then the git CLI is actually able to release the .lock file and causing errors similar to this:

[core] 🟠 No previous version tag found, fallback to version 0.0.0.
        New version will be calculated based on all changes since first commit.
        If your project is already versioned, please tag the latest release commit with core-x.y.z and run this command again.
[core] 🆕 Calculated new version "1.0.0".
[core] 📜 Generated CHANGELOG.md.
[core] ❌ Error: fatal: Unable to create '/home/ubuntu/myagent/_work/152/s/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.

[edbzn]

@DanielNetzer it should be fixed in 2.24.1

[DanielNetzer]

@edbzn Still experiencing it locally and in CI

[dereekb]

Is that with a new project? There may be another race condition somewhere for the first setup.

[DanielNetzer]

@dereekb fairly new project, using the latest version of semver. I even wen further with that and did a fresh install of all modules, cleared all releases and git tags.

[DanielNetzer]

@dereekb @edbzn Any chance to add to the README a contribution/running the project locally guide? I want to help out as well, the idea and concept are great but there seems to be lots of issues.

[edbzn]

I don't understand why we don't catch anything in the e2e tests. I would need a reproduction to investigate. @DanielNetzer There's a contributing guide here, let me know if I can help.

[dereekb]

Your issue sounds the same as the other person's here.

https://github.com/jscutlery/semver/issues/500#issuecomment-1126896815

It's a tricky bug to reproduce, although I guess we know it is still happening for new projects that might not have that big of size. Wasn't occurring on my machine at home, but was occurring sometimes on CircleCI, until I made a large enough release that it triggered every time. The race condition occurs on the filesystem.

I haven't checked the e2e tests so I can't speak to that, but if it isn't invoking git on the CI machine directly then it won't catch it.

[DanielNetzer]

I assume the easiest way to reproduce it as add the releaseAs=<SOME_SEMVER> flag to activate full release for all the packages. going to start and work on this now. btw, I believe that removing all RxJS in favor or promises will resolve this race condition issue...

[nradulaski-godaddy]

Did using Promises fix the issue?

[nradulaski-godaddy]

This issue happens all the time with affected and run-many unless I use --parallel=1.

[edbzn]

@nradulaski-godaddy thanks for the feedback, I think it means the underlying issue is fixed in semver, but we have to document the usage of --parallel=1 or handle it in the install generator so it's transparent to the user.

[nradulaski-godaddy]

I can't seem to find parallel option in this plugin and setting it for all tasks in nx.json would be bad since we do want no run our build, tests, linting, etc in parallel. I don't know if there is a way to have parallel=1 only for semver?

[yp717]

This is still happening and its not clear that this race condition exists unless you have to force it to run serially. Our pipeline hangs when we run everything serially so this doesnt fix the issue on our end

[sunglam]

I would like to upvote this as it seems that the process is running faster then the git CLI is actually able to release the .lock file and causing errors similar to this:

[core] 🟠 No previous version tag found, fallback to version 0.0.0.
        New version will be calculated based on all changes since first commit.
        If your project is already versioned, please tag the latest release commit with core-x.y.z and run this command again.
[core] 🆕 Calculated new version "1.0.0".
[core] 📜 Generated CHANGELOG.md.
[core] ❌ Error: fatal: Unable to create '/home/ubuntu/myagent/_work/152/s/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.

I am still facing this issue "@jscutlery/semver": "^4.1.0", MY COMMAND npx nx affected --target version --parallel=1 --base=${BRANCH} --head=${TEAMCITY_BRANCH} --push=true --baseBranch=${TEAMCITY_BRANCH} --commitMessageFormat="fix: release {projectName} version {version}" --noVerify=true --exclude=test-ui || exit $?

[sunglam]

This issue happens all the time with affected and run-many unless I use --parallel=1.

I am still facing this issue "@jscutlery/semver": "^4.1.0", using --parallel=1 as well MY COMMAND npx nx affected --target version --parallel=1 --base=${BRANCH} --head=${TEAMCITY_BRANCH} --push=true --baseBranch=${TEAMCITY_BRANCH} --commitMessageFormat="fix: release {projectName} version {version}" --noVerify=true --exclude=test-ui || exit $?