Concurrency Issues

[pjbgf]

go-git is not thread-safe. In some cases, concurrent reads may lead to repositories becoming corrupted or actions to yield invalid results (e.g. not finding an existing ref).

There is quite a bit of work until we can answer: What operations are concurrency safe and I can call from different Goroutines?

Current status

• [ ] Support read concurrency
  • [x] Map concurrency issues (v6-exp)
    • [x] repository.CommitObjects().Foreach() (v6-exp)
    • [x] repository.Tags().Foreach() (v6-exp)
  • [ ] Create concurrency tests to avoid regression
• [ ] Off-set performance costs for using mutex (or add an opt-in/out flag)
• [ ] Worktree.Add()

Please share in the comments any specific operations you may having issues to use concurrently.

Relates to: https://github.com/go-git/go-git/pull/186 https://github.com/go-git/go-git/issues/48 https://github.com/go-git/go-git/pull/175

[pjbgf]

The loading of object / pack maps (f3311ea24673e5eabb3f7b4ba0d585730548979c) and the index (377731d38d23dc9f7857757271ca90d4679d1c0a) seem to be big culprits of read concurrency. After the changes, I could no longer observe "object not found" when going through all commits and tags concurrently.

[github-actions[bot]]

To help us keep things tidy and focus on the active tasks, we've introduced a stale bot to spot issues/PRs that haven't had any activity in a while.

This particular issue hasn't had any updates or activity in the past 90 days, so it's been labeled as 'stale'. If it remains inactive for the next 30 days, it'll be automatically closed.

We understand everyone's busy, but if this issue is still important to you, please feel free to add a comment or make an update to keep it active.

Thanks for your understanding and cooperation!

[mpldr]

Thanks for the info, at least for my usecase (showing various commits from the Kernel tree), the impact was within what I'd consider to be noise. This could obviously be different for more complex operations, but I do not have the insight into the codebase to properly benchmark this.

[d4x1]

I want to collect commits details with go-git/v5, but it spent too much time, so I want to collect commits concurrently.

@pjbgf Are you still working on it? Let me know if I can do some help.

[pjbgf]

@d4x1 happy for you to take over, if you can, as I have been quite busy of late. Here are three commits (not merged) that I worked on:

https://github.com/go-git/go-git/commit/f3311ea24673e5eabb3f7b4ba0d585730548979c https://github.com/go-git/go-git/commit/377731d38d23dc9f7857757271ca90d4679d1c0a https://github.com/go-git/go-git/commit/0a7b552ae2d70f95e8cad6a8864e8505a496a158

The main blocker on merging any of those changes are performance regression. So we need to benchmark and call out the impact. If the impact is too great, we may need to add a way for users to opt-out/in.