restic icon indicating copy to clipboard operation
restic copied to clipboard

Speed up MasterIndex.Each

Open MichaelEischer opened this issue 3 years ago • 1 comments

What does this PR change? What problem does it solve?

MasterIndex.Each currently uses channels and along with a goroutine to relay data from Index.Each to a unified channel. This PR replaces the channel with a simple callback which drastically reduces the CPU overhead:

name                old time/op  new time/op  delta
MasterIndexEach-16   6.68s ±24%   0.96s ± 2%  -85.64%  (p=0.008 n=5+5)

Profiling MasterIndexEach shows that now most time is spent inside index.toPackedBlob which converts the indexEntry to a PackedBlob. It might be possible to make that conversion lazy, although that might just shift some of the costs.

Was the change previously discussed in an issue or on the forum?

The problem showed up in https://github.com/restic/restic/issues/3916 .

Checklist

  • [x] I have read the contribution guidelines.
  • [x] I have enabled maintainer edits.
  • [x] I have added tests for all code changes.
  • ~~[ ] I have added documentation for relevant changes (in the manual).~~
  • [ ] There's a new file in changelog/unreleased/ that describes the changes for our users (see template).
  • [x] I have run gofmt on the code in all commits.
  • [x] All commit messages are formatted in the same style as the other commits in the repo.
  • [x] I'm done! This pull request is ready for review.

MichaelEischer avatar Sep 11 '22 09:09 MichaelEischer

LGTM.

greatroar avatar Sep 17 '22 16:09 greatroar

LGTM. I've rebased the PR and updated the comment.

MichaelEischer avatar Sep 24 '22 10:09 MichaelEischer