restic
restic copied to clipboard
Speed up MasterIndex.Each
What does this PR change? What problem does it solve?
MasterIndex.Each currently uses channels and along with a goroutine to relay data from Index.Each to a unified channel. This PR replaces the channel with a simple callback which drastically reduces the CPU overhead:
name old time/op new time/op delta
MasterIndexEach-16 6.68s ±24% 0.96s ± 2% -85.64% (p=0.008 n=5+5)
Profiling MasterIndexEach shows that now most time is spent inside index.toPackedBlob which converts the indexEntry to a PackedBlob. It might be possible to make that conversion lazy, although that might just shift some of the costs.
Was the change previously discussed in an issue or on the forum?
The problem showed up in https://github.com/restic/restic/issues/3916 .
Checklist
- [x] I have read the contribution guidelines.
- [x] I have enabled maintainer edits.
- [x] I have added tests for all code changes.
- ~~[ ] I have added documentation for relevant changes (in the manual).~~
- [ ] There's a new file in
changelog/unreleased/that describes the changes for our users (see template). - [x] I have run
gofmton the code in all commits. - [x] All commit messages are formatted in the same style as the other commits in the repo.
- [x] I'm done! This pull request is ready for review.
LGTM.
LGTM. I've rebased the PR and updated the comment.