macos-UnifiedLogs icon indicating copy to clipboard operation
macos-UnifiedLogs copied to clipboard

Introduce an iter_log method to iterate through logs of a same file

Open glumn opened this issue 1 year ago • 2 comments

Responding to the previously created issue #14, here's a first proposition.

We had to test this on our own sysdiags since we do not have yours available, so it'll need more testing on your end, but we noticed an improvement in peak memory usage from ~970mb to ~600mb when parsing the logs.

There's still improvements to be made of course, we introduced an iterator on the surface but it could potentially be iterators all the way down, however the changes required would be substantial.

Here you'll notice that we moved most of the logic from build_log to a new LogIterator structure that itself is used by build_log. We didn't address the fact that all data is first loaded as a UnifiedLogData, which in our opinion constitutes most of the remainder of the memory usage, since we don't know if that is even possible at that point.

Let me know what you think !

glumn avatar Apr 30 '24 10:04 glumn

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

google-cla[bot] avatar Apr 30 '24 10:04 google-cla[bot]

ho wow ! it's hell complicated to sign the CLA !

jrouaix avatar Apr 30 '24 10:04 jrouaix

@puffyCid you think you can merge this PR ?

jrouaix avatar Aug 06 '24 09:08 jrouaix

hi @jrouaix I tested the changes, all the tests passed. However, there are a few conflicts between this PR/branch and main. Could you update the PR to sync with the latest changes? thanks!

puffyCid avatar Aug 06 '24 22:08 puffyCid

updated & ready to merge @puffyCid

jrouaix avatar Aug 07 '24 08:08 jrouaix