MSBuildStructuredLog icon indicating copy to clipboard operation
MSBuildStructuredLog copied to clipboard

Parallelism for secret search

Open KirillOsenkov opened this issue 1 year ago • 4 comments

One todo would be to investigate adding parallelism to secret detection because right now it's really slow on real-life binlogs, even moderately sized. Takes over two minutes on a 5 MB binlog I use often. I think a judicious use of Task.Run() will really make things better here. You can maybe spawn one task per string, and one task per file, then just foreach the tasks and do task.Wait() on each of them. No need for async as only one thread will be blocked (the one where the computation happens)

KirillOsenkov avatar Nov 04 '24 22:11 KirillOsenkov

Once improved - what would you think about spinning it on background for all opens and then showing some form of info (node in the tree, goldbar, ...) if sensitive data found (except probably username)? It might feel too heavy - but preventing spilling secrets unintentionaly feels worth it.

JanKrivanek avatar Nov 05 '24 12:11 JanKrivanek

definitely too heavy, can't have it burn CPU and battery and interfere with the investigation. Needs to be an explicit gesture from user.

KirillOsenkov avatar Nov 05 '24 16:11 KirillOsenkov

I'm sad about it - but it makes perfect sense

JanKrivanek avatar Nov 05 '24 19:11 JanKrivanek

also from looking at the current perf this seems like it will be very expensive computationally, we should get some data for larger binlogs (and I can share some large binlogs if you need them)

KirillOsenkov avatar Nov 05 '24 20:11 KirillOsenkov