leakdb
leakdb copied to clipboard
moloch--
"source" field in normalized JSON?
Would it be feasible to add a "source" field to the JSON/indexed data, so you could "tag" entries as being from certain leaks.
This could be very useful when trying to go back later and attribute where a piece of data came from - but unsure if it would have performance impacts?
I don't think it would have much of an impact on performance, most of the code operates on lines not the actual content of the line, so there's little code that would need to change too. A few other folks have been asking for something like this so I'll probably look at adding it. It would affect the bloom filter's ability to effectively de-duplicate identical user/password combos since they'd be from different sources, so there'd could be a modest impact to index/sort times but i don't think there'd be a large impact to search times.
On 16.02.2021, at 01:25, Joe [email protected] wrote:
Not had time to work on it yet sorry!
No worries, just wanted to figure out what the status ist. What would be needed ? I.e. is it simple enough as a non-go coder to add it?
Best, a.
Maybe, most of the code only cares about "lines" in a file, you'd have to extend the normalizer to add a "source" field to the JSON format, and extend the few parts of the code that parse the JSON to optionally deal with the extra field.