Adding base functionality for source confidence scoring

[redsand]

Adds base functionality for new scoring technique. Implemented as a misp module, hoping to see it brought internally to the app

[lgtm-com[bot]]

This pull request introduces 24 alerts when merging e1e7d49e462c6a3b56abd5b327f375eef800d598 into 900fe56fbbdb6165ad9659634d3af8c369c84214 - view on LGTM.com

new alerts:

• 17 for Unused import
• 4 for Except block handles 'BaseException'
• 2 for Unused local variable
• 1 for Module is imported more than once

[lgtm-com[bot]]

This pull request introduces 13 alerts when merging 7e77058b28736e2940b414f521d76edced5f55a1 into 900fe56fbbdb6165ad9659634d3af8c369c84214 - view on LGTM.com

new alerts:

• 7 for Unused import
• 4 for Except block handles 'BaseException'
• 2 for Unused local variable

[lgtm-com[bot]]

This pull request introduces 13 alerts when merging 79acdecb2c50e8444319d77327c72b891daf6b84 into ab2354784479f8e5dce5eb2b1e9a79d3c4917731 - view on LGTM.com

new alerts:

• 7 for Unused import
• 4 for Except block handles 'BaseException'
• 2 for Unused local variable

[mokaddem]

Hello @redsand! Thanks a lot for your pull request. Please find my comments below:

1. I am curious why this script is not relaying on the built-in decaying model of MISP.
   • Are you missing something that is not implemented or not working the way you'd like in the default MISP's implementation?
2. I see some debugging leftovers (comments and commented code) which should be cleaned
3. Querying back MISP for every value is extremely costly and not doable in a production system
   • https://github.com/MISP/misp-modules/blob/79acdecb2c50e8444319d77327c72b891daf6b84/misp_modules/modules/expansion/source_confidence.py#L152
   • You could query back MISP from another script but having this step in the pipeline for every value is too costly
     • You have the query handshake + authentication + full database search + returning the value + processing the output
4. The way I would have seen this module only handle the confidence part and relying on the MISP's built-in decaying implementation. So, only this part
   • https://github.com/MISP/misp-modules/blob/79acdecb2c50e8444319d77327c72b891daf6b84/misp_modules/modules/expansion/source_confidence.py#L211
   • Where the complete steps would be:
     1. User issue a restSearch with decaying enabled and filtering out expired data
     2. MISP compute decaying score
     3. MISP provide the score and more data to the MISP module
     4. The MISP-module returns back either a weight or the modified score
     5. MISP filter out results based on the MISP-module feedback and return data to the user

Let us know what you think!

[redsand]

1. This implementation was chosen because it was recommended per our meeting with the MISP team last year on a conference call. I am not familiar with the broader MISP project's codebase, per this suggestion.

2. I can certainly remove any debugging output, oopsie!

3. Querying back for all the data for the attribute is required for properly calculating the score (total_score), since its a representation of the attribute and its properties for each source provider. I have solved for the cost by precalculating all attributes and updating their scores periodically. More specifically, for our implementation internally, all items are scored and exported out as csv's for real-time processing of our MDR platform.

4. This is meant for your team to better understand how the paper is written and identify the best way for this feature to be applied at the production level. I noticed several workflows (as you have) that do not compliment the method of how the research paper was written. For us, we are able to use the source confidence tables along with processing the data on export to calculate all values at that time, and we simply then perform this export every X hours or days.