System version more plugins

[jstucke]

• added system_version for more plugins
  • file_type: used version of FW magic DB
  • cve_lookup: used version of NVD JSON data feeds

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 79.48718% with 8 lines in your changes missing coverage. Please review.

Project coverage is 91.84%. Comparing base (ab95ac5) to head (181850b). Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
...ugins/analysis/cve_lookup/internal/data_parsing.py	52.94%	8 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1317      +/-   ##
==========================================
- Coverage   92.48%   91.84%   -0.64%     
==========================================
  Files         379      378       -1     
  Lines       24115    21157    -2958     
==========================================
- Hits        22302    19432    -2870     
+ Misses       1813     1725      -88     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[maringuu]

What problem does this solve? I do not quite understand how the system version is useful.

[jstucke]

What problem does this solve? I do not quite understand how the system version is useful.

It is useful for two things:

• track the internal version of the data that the analysis is based on (since the outcome of the analysis may change based on this data)
• if you run the analysis again with the same version of the plugin, it depends on the system version whether or not the analysis is run again or skipped (and since the outcome may change, it makes sense to update it)

[maringuu]

To me the system version still seems like a design flaw. Imo it would make more sense to fix the version of whatever tool a plugin needs and bump the plugin version if we update the tool.

Let's consider the file_type plugin for example. If I'm on ubuntu and happen to update my system (via apt upgrade) and the file package is updated, this results in a new system version for the file_type plugin. When I then upload a new firmware image to FACT, the results are different than what I would have gotten when I uploaded the same firmware before upgrading file. To me this does not make sense. The analysis results should not change, if I didn't touch the fact install.

[jstucke]

I tend to agree with you. That is one of the reasons why we don't install some tools through apt: To get reproducible results. In some cases it is hard to address this, e.g. in case of the cve_lookup plugin it wouldn't make a lot of sense to pin the commit of the CVE sources (because you would need to update the plugin all the time to get the latest entries). But it adds some meta information which makes it easier to track why we see certain results. Maybe it would make sense to remove the "system version" from scheduling.

And I think that we should try to eliminate as many sources of indeterminism/uncertainty as possible (like the version of file/libmagic). In the case of file/libmagic I'm not sure what the best way to do that would look like, though: compiling it adds unnecessary installation overhead and running it inside a docker container adds runtime overhead during analysis (which is even worse). Maybe downloading a Debian/Ubuntu package with the pre-compiled magic file and extracting it could work :see_no_evil:

[maringuu]

I opened #1349 to track the discussion.