suricata icon indicating copy to clipboard operation
suricata copied to clipboard

hyperscan: add caching mechanism for hyperscan contexts v8

Open lukashino opened this issue 4 months ago • 2 comments

Followup of https://github.com/OISF/suricata/pull/11887

Cache Hyperscan serialized databases to disk to prevent compilation of the same databases when Suricata is run again with the same ruleset. The current work operates in the logging folder and caches individual Hyperscan databases - potentially the ruleset might be even slightly changed and it still can reuse part of the unchanged signature groups. Loading fresh ET Open ruleset: 19 seconds Loading cached ET Open ruleset: 07 seconds

Link to ticket: https://redmine.openinfosecfoundation.org/issues/7170

Describe changes: v7: (v6 was private)

  • fix docs and add ticket number to the commit
  • fix privilege drop issue, files are created after privilege drop, tested also rule reload - it worked fine
  • refactor the util-mpm-hs code, primarily prepare function
  • rebase

v5:

  • rebased
  • commit message update
  • docs update

v4:

  • rebased
  • changed the default caching directory to somewhere /var/lib/suricata/cache/hs
  • custom cache directory path option added
  • docs added
  • the default settings changed - enabled on the config generation, disabled when the option is not present in the config

v3

  • rebased
  • MPM caching is still left on by default.

v2

  • improved styling to follow Suricata code styleguide
  • increased cache file name length from 10 to 20 characters
  • cache file name is a hash of the patterns - now only HS relevant fields are hashed - as long as the group of patterns itself is not changed then it is reused
  • minor refactors
  • added a safe variant of littlehash2 function
  • added suricata.yaml option to enable/disable caching
  • changed the storage location to the configured logging directory

v1

  • initial work to cache and load Hyperscan databases from the disk

lukashino avatar Oct 06 '24 22:10 lukashino