suricata
suricata copied to clipboard
OISF
(draft) - rust configuration library - v8
Create a PR just for preview, and trigger QA.
This replaces libyaml with a Rust module using rust-yaml to load the yaml. While the rust-yaml scanner is used for the low level parsing, a custom loader (document builder) was required to handle the includes. This is largely a copy of the original from rust-yaml with some logic added for includes.
To get rid of libyaml, a converter was added to convert from a Rust Yaml datastructure to our configuration layout.
There's a bigger picture idea here which has some ideas starting to being implemented in the config crate. Like a YAML document with all the defaults. The idea being that a minimal user provided configuration gets every other field filled in with defaults, so something like "--dump-config" shows an accurate picture of the running config, and not just the user config. This is all very much a work in progress.
A benefit of including all the defaults as a hard coded YAML document is the possibility for type checking the user provided configuration. As YAML directly convertible to JSON (at least after handling includes) we could even check with jsonschema.
Issue: https://redmine.openinfosecfoundation.org/issues/4780
Codecov Report
Merging #7528 (dda4d9f) into master (2ba9da4) will increase coverage by
0.01%. The diff coverage is71.69%.
:exclamation: Current head dda4d9f differs from pull request most recent head 46d4c64. Consider uploading reports for the commit 46d4c64 to get more accurate results
@@ Coverage Diff @@
## master #7528 +/- ##
==========================================
+ Coverage 75.75% 75.77% +0.01%
==========================================
Files 657 657
Lines 186368 186175 -193
==========================================
- Hits 141176 141066 -110
+ Misses 45192 45109 -83
| Flag | Coverage Δ | |
|---|---|---|
| fuzzcorpus | 59.94% <55.10%> (-0.03%) |
:arrow_down: |
| suricata-verify | 52.24% <62.00%> (+0.04%) |
:arrow_up: |
| unittests | 60.77% <73.26%> (-0.01%) |
:arrow_down: |
Flags with carried forward coverage won't be shown. Click here to find out more.
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
I've pushed some changes to better show what I was heading towards with this work. With this PR if you provide a minimal configuraition like:
%YAML 1.1
---
outputs:
- eve-log:
enabled: true
then do a --dump-config you'll get something that looks like:
outputs.0 = eve-log
outputs.0.eve-log = enabled
outputs.0.eve-log.enabled = true
outputs.0.eve-log.filetype = regular
outputs.0.eve-log.filename = eve.json
outputs.0.eve-log.threaded = false
outputs.0.eve-log.identity = suricata
outputs.0.eve-log.facility = local5
outputs.0.eve-log.level = Info
outputs.0.eve-log.ethernet = false
outputs.0.eve-log.redis = server
outputs.0.eve-log.redis.server = 127.0.0.1
outputs.0.eve-log.redis.port = 6379
outputs.0.eve-log.redis.async = true
outputs.0.eve-log.redis.mode = lpush
outputs.0.eve-log.redis.key = suricata
outputs.0.eve-log.redis.pipelining = enabled
outputs.0.eve-log.redis.pipelining.enabled = true
outputs.0.eve-log.redis.pipelining.batch-size = 10
outputs.0.eve-log.metadata = true
outputs.0.eve-log.pcap-file = false
outputs.0.eve-log.community-id = false
outputs.0.eve-log.community-id-seed = 0
outputs.0.eve-log.xff = enabled
outputs.0.eve-log.xff.enabled = false
outputs.0.eve-log.xff.mode = extra-data
outputs.0.eve-log.xff.deployment = reverse
outputs.0.eve-log.xff.header = X-Forward-For
outputs.0.eve-log.types = alert
outputs.0.eve-log.types.alert = payload
outputs.0.eve-log.types.alert.payload = false
outputs.0.eve-log.types.alert.payload-buffer-size = 4kb
outputs.0.eve-log.types.alert.payload-printable = false
outputs.0.eve-log.types.alert.packet = false
outputs.0.eve-log.types.alert.http-body = false
outputs.0.eve-log.types.alert.http-body-printable = false
outputs.0.eve-log.types.alert.tagged-packets = false
outputs.0.eve-log.types.alert.metadata = app-layer
outputs.0.eve-log.types.alert.metadata.app-layer = true
outputs.0.eve-log.types.alert.metadata.flow = true
outputs.0.eve-log.types.alert.metadata.rule = raw
outputs.0.eve-log.types.alert.metadata.rule.raw = false
outputs.0.eve-log.types.alert.metadata.rule.metadata = true
vars = address-groups
vars.address-groups = HOME_NET
vars.address-groups.HOME_NET = [192.168.0.0/16,10.0.0.0/8,172.16.0.0/12]
vars.address-groups.EXTERNAL_NET = !$HOME_NET
vars.address-groups.HTTP_SERVERS = $HOME_NET
vars.address-groups.SMTP_SERVERS = $HOME_NET
vars.address-groups.SQL_SERVERS = $HOME_NET
vars.address-groups.DNS_SERVERS = $HOME_NET
vars.address-groups.TELNET_SERVERS = $HOME_NET
vars.address-groups.AIM_SERVERS = $EXTERNAL_NET
vars.address-groups.DC_SERVERS = $HOME_NET
vars.address-groups.DNP3_SERVER = $HOME_NET
vars.address-groups.DNP3_CLIENT = $HOME_NET
vars.address-groups.MODBUS_CLIENT = $HOME_NET
vars.address-groups.MODBUS_SERVER = $HOME_NET
vars.address-groups.ENIP_CLIENT = $HOME_NET
vars.address-groups.ENIP_SERVER = $HOME_NET
vars.port-groups = HTTP_PORTS
vars.port-groups.HTTP_PORTS = 80
vars.port-groups.SHELLCODE_PORTS = !80
vars.port-groups.ORACLE_PORTS = 1521
vars.port-groups.SSH_PORTS = 22
vars.port-groups.DNP3_PORTS = 20000
vars.port-groups.MODBUS_PORTS = 502
vars.port-groups.FILE_DATA_PORTS = [$HTTP_PORTS,110,143]
vars.port-groups.FTP_PORTS = 21
vars.port-groups.GENEVE_PORTS = 6081
vars.port-groups.VXLAN_PORTS = 4789
vars.port-groups.TEREDO_PORTS = 3544
app-layer = protocols
app-layer.protocols = telnet
app-layer.protocols.telnet = enabled
app-layer.protocols.telnet.enabled = true
app-layer.protocols.rfb = enabled
app-layer.protocols.rfb.enabled = true
app-layer.protocols.rfb.detection-ports = dp
app-layer.protocols.rfb.detection-ports.dp = 5900, 5901, 5902, 5903, 5904, 5905, 5906, 5907, 5908, 5909
app-layer.protocols.mqtt = enabled
app-layer.protocols.mqtt.enabled = true
app-layer.protocols.mqtt.max-msg-length = 1mb
app-layer.protocols.mqtt.subscribe-topic-match-limit = 100
app-layer.protocols.mqtt.unsubscribe-topic-match-limit = 100
app-layer.protocols.mqtt.max-tx = 4096
app-layer.protocols.krb5 = enabled
app-layer.protocols.krb5.enabled = true
app-layer.protocols.snmp = enabled
app-layer.protocols.snmp.enabled = true
app-layer.protocols.ike = enabled
app-layer.protocols.ike.enabled = true
app-layer.protocols.tls = enabled
app-layer.protocols.tls.enabled = true
app-layer.protocols.tls.detection-ports = dp
app-layer.protocols.tls.detection-ports.dp = 443
app-layer.protocols.tls.ja3-fingerprints = auto
app-layer.protocols.tls.encryption-handling = default
app-layer.protocols.pgsql = enabled
app-layer.protocols.pgsql.enabled = false
app-layer.protocols.pgsql.stream-depth = 0
app-layer.protocols.dcerpc = enabled
app-layer.protocols.dcerpc.enabled = true
app-layer.protocols.ftp = enabled
app-layer.protocols.ftp.enabled = true
app-layer.protocols.ftp.memcap = 64mb
app-layer.protocols.rdp = enabled
app-layer.protocols.rdp.enabled = true
app-layer.protocols.ssh = enabled
app-layer.protocols.ssh.enabled = true
app-layer.protocols.ssh.hassh = true
app-layer.protocols.http2 = enabled
app-layer.protocols.http2.enabled = true
app-layer.protocols.http2.max-streams = 4096
app-layer.protocols.http2.max-table-size = 65536
app-layer.protocols.smtp = enabled
app-layer.protocols.smtp.enabled = true
app-layer.protocols.smtp.raw-extraction = false
app-layer.protocols.smtp.mime = decode-mime
app-layer.protocols.smtp.mime.decode-mime = true
app-layer.protocols.smtp.mime.decode-base64 = true
app-layer.protocols.smtp.mime.decode-quoted-printable = true
app-layer.protocols.smtp.mime.header-value-depth = 2000
app-layer.protocols.smtp.mime.extract-urls = true
app-layer.protocols.smtp.mime.extract-urls-schemes.0 = http
app-layer.protocols.smtp.mime.log-url-scheme = false
app-layer.protocols.smtp.mime.body-md5 = false
app-layer.protocols.smtp.inspected-tracker = content-limit
app-layer.protocols.smtp.inspected-tracker.content-limit = 100000
app-layer.protocols.smtp.inspected-tracker.content-inspect-min-size = 32768
app-layer.protocols.smtp.inspected-tracker.content-inspect-window = 4096
app-layer.protocols.imap = enabled
app-layer.protocols.imap.enabled = detection-only
app-layer.protocols.smb = enabled
app-layer.protocols.smb.enabled = yes
app-layer.protocols.smb.detection-ports = dp
app-layer.protocols.smb.detection-ports.dp = 139, 445
app-layer.protocols.smb.stream-depth = 0
app-layer.protocols.nfs = enabled
app-layer.protocols.nfs.enabled = yes
app-layer.protocols.nfs.max-tx = 1024
app-layer.protocols.tftp = enabled
app-layer.protocols.tftp.enabled = yes
app-layer.protocols.dns = tcp
app-layer.protocols.dns.tcp = enabled
app-layer.protocols.dns.tcp.enabled = yes
app-layer.protocols.dns.tcp.detection-ports = dp
app-layer.protocols.dns.tcp.detection-ports.dp = 53
app-layer.protocols.dns.udp = enabled
app-layer.protocols.dns.udp.enabled = yes
app-layer.protocols.dns.udp.detection-ports = dp
app-layer.protocols.dns.udp.detection-ports.dp = 53
app-layer.protocols.http = enabled
app-layer.protocols.http.enabled = yes
app-layer.protocols.http.libhtp = default-config
app-layer.protocols.http.libhtp.default-config = personality
app-layer.protocols.http.libhtp.default-config.personality = IDS
app-layer.protocols.http.libhtp.default-config.request-body-limit = 100kb
app-layer.protocols.http.libhtp.default-config.response-body-limit = 100kb
app-layer.protocols.http.libhtp.default-config.request-body-minimal-inspect-size = 32kb
app-layer.protocols.http.libhtp.default-config.request-body-inspect-window = 4kb
app-layer.protocols.http.libhtp.default-config.response-body-minimal-inspect-size = 40kb
app-layer.protocols.http.libhtp.default-config.response-body-inspect-window = 16kb
app-layer.protocols.http.libhtp.default-config.response-body-decompress-layer-limit = 2
app-layer.protocols.http.libhtp.default-config.http-body-inline = auto
app-layer.protocols.http.libhtp.default-config.swf-decompression = enabled
app-layer.protocols.http.libhtp.default-config.swf-decompression.enabled = yes
app-layer.protocols.http.libhtp.default-config.swf-decompression.type = both
app-layer.protocols.http.libhtp.default-config.swf-decompression.compress-depth = 100kb
app-layer.protocols.http.libhtp.default-config.swf-decompression.decompress-depth = 100kb
app-layer.protocols.http.libhtp.default-config.randomize-inspection-sizes = yes
app-layer.protocols.http.libhtp.default-config.randomize-inspection-range = 10
app-layer.protocols.http.libhtp.default-config.double-decode-path = no
app-layer.protocols.http.libhtp.default-config.double-decode-query = no
app-layer.protocols.http.libhtp.default-config.lzma-enabled = false
app-layer.protocols.http.libhtp.default-config.lzma-memlimit = 1mb
app-layer.protocols.http.libhtp.default-config.compression-bomb-limit = 1mb
app-layer.protocols.http.libhtp.default-config.decompression-time-limit = 100000
app-layer.protocols.modbus = request-flood
app-layer.protocols.modbus.request-flood = 500
app-layer.protocols.modbus.enabled = no
app-layer.protocols.modbus.detection-ports = dp
app-layer.protocols.modbus.detection-ports.dp = 502
app-layer.protocols.modbus.stream-depth = 0
app-layer.protocols.dnp3 = enabled
app-layer.protocols.dnp3.enabled = no
app-layer.protocols.dnp3.detection-ports = dp
app-layer.protocols.dnp3.detection-ports.dp = 20000
app-layer.protocols.enip = enabled
app-layer.protocols.enip.enabled = no
app-layer.protocols.enip.detection-ports = dp
app-layer.protocols.enip.detection-ports.dp = 44818
app-layer.protocols.enip.detection-ports.sp = 44818
app-layer.protocols.ntp = enabled
app-layer.protocols.ntp.enabled = yes
app-layer.protocols.quic = enabled
app-layer.protocols.quic.enabled = yes
app-layer.protocols.dhcp = enabled
app-layer.protocols.dhcp.enabled = yes
app-layer.protocols.sip = enabled
app-layer.protocols.sip.enabled = yes
The embedded defaults aren't fully complete yet, but this should get the idea across - you'll see the complete running config, not just the user provided config. I could see adding options to only show values that differ from the defaults, or a --dump-defaults, etc.
Nice! Any thoughts on how to make sure any built-in default goes through the proper API?
Nice! Any thoughts on how to make sure any built-in default goes through the proper API?
Sort of. There is no way to enforce it of course, but we can try to make a convention of sorts. Take a look here: https://github.com/OISF/suricata/blob/master/src/app-layer-ftp.c#L142, if the configuration value isn't present we use a hard-coded default. An alternative is to error our with a fatal error if the configuration key doesn't exist. As S-V has a lot of tests with incomplete tests, I doubt we'd ever end up with something getting into master without its default value being added to default.yaml.
I'm not sure how this works for libsuricata tho. You'd have to provide a default configuration for every component you use. Maybe a move to the builder pattern is something to consider for components. You could have CreateNewFtpAppPlayerParseFromConfig(configuration_node) as one builder, and then a purely programmatic builder for library users who may be getting configuration from other sources. I don't think it would be too hard to prototype this idea with FTP as a C example, or DNS or DHCP as a Rust example
Loosely related question : I am getting a bit confused about outputs.0.eve-log = enabled when I read in the yaml outputs.eve-log = enabled
How does it get used with --set ? Can we set outputs.*.eve-log.types.alert.payload = false ?
Loosely related question : I am getting a bit confused about
outputs.0.eve-log = enabledwhen I read in the yamloutputs.eve-log = enabledHow does it get used with
--set? Can we setoutputs.*.eve-log.types.alert.payload = false?
Unfortunately setting items in lists is a little tricky and is not handled very well, even today. Given the YAML is:
outputs:
- eve-log:
enaled: yes
This expands out like outputs[0].eve-log = enabled, not outputs.eve-log = enabled. I think the answer is to come up with an expression syntax, perhaps like the one you mention above with the wildcard that is run after the configuration is loaded, and does a pattern match of sorts. Maybe we can take ideas from jsonpath here? Anyways, this work does nothing to improse this issue, but should not be changing its current behaviour either.
