beats icon indicating copy to clipboard operation
beats copied to clipboard

Add the missing process.name field to System module, Syslog fileset

Open belimawr opened this issue 1 year ago • 3 comments

Proposed commit message

This PR adds the missing process.name field to System module, Syslog fileset

Checklist

  • [x] My code follows the style guidelines of this project
  • [x] I have commented my code, particularly in hard-to-understand areas
  • ~~[ ] I have made corresponding changes to the documentation~~
  • ~~[ ] I have made corresponding change to the default configuration files~~
  • [x] I have added tests that prove my fix is effective or that my feature works
  • ~~[ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.~~

~~## Disruptive User Impact~~

~~## Author's Checklist~~

How to test this PR locally

  1. Make sure you're testing on a system where system logs are sent to Journald. The Debian 12 Vagrant VM from Beats is a good option.
  2. Package Filebeat (adjust for your system/platform). If you don't package it, dashboards won't be loaded
    DEV=true SNAPSHOT=true EXTERNAL=true PACKAGES="tar.gz" PLATFORMS=linux/amd64 mage -v package
    
  3. Extract it, enable the system module
    ./filebeat modules enable system
    
  4. Edit filebeat/modules.d/system.yml to enable the syslog fileset and ensure Journald input will be used
    - module: system
      syslog:
        enabled: true
        var.use_journald: true
    
  5. Edit filebeat.yml with your ES and Kibana credentials
    filebeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false
      reload.period: 1s
    
    setup.template:
      settings:
        index.number_of_shards: 1
    
    setup.kibana:
      host: "http://kibana:5601"
      username: admin
      password: testing
      ssl.verification_mode: none
    
    output.elasticsearch:
      hosts: ["http://elasticsearch:9200"]
      preset: latency
      protocol: "http"
    
      username: admin
      password: testing
      ssl.verification_mode: none
    
  6. Run the setup command
    ./filebeat setup --modules system
    
  7. Start Filebeat
  8. Ensure the events contain process.name and the [Filebeat System] Syslog dashboard ECS contains data. You might have to adjust the time window

Related issues

  • Fixes https://github.com/elastic/beats/issues/41353

~~## Use cases~~ ~~## Screenshots~~ ~~## Logs~~

belimawr avatar Oct 21 '24 20:10 belimawr

This pull request does not have a backport label. If this is a bug or security fix, could you label this PR @belimawr? 🙏. For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed branches, such as:

  • backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

mergify[bot] avatar Oct 21 '24 20:10 mergify[bot]

backport-8.x has been added to help with the transition to the new branch 8.x. If you don't need it please use backport-skip label and remove the backport-8.x label.

mergify[bot] avatar Oct 21 '24 20:10 mergify[bot]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

elasticmachine avatar Oct 22 '24 14:10 elasticmachine

Shouldn't we have additional testing to ensure we won't break it in the future?

pierrehilbert avatar Oct 23 '24 14:10 pierrehilbert

Shouldn't we have additional testing to ensure we won't break it in the future?

We technically have those tests in place already, when the tests run on CI they ingest some files and compare the final event with a golden file, if fields are different, the test fails.

The problem is that when we add a new source for the test, we don't have a reference, so they're generated. You can see the *-expected.json files that this PR updated. Most of the test data I added for the journald input do not have the process.name, so ad the time I did not realised it was mostly required.

One type of test that do not have and would be nice to have is to have a test dataset that can populate all visualisations from the dashboards and ensure that when ingested, the dashboard is correctly populated. That's not as simple to implement but it would be nice to have.

belimawr avatar Oct 23 '24 20:10 belimawr