add a topic for customization

[mshensg]

looking at the code at package/etc/conf.d/sources/source_syslog/plugin.jinja . it will send the data to the app-parsers registered at package/etc/conf.d/plugin/app_parser_topics.conf directly. This implies that all the built in parsers will take higher precedence over all customized parsers. If I create a parser, the data will be processed by the built in parsers first and then be sent to my parser. This creates complexity if we want our own parser to be the first one. So I would recommend to add an empty (no parser subscribed) topic as first topic so if we would like to make our own parser as the first, it can subscribe to that topic. Message ----> [sc4s-syslog] ===> built-in parser > customized parser (for sc4s-syslog) ----- Change To ------ Message -----> [sc4s-first] ------> [sc4s-syslog] So if there is a need to process the message before any built-in parser kicks in, we can just simply do application app-first[sc4s-first]. I have several customers having such requirements so I believe this will be helpful for them to pick up these messages without being processed by the built in parsers.

[rjha-splunk]

We will release it in next major version

[rjha-splunk]

i will relook into this.

[rjha-splunk]

@mshensg I will request you to test the feature mentioned in https://github.com/splunk/splunk-connect-for-syslog/pull/1794 , it should address it based on port ID and should work as required.

[mshensg]

Thanks @rjha-splunk , I will try it out and feedback. It will take some time.

[jrehm-mmm]

@mshensg - I had been subscribed to this thread as I had been struggling with the same issues before finding the "loophole" in the source parser logic. I'm keen to see if this solves all of your customers issues for which you envisioned a sc4s-first topic for. If not, I'd appreciate if you'd be sure to share your gaps here or be sure to tag me in slack @jrehm - TIA

[mshensg]

@rjha-splunk , I checked the description of #1794, it appears that this requires the data to be sent to a different port right? for example, I have special product a and b and general data sources, then I need 3 different ports to accept the data. can I use a single port to accept the data?

[mshensg]

@rjha-splunk , also, as per plugin.py, it has topic=os.getenv(f"SC4S_LISTEN_{ port_id }_TOPIC", "sc4s") to get a topic information, however, this topic is never referred in plugin.jinja. it will still goto the preset topics instead of using a customized topic. Please help check and let me know how this is supposed to work.

[rjha-splunk]

yes thats how it works , for special products data need to be sent to different port for the logic to work, for plugin.py yes i agree its not referenced but #1794 works well.

[mshensg]

Thanks @rjha-splunk . Then which topic should my custom parser listen on when I use this feature. Actually, can we have prefilter just like the postfilter so we can filter based on the raw event not the processed one?

[rjha-splunk]

list of the topics are https://github.com/splunk/splunk-connect-for-syslog/blob/main/package/etc/conf.d/plugin/app_parser_topics.conf

[mshensg]

I know the list but which one can allow me preprocess the event before it being parsed by built in parsers? If that is processed by these parsers, it might break my event format causing me cannot pick it up at post processing topic and it will also waste CPU to process an event.