Add ASIM filtering parser for vimNetworkSessionPfELK

[deggis]

Required items:

Change(s):

• Add filtering(?) ASIM parser for PfELK: vimNetworkSessionPfELK

Reason for Change(s):

• To add support for ingesting PfSense/OPNsense logs for users who already use PfELK, or who want to ship logs using Logstash. More info here https://github.com/deggis/pfelk-azure-sentinel

Version Updated:

• N/A

Testing Completed:

• vimNetworkSessionPfELK: ASimSchemaTester: passes without errors. Added also some optional + recommended fields and aliases.
• vimNetworkSessionPfELK: ASimDataTester. 2 errors: EventProduct and EventVendor does not match enumerated values, as expected.
  • To be first discussed here: is PfELK is a thing to be supported? Secondary, what values for those fields should be.
• _Im_NetworkSession's filtering parameters appear to work in testing.
  • Three exceptions: url_has_any, httpuseragent_has_any, hostname_has_any. I kept them to have interface match, but commented them with // not used.

Checked that the validations are passing and have addressed any issues that are present:

• (Added in above part)

[deggis]

@microsoft-github-policy-service agree

@microsoft-github-policy-service agree

[devikamehra]

@v-atulyadav Can you work with Prateek to decide whether this contribution should come into Data Connector folder or should be packaged as a solution. In case the solution gets added, we can review the ASIM Parser otherwise there should only be a normal product parser. Please work out the process for this PR.

[v-atulyadav]

Hi @deggis, please act on below points

1. Create a solution in solution folder and move the file into the solution folder (only data connector needs to be moved in the solution folder. Data connector needs to be created instead of README file). Refer below data connector templates link for the same. https://github.com/Azure/Azure-Sentinel/tree/master/DataConnectors/Templates

2. create ASIM parser ( non parameterized )along with VIM parser ( parameterized). Please refer link below to build these contents. https://github.com/Azure/Azure-Sentinel/tree/master/Parsers#guide-to-build-asim-parsers-for-microsoft-sentinel

3. run data and schema test and attach the report in test folder

4. Please add sample data in below path

5. Please add schema of custom table which is referred in parser file i.e PfELK_CL into below location https://github.com/Azure/Azure-Sentinel/tree/master/.script/tests/KqlvalidationsTests/CustomTables

[v-atulyadav]

Hi @deggis, I look forward to hearing back from you. Thanks

[v-atulyadav]

Hi @deggis, we are waiting for your response. Thanks

[v-atulyadav]

Hi @deggis, waiting for your response. Thanks

[v-atulyadav]

Hi @deggis, we are still waiting for your response. Thanks

[deggis]

@v-atulyadav thanks for reaching out, I had forgotten about this. I'm currently on vacation from IT stuff. I will be back in August, I'll look into this then.

[v-atulyadav]

Hi @deggis, is it possible to move this PR into draft until your changes are complete? Thanks

[v-atulyadav]

Hi @deggis, could you please let us know when this PR will be ready for review? Thanks

[deggis]

Hi @deggis, could you please let us know when this PR will be ready for review? Thanks

Sure, I'll let you know. This got delayed by a bit.

[v-atulyadav]

Thanks @deggis, Once you have finished making your changes, please let us know. Thanks

[deggis]

Not ready yet, but I have a question: for 1) "Create a solution in solution folder", would it be ok in Data Connector json's instructions to just link to an external guide?

This seems to be still up to date https://in.security/2022/11/28/logstash-sentinel-round-two/ . By checking other Data connectors, I saw a DCR being created, but here we would also need a new DCE, the application identity, new secret for the app, and the deployment should provide some values (including the app secret) as output.

2nd question: is it possible for me to test the Solution beforehands? I added templates to a temporary repository (at https://github.com/deggis/azure-sentinel-pfelk-solution ), added it as a Repository in my Sentinel instance, but out of the supported content types, Solutions aren't in this list.

Otherwise, first version of filterless ASIM parser ready and I have a separate demo env with data which I can use to collect sample data now.

[v-atulyadav]

Hi @deggis, we will provide you an update on your asks soon. Thanks

[v-atulyadav]

Hi @deggis, Do we need this PR? Thanks

[deggis]

@v-atulyadav I don't believe I'm in a position to make that call.

Regardless of how that turns out: is it already possible to test including a solution from a private repository? I don't see Solutions as supported content types yet at least in the user interface.