flowpipeline
flowpipeline copied to clipboard
Add diskbuffer segment.
Allows to buffer flows in memory and (additionally) on disk in zstd compressed files.
Add a small segment (filegate) for testing purposes to pause the pipeline when a file exists.
Hi, thank you for the PR. I took a first look on it and have some comments and remarks to be adressed. Maybe you have already identified some of my points and maybe we need some more detailed testing and validation of this feature.
-
General structure and categorization:
- i think the
filegatesegments suits better to the already existingtestingdirectory/category instead of a new category like dev. - the new
diskbuffersegment will be better categorized within thecontrolflowcategory. i think its rather an offloading and manipulation of the control-flow in the current flowpipeline instance - if possible, it would be nice to add some test cases to the segment. To start maybe a simple "passthrough" test for knowing, that the segment works and passes through flows. For an example please find pass_test.go
- i think the
-
Diskbuffer segment
diskbuffer.goline 89 is probably reading the wrong configuration variable- it looks like data is still written to disk after
filesizeis reached. (after debug log in line 298 was triggered)