Stream YAML/JSON parsing to reduce memory consumption on refresh

[jsvd]

During refresh parse operations like Psych::Parser.parse will consume a significant amount of memory due to the construction of parser-specific objects like TreeNode, before the final - and simpler - ruby Hash is constructed.

While this memory is then garbage collected, there is still a concern that multiple translate filters and multiple pipelines that happen to perform refresh at the same time (likely if the pipelines start at the same time and have the same refresh_interval), then an OOM may occur.

We should aim at having a refresh operation that does not consume a significant amount of memory besides the one needed for the final object.

For the yaml file loading, this should be possible by having a custom Psych::Handler or switching to a snakeyaml-engine based implementation.

• [ ] JSON file stream parsing
• [ ] CSV file stream parsing