Google's Crush Tools
I don't know if you've seen these but these are a group of file manipulation tools created by the Google engineering team. I thought I'd forward them to you as it seems they could fit and be useful in your framework.
https://github.com/google/crush-tools
Here are the docs:
https://github.com/google/crush-tools/tree/wiki
Frank
Thanks! Good to know this cool CLI tool.
Here's a doc (I know, the repo is a bit hard to navigate) that gives a good outline of how they are used traditionally: https://github.com/google/crush-tools/blob/wiki/DataProcessingScenarios.md
Thank you @davidsonff these tools are proving to be very helpful for me together with gleam.