framework
framework copied to clipboard
first draft of segmented data loaders
This is a draft of segmented data loaders. It's probably missing a lot, and needs a lot more testing and polish. I wanted to share it early though, since I'm working with code I'm not familiar with. There are a few problems I know about already:
- The config loading isn't good enough, and I want to revisit it, maybe with some library support.
- I think that accessing files from archives throws an error if the archives are generated from a segmented data loader.
- The CLI output of build doesn't make much sense.
- I think we will need some extra metadata in the output so it is servable, even in principle.
Feedback wanted!