framework icon indicating copy to clipboard operation
framework copied to clipboard

first draft of segmented data loaders

Open mythmon opened this issue 9 months ago • 0 comments

This is a draft of segmented data loaders. It's probably missing a lot, and needs a lot more testing and polish. I wanted to share it early though, since I'm working with code I'm not familiar with. There are a few problems I know about already:

  • The config loading isn't good enough, and I want to revisit it, maybe with some library support.
  • I think that accessing files from archives throws an error if the archives are generated from a segmented data loader.
  • The CLI output of build doesn't make much sense.
  • I think we will need some extra metadata in the output so it is servable, even in principle.

Feedback wanted!

mythmon avatar May 09 '24 23:05 mythmon