framework
framework copied to clipboard
File-based archive/extractor data loaders?
In the discussion in #1222 we're thinking of using the filesystem to generate multiple files from a data loader. This could be better for huge files (e.g. video) than having to go through “zipping”; another benefit would be to make it easier for the user, who is used to saving many files, and not so much to load a zip module/extension (although we provide clear examples of how to do that).
An idea could be to use the .list extension (based on the research I did in #207, both .uri and .list seem to be usable, and .list is a better name); the data loader would write files "somewhere"[1], and return a list of (relative) file names.
[1] Would we provide this location (directory name) with an ENV variable? Would we create that directory? Would we leave it up to the script to create it—with risks of conflicts and no proper way to clean up?