Split file format should be more strict

[thaJeztah]

The single-file / split-file format currently uses a fixed order for the files that are combined; this approach may not be future-proof / extensible for future enhancements (for example; multiple compose files, or multiple (named) configurations). (Just the file content of each section is likely insufficient to determine its contents, without attempting to fully parse each part, and trying to match it to a schema)

I suggest to either;

• add meta-data to each section in the file (name of file, schema/file format used)
• add a header-section that describes the content of each section/file included in the split-file
• other options?

This is just a quick write-up to start the discussion, because I keep forgetting to bring this up, and it's important to decide on this before the project reaches GA

[thaJeztah]

ping @garethr @vdemeester @chris-crone

[thaJeztah]

Also wondering if a (g)zip approach of the directory format has been considered as an alternative; doing so would not require us to invent a new file format, and still be able to distribute as a single file

[chris-crone]

This is something I've been thinking about too.

Also wondering if a (g)zip approach of the directory format has been considered as an alternative; doing so would not require us to invent a new file format, and still be able to distribute as a single file

We did (do?) have an option to make a tarball of the directory. @garethr asked for its inclusion after meeting with people in SF so he will have more context. If it's just about sharing then I think the image format is much better. If the use case requires some sort of editing of the file, a tarball/zip is cumbersome.

[thaJeztah]

Wondering how much the split-file format brings to the table; having everything in a single file makes comparing (eg config a/b) difficult; editors will likely have trouble providing code completion / hinting (because it's not a single schema)

[garethr]

The single file is super-nice when it comes to just editing one file. It's a much nicer user experience in my experience so far, although we'll do some testing of this when we have the bandwidth.

With regards the ordering, we're currently doing some work to formally specify the package format and components. That should include schemas and some parsing instructions bits. A fixed order is certainly one option, but I think I agree that some of the other options are nicer. @silvin-lubecki is currently looking at that I believe.

I'm not sure what a gzip option gives us that's not better served by the options we have, but if there is a concrete usecase let me know.