perrygreenfield
perrygreenfield
Perhaps the right thing to do is define a clear mechanism for custom binary formats that aren't part of the standard. Nonstandard YAML tags are not a big obstacle since...
Hmmm, specifically Pickle, but making such a long laundry list a required part of the standard just seems unreasonable to me. "You don't meet the standard unless you support all...
Another aspect of doing that is to be careful not to encode into the file peculiar implementation aspects that are especially targeted to one language. Generally Python libraries are reasonable,...
@eslavich and I have been discussing this a bit the last day and I think we are starting to converge on how we would like to go ahead with this...
More work on the Python interface for specifying the compression to be applied when creating a binary entity is needed, but that should be fairly straightforward. I'll be taking off...
Isn't this basically supporting chunking?
I guess the question becomes whether the chunks should be in separate blocks. Yes, if they are compressed; is that a common usage? I worry that a large number chunks...
Given the fact that zarr supports compressed chunks, I don't think there is any sensible way of having the chunks within or or many asdf binary blocks that would be...
I think both ways are useful. Immutable data could be supported as well saving data within the file. I'll see if I can come up with an outline for both...
@eschnett we've been discussing this quite a bit internally to come up with proposals for how to deal with these kinds of cases and are beginning to firm up our...