Rory Finnegan

I don't think that's a fair comparison. Since the object serialization format is independent of the metadata, that would be more like: ``` julia> BSON.bson(bson("junk.bson", NamedTuple())) ERROR: MethodError: no method...

During an upgrade, if we can't add specific package versions then we fallback to just trying to the package without any version constraints, since they're sufficiently different between julia 0.6...

You can still load the raw dictionary with BSON though, the point of JLSO is to avoid throwing away information about the original environment as best as possible.

> We can always display a warning and a recommendation to pin JLSO to old version til we fix what ever bug. The issue is that this isn't a bug......

To simplify things, I've condensed the discussion about other file formats into the first paragraph and extended the format specification part.

A stacktrace would have been helpful as those snippets aren't the most general.

NOTE: Fannie Mae 2004Q3 data (2.7 Gb) isn't what's in the gist and I don't see a 2.7 Gb file in the dataset documented. Running the code on Julia 1.0.3...

I also couldn't get it to throw an error with a 9Gb CSV to push things a little more. https://www.kaggle.com/city-of-seattle/seattle-library-collection-inventory Fun fact, the default JLSO file format compresses the 9.1G...

Yep, can't reproduce on 1.2 either. ``` write_perf = [8.97624532, 6.488284407, 37.267838932, 24.014278148, 67.626921068, 64.134563347, 23.585749921, 12.960553185, 311.955533758, 318.826512276] read_perf = [11.179993041, 8.72340904, 21.159888766, 10.411692382, 0.887963974, 0.001172795, 14.963994836, 7.80005972, 43.608864438,...

Interesting, looks like this is a known issue with the BSON spec. Apparently, array types are indexed with an Int32. I'm not entirely sure why I wasn't able to hit...