FileIO.jl icon indicating copy to clipboard operation
FileIO.jl copied to clipboard

How popular should a file format be, to be added here?

Open Luapulu opened this issue 4 years ago • 5 comments

I have a loader package for a format used in a little corner of physics. I'd guess there might be tens of people using this format at most. Nonetheless, I've followed FileIO conventions while writing it so I could easily add it here.

How popular should a file format be, to be added here?

Luapulu avatar Nov 03 '20 15:11 Luapulu

My view: if it has nontrivial magic bytes, it's presumably unique so it won't get in the way of anything else. In that case, add it without hesitation.

If there are no magic bytes, and if the extension is not unique or is too short to be plausibly unique, then popularity becomes an important consideration.

timholy avatar Nov 03 '20 15:11 timholy

There aren't magic bytes but every line ends in "physiDet" and the first line has to contain three integers. (It's a human readable delimited-ish format) The ending is ".root.hits" so conceivably there may be some other simulation / data analysis software based on ROOT, that saves to a ".root.hits" file format.

Example

Luapulu avatar Nov 03 '20 15:11 Luapulu

Interesting. :+1: for it being easily recognizable. :-1: for the fact that it needs a custom detector, and the more of those we have to run the slower the general thing becomes. hits seems like a pretty unique extension, though. I forget (and don't have time to check) if we only run detectors on extensions that match.

timholy avatar Nov 03 '20 23:11 timholy

src/query.jl#L72-L90 seems to suggest, that if only one format exists for an extension and it has no magic bytes, it will get loaded as is without further checking.

Luapulu avatar Nov 04 '20 00:11 Luapulu

Ok, quick correction on my specific file format: The line ending may be different. First line still has three integers, but following lines may vary in terms of the ending string. For my use case this hasn't made a difference, but if this is to be added here I'd have to generalise my code first.

I'll let things mature a bit and may be come back to this later if makes sense to add rarely used formats here.

Luapulu avatar Nov 04 '20 22:11 Luapulu