echopype
echopype copied to clipboard
Proposed reorganization of AD2CP data format
In v0.6.0 we left out AD2CP data because it takes quite a different flavor from all the other sonar models. Now we are in the position to reorganize the data format based on the previous experience.
In this PR I put in a markdown file that details my proposed changes of the data format. The file is put under docs/source
but my intension is not to merge this file but use this PR as a place where we can do line by line exchanges/discussions if needed.
@emiliom @imranmaj : I think it may work better if @emiliom and I meet first to settle down the proposed changes (which will be become new commits to this PR), and then @imranmaj and I meet to go through the required changes.
@emiliom: I can give you an overview of how this instrument operations and the main differences from the other sonar models. The variables are in such a way that can be grouped into a few conceptual groups, so even though there are a large number of them I think it won't be too hard for us to make decisions.
@imranmaj: There are a few questions that are independent of the format changes. We can discuss those in parallel.
A new question as I test the new parser/organization from #731 using data from my fieldwork last year: I think it'll be useful if we can save the entire configuration string as an attribute (or another appropriate form) in the converted data. This will ensure no loss of information, since right now the config info is parsed but not save.
This is an example of the config string saved as a txt file: https://drive.google.com/file/d/1q-Njr7ZlebLiHjnZElsRlcYMksudPw9q/view?usp=sharing
@imranmaj has a nice function in the parser to convert the config string to dict, and it is used in the parser for getting some specific info, but the config params are not saved in general.
@emiliom : Do you have recommendation on how this should be saved? I see this as similar to the config XML that is saved in its entity for EK80 in the Vendor_specific
group:
https://github.com/OSOceanAcoustics/echopype/blob/12693a72badcf54a18408e694b72e7b0e6d48023/echopype/convert/set_groups_ek80.py#L988
Given that config_xml
attribute precedent, it seems to me that what you're proposing makes sense. Plus, preserving config info is always a good thing, especially if there are no downsides. You could store the raw text file as is, or convert it to a dict then save it as a JSON string. The only comment I'd make about the format of that dict is that each of its items (eg, item key "BEAMCFGLIST") often has content in the form of a list of dicts, eg:
"BEAMCFGLIST": [dict1, dict2, ...]