[DATA] Support for Biologic .mpr files

[battery-sandy]

• What instrument/software/file type are you requesting support for? *

Biologic's new data file ".mpr". The exact headers are difficult to decipher at this moment. But galvani has reverse engineered the file to extract the data.

github.com/echemdata/galvani/blob/master/galvani/BioLogic.py

This needs to be implemented so that new data files can also be used.

[Please attach a sample file by dragging and dropping an example file (this file will be public)]

[hkennyv]

@battery-sandy is it possible to upload a sample .mpr file here?

[BGerwe]

@hkennyv I can share an .mpr from EC-Lab v11.18, but I've never worked with reading binary files so I didn't want to mess with it originally. But we should discuss as a group whether to include galvani as a dependency or work up a similar solution of our own.

[BGerwe]

The data file is now in the data/BioLogic_mpr branch

[hkennyv]

Thanks @BGerwe! I'd imagine rolling your own solution would be in the package's best interest to avoid introducing new deps. I don't have a ton of exp working w/ binary files either

[mdmurbach]

Cool, thanks for sharing the example file Brian :) One other thing to note is that galvani’s license (GPL3) is incompatible with the MIT license we use, so we will need to do the reverse engineering of the binary file ourselves if we want to add this feature.

Working with binary files isn’t too bad (struct.unpack is a great tool) but the reverse engineering seems like the slightly harder piece. Any chance you have the resulting data in an easily accessible format or know more about what the file actually contains Brian?

[BGerwe]

Yep! Just pushed a commit replacing the .mpr with the same data that's in exampleDataBioLogic.mpt, and added a .txt version too. The .txt is also attached here.

To explain a bit, .mpr files are binary files output by the instrument, but with EC-Lab they can be exported to .mpt (retaining the metadata), or .txt (containing only the frequencies and impedance data).

[battery-sandy]

@BGerwe @mdmurbach I have attached a Test.mpr file and the text file with all the data that it contains. (Extracted using galvani's code) If you need more data files, I can provide them. I have tried to go over the code for .mpr and as you rightly pointed out is a bit tricky. Test.zip

[mdmurbach]

Played around a bit today at lunch and have been able to partially figure out the structure of the data. For example,

import struct

with open('exampleDataBioLogic.mpr', 'rb') as f:
    r = f.readlines()

# The data appears to start at byte 570
# f, z', z'', |z|, phase, time, E, I, cs, cp, ...
# 10 unknown bytes but probably cycle number and i range,  mag_V, mag_I

struct.unpack('<5fd4f10s2f', r[1][570:632])

Output: (matches first line of mpt file data)

(1000.320068359375,
 65.47088623046875,
 0.38998979330062866,
 65.4720458984375,
 -0.3412891924381256,
 0.7836904905780102,
 -0.0002867755538318306,
 -0.04804118722677231,
 407.9697265625,
 0.014475133270025253,
 b'\x00\x00\x00\x00\x00\x00\xf0?\x0b\x00',
 0.020979473367333412,
 0.00032043407554738224)

It appears this 62-byte structure (5fd4f10s2f) repeats for the rest of the data.

Many open questions: What's in the first 570, I've seen -2.5, 2.5, etc. so I'm pretty sure it's the file header, but can we figure out how to parse it? Is this 570 offset true for all .mpr files or do we need to do some other parsing to know what to look for (would be good to try with battery-sandy's file too)?

Feel free to run with this if anyone is interested, I probably won't be able to do much more until this weekend, but happy to answer any questions.

[nickbrady]

If you look at the corresponding .mpt file, yes they have header information above the data. -2.5, 2.5 probably corresponds to "Ewe ctrl range : min = -2.50 V, max = 2.50 V" in the .mpt file.

In the .mpt file, the first line of data is dynamic: "Nb header lines : 61", so it is possible the 570 could change.

I also have .mpr / .mpt files with different number of columns and types of columns - could the 62-byte structure also change?

I am not familiar with binary data structures, but I would like to help with this. Naive question: how did you recognize the structure as (5fd4f10s2f)?

[mdmurbach]

Cool! yeah, I think you're right that the structure could definitely change depending on what information is there.

In terms of how to recognize the structure (f is float, d are doubles, docs here), I haven't spent much time trying to see if there are headers or anything, I simply took a pretty brute force approach of just looking for floats that were the right order of magnitude until finding some sort of a pattern. It seems like a lot of the file is actually a bunch of zeros.

E.g.

with open('impedance.py/data/exampleDataBioLogic.mpr', 'rb') as f:
    r = f.readlines()

for start in range(500, 700):
    print(start, struct.unpack('f', r[1][start:start+4])[0])

print(struct.unpack('<5fd4f10s2f', r[1][570:632]))

[p2jones]

@MikeJewski, I think you've solved this issue elsewhere. Care to take a look?

[MikeJewski]

I had also used the code from https://github.com/echemdata/galvani/blob/master/galvani/BioLogic.py, as I just needed to get it working. But trying to reimplement what he had done was a bit too much work for me. Calling his script and loading the MPR data was fine for my needs.

[mdmurbach]

@petermattia also showed (#132) an example of how to import/call the galvani package like @MikeJewski mentioned. I don't think we can directly import the galvani package since it's GPL (although I'm not a lawyer), but we could definitely add an example of how to do it to the documentation.

[chatcannon]

Continuing the discussion on https://github.com/echemdata/galvani/issues/51

No copyright I have on my own code can forbid someone from writing import galvani in a file and distributing that, either on its own or in a source tarball. I'm pretty sure that even a setup.py bdist will just create the bytecode IMPORT_NAME("galvani") which is also not a derivative work.