brukeropusreader
brukeropusreader copied to clipboard
qedsoftware
Unreadable measurement
Firstly, thank you very much for providing this python package.
One of my recordings done on a Bruker Tensor 27 spectrometer can't be read by opusreader.
The OPUS viewer displays is able to display the measurement (blue). I also show the 'readable' example in red (See below)
When reading the two files I get widely different absorption values:
from brukeropusreader import read_file
unreadable = read_file('C:\\Users\\David\\Desktop\\opus troublshooting\\unreadable.0000')
readable = read_file('C:\\Users\\David\\Desktop\\opus troublshooting\\readable.0006')
"""
unreadable['AB'][0:30]
array([1.68155816e-44, 2.80259693e-42, 1.40129846e-45, 6.27109369e-39,
1.83670992e-40, 5.88323249e-39, 7.37636505e-39, 1.83670992e-40,
1.40129846e-45, 8.29149843e-39, 1.83670992e-40, 4.09179152e-43,
7.74048685e-39, 4.77544860e-39, 7.21432390e+22, 7.14469695e+31,
2.01962467e-19, 7.43988911e+28, 3.72261848e+27, 6.68376203e+22,
1.31832032e+25, 2.28019990e-12, 1.10184478e+24, 1.93459115e-19,
1.57949750e-19, 2.96726393e+26, 2.69185152e+09, 2.82229577e+26,
6.48017418e-10, 6.37359499e-10])
readable['AB'][0:30]
array([0.00488466, 0.0047653 , 0.00466482, 0.00459077, 0.00451578,
0.00444223, 0.00436925, 0.00426781, 0.00415329, 0.00407281,
0.0040019 , 0.00388978, 0.00375876, 0.00363817, 0.00352651,
0.00342261, 0.00332389, 0.00322775, 0.00312658, 0.00301312,
0.00290017, 0.00279764, 0.00268841, 0.0025628 , 0.00243541,
0.00232032, 0.00221839, 0.00211131, 0.00198241, 0.00184446])
"""
The files can be found here: https://drive.google.com/drive/folders/1tTz6-GYi1mNkF85JtXe4k_GY7ffpiY6_?usp=sharing
Note: The files are from two different measurements which were ran consecutively. All measurement during the 'unreadable' measurement are unreadable. All measurements during the 'readable' measurement are readable.
Any idea what the problem could be? I initially thought that the file might be corrupted. However, the OPUS viewer is able to extract the correct information.
I've got the same issue recently. The following fixed it: PR https://github.com/qedsoftware/brukeropusreader/pull/21
Below is how your files are read with spectrochempy (which uses brukeropusreader), with this fix.
import spectrochempy as scp
D1 = scp.read("unreadable.0000")
D2 = scp.read("readable.0006")
D = scp.stack(D1, D2)
ax = D.plot()
ax.set_ylim(-0.02, 0.05)
scp.show()
have you guys been thinking of reading the magic bytes with spectral type numbers and offsets in the header? This would probably universally fix such problems as above.
I have been thinking of working on that based on some earlier python package. So if you are interested on working jointly on some stuff, just let me know :-)
That would be great. I don't know much about the opus format -- this is why I use brukeropusreader -- , but will be glad to help if I can.
Best, Arnaud
Hi Arnaud You are on the right track with the referenced merge pull request. We have been working further on this in a new R reader as part of the spectral-cockpit collective. https://github.com/spectral-cockpit/opusreader2/blob/e503ce3ae51155a77a1f94bae8eff9af1092fae4/R/read_header.R#L33 There is for example atmospheric correction routines done by Bruker software. These algorithms work for water and CO2 bands, and can selectively done. If done, there are both raw reflectance/absorbance spectra and atmospherically corrected ones. Interestingly, we found a new offset +3 that for each data block in the header (see referenced line above). This is either 0 or 64 when read as unsigned integer. :-) so what does this mean: we strongly believe that 64 stands for "raw" (relatively untreated) measurements, and 0 stands for macro processing (there is lots of macro language etc. reporting).
We for know call this bit of information in the header additional_type. btw, we get almost all parameters know with some more tweaks: https://github.com/spectral-cockpit/opusreader2/blob/main/R/get_nice_parameter_name.R
Hi Philipp,
Thank you very much for sharing.
On our side we have fixed an error when using brukeropusreader with files containing additional informations on the 'raw' AB spectrum (single channel spectra, interferograms, and acquisition parameters of the sample and reference).
our fix is here: https://github.com/spectrochempy/brukeropusreader/commit/c03aee303d3ffd73778e172bc189db68e961d2a8
The qed/brukerpousreader repository has not been updated since a while now. If this persists, and if others are interested, I could work on a python implementation of opusreader2 :-).
Best regards Arnaud
Hi Arnaud,
That is great to hear. With your latest commit we will be able to get the phase spectrum, which we will happily implement soon :-)
I would the idea of a double language implementation, so we can actually jointly work on mapping new block types etc. We have been recently added support for a specific Bruker Tango NIR device, @ThomasKnecht has tweaked another special block to parse ( https://github.com/spectral-cockpit/opusreader2/pull/34 ), plus read_opus() now has an informative error when there is files that fail to be parsed. Typically, this is because we do not (yet) have the full list of header mappings.
It really looks like qed.ai is not actively maintaining the package anymore. Maybe a quick email can resolve? otherwise really like the idea of opusreader2 in python!
Cheers, Philipp
Great to see that there is some active development still going on! Do you have any plans on releasing an updated opusreader package on pip or conda in the near future? @philipp-baumann @atravert
I recently published a new package that can read OPUS files. It is able to read the "unreadable.0000" file (absorbance, reference, and sample spectra as well as standard metadata). However, there are two mysterious blocks in this file that do not have any block type information stored. I'm guessing this has to do with the integral analysis. If you need to extract that data, it will take some new logic to extract those undefined data blocks. Below is the file block directory as read by my package:
========================================================================================================================
c:\Users\duranjm\Desktop\examples\unreadable.0000
Version: 920622.0 Directory start: 24 Max Blocks: 40 Num Blocks: 18
========================================================================================================================
Directory
Block Type Size (bytes) Start (bytes) Friendly Name
(0, 0, 0, 13, 0, 0) 480 24 Directory
(0, 0, 6, 0, 0, 0) 232 504 Optical Parameters
(0, 0, 4, 0, 0, 0) 116 736 Fourier Transform Parameters
(0, 0, 3, 0, 0, 0) 188 852 Acquisition Parameters
(0, 0, 10, 0, 0, 0) 284 1040 Sample Origin Parameters
(3, 1, 0, 1, 0, 0) 4876 1324 Sample Spectrum
(0, 0, 0, 0, 0, 0) 12 6200
(3, 3, 0, 4, 0, 0) 4876 6312 Absorbance
(0, 0, 0, 0, 0, 0) 112 11188
(0, 2, 2, 0, 0, 0) 552 11300 Reference Instrument Status Parameters
(0, 2, 6, 0, 0, 0) 232 11852 Reference Optical Parameters
(3, 2, 0, 1, 0, 0) 4876 12084 Reference Spectrum
(3, 2, 1, 1, 0, 0) 176 16960 Reference Data Status Parameters Spectrum
(3, 3, 0, 4, 0, 0) 2564 17136 Absorbance
(3, 3, 1, 4, 0, 0) 176 19968 Data Status Parameters Absorbance
(3, 1, 1, 1, 0, 0) 176 20144 Sample Data Status Parameters Spectrum
(0, 0, 2, 0, 0, 0) 552 20320 Instrument Status Parameters
(0, 0, 0, 0, 0, 5) 8040 20872 File Log
The two mysterious blocks have a block type of (0, 0, 0, 0, 0, 0). At first glance at the raw bytes in these undefined blocks, I can see metadata stored in these blocks, but also likely a data array. If you need to extract the info from these blocks as well (and not just the absorption and standard metadata blocks), you can open an issue in my package:
https://github.com/joshduran/brukeropus
