Milan Klöwer

I believe this memory issue is something general we have to work on. Technically the algorithm should be almost allocation free as mentioned above (only a small counter array has...

The error originates from here ```python mutual_info = (p * np.ma.log(p / (pr * ps))).sum(axis=(-1, -2)) / np.log(base) ``` in https://github.com/observingClouds/xbitinfo/blob/0d8852b5b7f3493dcd8a8bc88ec0dd97feb90dff/xbitinfo/_py_bitinfo.py#L143-L152 which confuses me because while `a,b` should scale with...

Maybe related https://github.com/observingClouds/xbitinfo/blob/0d8852b5b7f3493dcd8a8bc88ec0dd97feb90dff/xbitinfo/_py_bitinfo.py#L128-L139 this seems to have an outer loop over the number of bits then an inner loop over all elements in the data. Which means that I suspect...

This issue sounds weirdly familiar, and indeed we discussed this already at the beginning of this year: https://github.com/observingClouds/xbitinfo/pull/156#issuecomment-1424618296

We tried possibility 2 from the graph above, which is trickier than we thought, because the "somehow" isn't well constrained and so it doesn't generalise as easily. We haven't tried...

I think we're talking in different directions here. Dequantizing is what can cause the artificial information in the first place?

Unfortunately, your bitwise information has some artificial information included, which becomes evident in the last mantissa bits: 1) u (or v) I believe this data was somehow lossily compressed before....

> My rule of thumb that nearly always works: Take 99% and then two additional bits to be on the safe side Haha, sure, but for @coxipi's precipitation example +2...

Just a note on *artificial information* (what we call it in the paper): - it can plateau, as you say, in which case you may want to define some reasonably...

I realised that in _2. analyse the bitinformation..._ is missing that you should also apply the rounding before subtracting the initial offset again!