writing SpecFile possible?

[kklmn]

Hello,

Looking at silx.io: Input-output, I see a module specfile for reading SpecFiles and a top-level function save1D() that can write a two-column SpecFile. Nothing about writing a multi-column SpecFile. Is this possible in silx?

What I want at the end: I want to read our Sardana generated dat file, get the last entry, insert a few columns together with column headers and a few comment lines into the entry header and save it in a separate dat file.

[vasole]

I guess #3199 allows it

[kklmn]

There was this comment https://github.com/silx-kit/silx/pull/3199#issuecomment-697525885 @kif but no solution, as I see it. In silx/io/utils.py, savespec() still says "The curve is saved as a scan with two data columns."

[maurov]

[OT] @kklmn if you need a quick&dirty solution you can use this simple code.

[kklmn]

Thank you @maurov! It looks good!

[kklmn]

@kif I've now tried silx.io.utils.savespec() and it almost works! A few issues:

• Adapt the docstrings.
• If y is 2D, ylabel can't remain with the default value "Y". In this case, I suggest to do before the two assertions:

    if y_array.ndim == 2:
        if ylabel == "Y":
            ylabel = ["Y{0}".format(i+1) for i in range(y_array.shape[0])]

• There is a print() in the body of savespec() which I believe should be removed.
• (most important) Please insert two optional parameters: entry_header and entry_footer and generate the "#..." lines only when entry_header is not given.

My intermediate objective is to copy this model entry, please note the footer under the data:

#S 6755 dscan mono1_x2pit -20.0 20.0 60 0.1
#U root
#D Thu Dec 12 08:26:21 2019
#C Acquisition started at Thu Dec 12 08:26:21 2019
#N 8
#L Pt_No  mono1_x2pit  albaem01_ti  albaem01_ch1  albaem01_ch2  albaem01_ch3  albaem01_ch4  dt
0 3978.425 nan nan nan nan nan 0.306545972824
#C Acquisition ended at Thu Dec 12 08:26:33 2019

In the present version of silx.io.utils.savespec() I can copy the data (nans convert to 0, which is acceptable for me) and the column headers but cannot copy the rest. My test script for this:

from silx.io.utils import savespec
from silx.io.specfile import SpecFile


def read_entry(fname, entry='last'):
    sf = SpecFile(fname)
    if entry == 'last':
        entry = sf[-1]
    else:
        entry = sf["{0}.1".format(entry)]
    return entry


def write_entry(fname, data):
    x = data.data[0]
    ys = data.data[1:]
    xlabel = data.labels[0]
    ylabels = data.labels[1:]
    savespec(fname, x, ys, xlabel, ylabels,
             write_file_header=False, close_file=True)


if __name__ == "__main__":
    data = read_entry('20191212_CuO_HERFD.dat')
    write_entry('testOut.dat', data)

Thank you!

[vasole]

@kklmn

Call that file whatever you want but that is not a specfile.

SPEC never supported writing anything else than comments and numbers in data block.

To have text (and nan is text) is not SpecFile.

[kklmn]

I don't care about nans unless they crash my software. So conversion to 0 or very big number are both ok. I also don't mind if it is named spec or ceps. What is good though is that silx can help me, and it is almost there. :)

[kif]

Hi Konstantin,

I don't mind much about specfiles. I have been maintaining FabIO for a decade but all the support for tools from the last millennium is getting me bored. For sure parser to read and write specfiles (replace spec with the tool you wish) is useful, at least to you. As I am not interested in spec, and that ESRF (my employer) is moving away from it, I would suggest you to create a pull request with the modification you are suggesting. It is unlikely I will review it as I hardly know spec but we should find the time to review your PR and help you. Cheers,

Jerome

[vasole]

With current implementation your program will not crash, but the lines containing nans will be completely ignored.

[vasole]

@kif

Please, abstain from using other names than the user name.

It is up to the GitHub user to decide the identity associated to the user name and the possibility to change it.

[kklmn]

With current implementation your program will not crash, but the lines containing nans will be completely ignored.

No, they are not ignored: silx.io.specfile.SpecFile() reads nans as zeros, which is fine for me because it doesn't crash.

[vasole]

I have just tried using the specfile library under windows and gives an error and ignores the line.

Can the implementation of atof("nan") be different among platforms?

[kklmn]

mine is also Windows, Python 3.8.3, today's silx from GH.

[vasole]

The behavior I observe on the same platform is that lines are ignored.

#F C:/GIT/REFERENCE\test.dat
#D Mon Oct  5 08:09:57 2020

#S 1 Steel.spe 1.1.1.1
#D Mon Oct  5 08:09:57 2020
#N 3
#L channel  counts  energy
0  0  0
1  9  1
2  5  2
3  5  3
4  5  4
5  8  5
6  8  6
7  7  7
8  7  8
9  1  9
10  5  10
11  14  11
12  9  12
13  3  13
14  8  14
15  7  15
16  7  16
17  7  17
18  7  18
19  2  19
20  6  20
21  4  21
22  9  22
23  1  23
24  1  24
25  3  25
26  4  26
27  3  27
28  2  28
29  2  29
30  4  nan
31  1  nan
32  3  nan
33  0  nan
34  1  34
35  3  35
36  2  36
37  4  37
38  1  38
39  5  39
40  4  40

[kklmn]

@vasole When I try your test.dat, the nans are indeed ignored, but with mine test data, the one with "#S 6755" above, I get zeroes. Please try it with this:

    entry = SpecFile('testShort.dat')[-1]
    print(entry.data)

[vasole]

It seems the behavior is different between nan being in the middle or at the end of the line. In the middle of the line nan and inf become zeros. I do not know if that is the expected behavior of atof in all platforms. Under windows, it is:

Each function returns the double value produced by interpreting the input characters as a number. The return value is 0.0 if the input cannot be converted to a value of that type.

[vasole]

The previous does not seem to be the proper behavior but Microsoft one.

https://en.cppreference.com/w/cpp/string/byte/atof

Perhaps the code should be compiled as C++ and not as C????