String concatenation with `//` doesn't work with `.fits` files and with `@` list processing

[dxwayne]

The error was first encountered with:

!ls -1 *fits > l.l

The l.l file could be called l.txt -- but Unix doesn't care about extensions and l.l is easy to type on an American keyboard.

So now I want to zero subtract a zeromaster.fits file from each original... and make a new file z_<basefilename> using the concatenation sequence of // for files in the list-file (@l.l):

zerocombine @l.l - zeromaster.fits z_//@l.l

but the at-cost (@) breaks this very old common usage pattern. The filename turns up with a Unicode sequence.

I tried with cl and it was broken there too, under pyraf like

cat > foo.cl
imstat z_//@l.l
<eof>

then cl < foo.cl the same error occurs.

Fix:

adding set LC_ALL=en_US.UTF-8 in loginuser.cl, to fix the problem at the user level.

Real question, given the legacy of cl and pyraf being strictly ASCII, should this be added to login.cl so the user does not have to figure out to make a ~/.iraf and come up with a loginuser.cl file?

[dxwayne]

I sit corrected. This did not fix the problem. How to use concatenation with list-files is still open for me.

[olebole]

Did you try this with using IRAFs ecl? Did it work correctly there?

Which IRAF version do you use?

What do you mean with

The filename turns up with a Unicode sequence.

Can you write the sequence you got?

[dxwayne]

Good suggestion, both cl and ecl do work inside an xterm. Thanks for looking into this.

This trick makes reducing things easy, been using it for years. At the end of the preliminary reductions it is not uncommon for f_d_z_c_t_ to get prepended for processes of trim, cr correction, zero and dark subtraction, and flat correction, all for the same l.l list of files without trips to vi or several lines of sed.

I just tested the concat operator // alone imstat a11030_FlatLU_0s_011206m2.fits works

But! Ah Ha! imstat t_//a11030_FlatLU_0s_011206m2.fits

stsci.tools.irafglobals.IrafError: Error running IRAF task imstatistics IRAF task terminated abnormally ERROR (1, "Image expansion/concatenation not yet supported.")

imstat t_//a11030_FlatLU_0s_011206m2.fits

Killing IRAF task `imstatistics' Traceback (innermost last): File "<CL script CL1>", line 1, in stsci.tools.irafglobals.IrafError: Error running IRAF task imstatistics IRAF task terminated abnormally ERROR (501, "segmentation violation")

Errors out before the "Image expansion/concatenation not yet supported." message can appear.

So, on another note: within pyraf, in splot, snap does not work but does from with cl and ecl. The snap thing seems to be related to tkinter. I did not try pyraf splot and snap with the PYTHONGRAPHICS set. I'm still working with that.

To be clear about pyraf, @l.l works where the concat and @ fail, the @ gets recognized as an illegal character.

Last night I pounded on this by hacking the .py files on my distro and changing the 'ascii' strings in relation to encoding. The programs still worked, but the problem persisted. I put everything back.

It dawns on me, since I just got up, that there was a switch in the parsing by pyraf to something Guido Rasmussen calls a PEG parser a few gens of pyraf ago -- this does a look ahead until failure. x//@ may have too far of a look-ahead.

On Sat, Sep 14, 2024 at 3:28 AM Ole Streicher @.***> wrote:

Did you try this with using IRAFs ecl? Did it work correctly there?

— Reply to this email directly, view it on GitHub https://github.com/iraf-community/iraf/issues/409, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACXLYL3WO4WCNUN7H7RTAPLZWP6UZAVCNFSM6AAAAABOGFNCR2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNJQHEZDSMBUGI . You are receiving this because you authored the thread.Message ID: @.***>

--

-- W

[olebole]

To have a minimal example using the file that comes with IRAF:

--> imcopy dev$pix  t_a11030_FlatLU_0s_011206m2.fits
dev$pix -> t_a11030_FlatLU_0s_011206m2.fits
--> imstat t_//a11030_FlatLU_0s_011206m2.fits
Killing IRAF task `imstatistics'

PANIC in `/usr/lib/iraf/bin/x_images.e': Write to IPC with no reader
Traceback (innermost last):
  File "<CL script CL1>", line 1, in <module>
pyraf.tools.irafglobals.IrafError: Error running IRAF task imstatistics
IRAF task terminated abnormally
ERROR (1, "Image expansion/concatenation not yet supported.")

However:

--> imstat t_//a11030_FlatLU_0s_011206m2
#               IMAGE      NPIX      MEAN    STDDEV       MIN       MAX
 t_a11030_FlatLU_0s_011206m2    262144     108.3     131.3       -1.    19936.

works (implicitly adding .fits).

But this all is part of imstatistics, not of pyraf. The same behaviour is in ecl:

ecl> imcopy dev$pix  t_a11030_FlatLU_0s_011206m2.fits
dev$pix -> t_a11030_FlatLU_0s_011206m2.fits
ecl> imstat t_//a11030_FlatLU_0s_011206m2.fits
ERROR: Image expansion/concatenation not yet supported.
ecl> imstat t_//a11030_FlatLU_0s_011206m2
#               IMAGE      NPIX      MEAN    STDDEV       MIN       MAX
 t_a11030_FlatLU_0s_011206m2    262144     108.3     131.3       -1.    19936.

[olebole]

Similar minimal example for @:

--> cat list.txt
a11030_FlatLU_0s_011206m2.fits
--> imstat t_//@list.txt
#               IMAGE      NPIX      MEAN    STDDEV       MIN       MAX
Error reading image t_��a11030_FlatLU_0s_011206m2.fits ...

and the same for ecl:

ecl> cat list.txt
a11030_FlatLU_0s_011206m2.fits
ecl> imstat t_//@list.txt
#               IMAGE      NPIX      MEAN    STDDEV       MIN       MAX
Error reading image t_�#a11030_FlatLU_0s_011206m2.fits ...

So it seems to me that this is a problem (or limitation) in the concatenation in IRAF itself, not in PyRAF.

[olebole]

Could you specify which IRAF version you used?

[dxwayne]

Hi,

I am now sure that I am running NOIRLabs version, as the MOTD is different from the latest stable Community edition I cloned yesterday. More to follow, I have commitments later today. --W

On Sun, Sep 15, 2024 at 4:22 AM Ole Streicher @.***> wrote:

Could you specify which IRAF version you used?

— Reply to this email directly, view it on GitHub https://github.com/iraf-community/iraf/issues/409, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACXLYL2ST2UES6OK66VSAFDZWVNUTAVCNFSM6AAAAABOGFNCR2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNJRGUYTQMRQGA . You are receiving this because you authored the thread.Message ID: @.***>

--

-- W

[dxwayne]

I just updated to the new IRAF via building and installing the current clone image. I had to do a sudo pip3 install pyraf to get pyraf its version is 2.2.2 -- the same as the head on github. The clone from github relies on old setup.py -- deprecated
so far I could not catch my Pypi python 3.12.5 up to date.

All reported issues above have persisted through to here. My OS is Ubuntu 22.04 LTS. The machine is an Intel I914000K (so no AMD issues).

I tried a simple concatenation at the command line with no use of a file-list (@) file: This blows up, but gets as far as a message that says the concat is not implemented. The full snippet is here:

imstat t_//a11000_CalHD51055_60s_20240811_095130m0.fits
Killing IRAF task `imstatistics'

PANIC in `/home/git/external/iraf-2.18/bin.linux64/x_images.e': Write to IPC with no reader
Traceback (innermost last):
  File "<CL script CL1>", line 1, in <module>
stsci.tools.irafglobals.IrafError: Error running IRAF task imstatistics
IRAF task terminated abnormally
ERROR (1, "Image expansion/concatenation not yet supported.")

This error is at:

./sys/imio/imt/imxpreproc.x:57:	    call error (0, "Image expansion/concatenation not yet supported.")

Now to see what that means.

Thanks for patience. I am working on coming up to speed with the build from source.

[olebole]

Did you know that IRAF and PyRAF are part of Ubuntu 22.04 and you can install them with sudo apt install iraf python3-pyraf? There is no need to compile it from source (most of the later changes are connected to the ability to build it with recent C compilers).

As I already wrote: the problem is not PyRAF, it is a problem with IRAF, and I am going to transfer the issue there. the direct // concatenation works if you don't use the .fits extension (that is automatically added), so a workaround here is

--> imstat t_//a11030_FlatLU_0s_011206m2
#               IMAGE      NPIX      MEAN    STDDEV       MIN       MAX
 t_a11030_FlatLU_0s_011206m2    262144     108.3     131.3       -1.    19936.

But the @ list processing doesn't work; also not when omitting the .fits extension.

Because the issue appears in both the NOIRLab and the community IRAF forks, it is most probably already present in 2.16.1 and therefore not a regression.

[olebole]

What pragmatically helps is to set use_new_imt to no.

[olebole]

Also reported in #237

[dxwayne]

In the past I used z_//@filelist ... Now z_@filelist... works the way I wanted to begin with! I consider this closed.