imexam over tunneled XPA

[SimonKrughoff]

I've had some success using DS9 running locally using ssh tunneled ports from a process running on a remote machine: e.g. QA on data produced by running on a shared cluster.

Are there any suggestions on best practices for doing this? It doesn't work out of the box, but I don't know how hard I should try to even attempt such a thing.

Is the answer just "Use ginga"?

[sosey]

I think the tunneled connection to DS9 should work fine, just make sure you're always using the inet connection (which is the default in imexam anyways). I'm assuming you're working from an xterm that was pulled through the tunnel and starting up ipython/imexam/ds9 from there? Otherwise imexam will be looking for the local ds9 address.

when you say it doesn't work out of the box...are you referring to imexam functionality or just ds9 itself? I just ran a quick test for myself, tunneled across the country onto a really slow connection, it took on the order of minutes to display, but I was able to connect to ds9. Make sure when you are using ssh to call into your remote machine you have x11 forwarding on with ssh -XY <addresss>

Let me know how it goes!

[SimonKrughoff]

I am not forwarding X, I'm actually forwarding the XPA calls so that the X rendering is done on my local machine.

To be more concrete, I have the following in my .ssh/config on the remote system:

Host local_machine
    Hostname [local machine name]
    User [username]
    LocalForward 22222 localhost:22222
    LocalForward 22223 localhost:22223
    ForwardAgent yes
    ForwardX11 yes
    ForwardX11Trusted yes

On the remote machine, I say ssh local_machine. On both my local machine and the remote one, I do export XPA_PORT="DS9:ds9 22222 22223". This makes ds9 listen on the ports that are being tunneled.

Finally, I fire up ds9 on my local machine. On the remote machine I issue imexam commands. If I already have an image displayed in the frame, I can make things like scale and zoom work, but load_fits and imexam don't.

Any thoughts would be much appreciated.

[sosey]

got it. I'm traveling until tuesday evening and will give a look at this wednesday. Thanks!

[sosey]

imexam speaks to the XPA directly, so I can see how working with just DS9 commands that have been directly aliased from the XPA works. load_fits and imexam are package specific functions that try and make communicating with different image visualizers follow the same interface.

Are you able to get a copy of the data that is displayed, as a numpy array, on the remote machine, using viewer.get_data()?

imexam is going to load the python environment on your remote, including the matplotlib interface, so I'm not sure the plots generated by imexam()will be displayed locally without x11 forwarding turned on for the whole connection

I should have asked before, but do you get any error printed to the terminal screen or does nothing happen at all when you try and load_fits()? If so, send it along?

imexam() is actually getting the data array from the local file itself, instead of transferring the data through the XPA. This allows for faster access. So when the function is called it looks for the data that is displayed, loads it into a numpy array and goes about it's business. As long as the full path to the dataset can be found on the remote machine I think this should work.

What are the OS of the local and server machines?

This is not my area of expertise, but you may also need to update the sshd_config for the ssh server process to allow it to port forward the plots and such.

[SimonKrughoff]

There are a few steps to get to the point of trying out imexam.

• First is making the connection. The ds9 connector currently assumes that I'm running an xpans. In the case that the code is running remotely to the ds9 instance, this isn't true. When I do viewer = imexam.connect('localhost:22222') the error is:

No active sessions registered

DS9 target: localhost:22222 doesn't exist.
[]
ValueError: Please choose an existing target.

If I just set self._xpa_name to target in the ds9 constructor, I get a connection.

• Next I try to send the image doing: viewer.load_fits('/home/krughoff/DATA/iacs01t4q_flt.fits') This results in: XpaException: Unknown XPA Error : XPAGet returned 0! This turns out to be because the XPA command is sending a request to load the file /home/krughoff/DATA/iacs01t4q_flt.fits, which, of course, doesn't exist on the filesystem where ds9 is running. I believe the solution is to send the whole binary string as a part of the XPA request. To test this, I copied the file to an identical location on the machine running ds9 and indeed load_fits now works. I also tried viewer.imexam(), and at least some of the functionality works with plots showing up inline in the notebook. This is as far as I've gotten.

I think to do what I'm looking for, at the very least there would need to be a way to ship the image over the wire to the ds9 client running remotely to the python code.

I would be happy to look into this more, if it's a direction this team is willing to explore, but I'm equally happy being told that mine is not a supported use case.

[pllim]

There's always Ginga...

[SimonKrughoff]

I'll try ginga. @pllim are you saying I should give up on ds9 in this context?

[sosey]

@pllim have you run the ginga gui with data tunneled from another machine?

[sosey]

@SimonKrughoff right, xpans is the nameserver and the code tries to validate that the connection named in target exists to speak to. I can try changing the logic to set the connection over the tunnel for you when it doesn't find the target locally.

For the data-send over the tunnel, we may need to add an extra parameter to the imexam.connect() specifying that the target is remote so that I can register that subsequent data data loads should use this command. I'll look at the above later today.

In the mean time, see if doing this helps. On your local server run : xpaset -p ds9 -acl localhost:22222 + to add that port to the access list. Then on your remote, setenv XPA_NSINET 'localhost:22222'.

[pllim]

I have seen a demo where Ginga is running on a machine and you can use it from a different machine. The demo was done in a Jupyter notebook interface. xref https://github.com/ejeschke/ginga/blob/master/ginga/examples/jupyter-notebook/Remote_control_of_a_Ginga_reference_viewer.ipynb

I believe the non-notebook GUI also has some remote connection feature but I haven't seen that one in action personally. Shall we ping Eric J?

[pllim]

If we can get this to work on Ginga instead of DS9, it would also benefit JWST in the near future. Just sayin'.

[SimonKrughoff]

If we can get this to work on Ginga instead of DS9, it would also benefit JWST in the near future. Just sayin'.

I'd be happy with either.

[sosey]

It's possible to do it with DS9, I just need to make sure I have all the options ironed out in imexam. I'm looking for a local system I can do testing with.

[SimonKrughoff]

As motivation for why I'm trying to do this, I have a bunch of data stored in the cloud on: e.g. Google Cloud Storage. I have processes that can run reduction/analysis routines on those data. I would like to then examine the data from within a container running in the Google Cloud (on GKE for example).

[sosey]

I'm with ya :) I hacked something rough together yesterday and then hosed my XQuartz :/ But I'm pretty sure this will be possible.