Add a guide for loading data from Tiled into Igor direclty

[danielballan]

@EliotGann has got Igor reading data directly out of Tiled (without going through files). This would make a nice guide because:

• It's another nice general proof of principle that Tiled can interoperate with existing, non-Python programs.
• Some other users may want to do exactly this. @prjemian mentioned an APS scientist who has Tiled -> Igor working. I don't know if he's going through files.

[prjemian]

@jilavsky is doing direct URL access to tiled from IgorPro, not using file transfer. (He can comment further.)

[EliotGann]

I'm working on a little browser gui that loads the monitor streams - eventually this will be an import gui for QANT - my igor spectroscopy code. The problem was this was it is a large number of URL requests, which was lagging considerably. It will eventually be a github repository, but for your purposes, the trick is using a threadsafe fetch function like this

Threadsafe function /s fetch_string(string url,string num)
	
		
	URLRequest /z /time=1 url=url, method=get
	if(v_flag || !cmpstr(S_serverResponse,"Internal Server Error"))
		// try again - server seems to ocassionally hickup
		URLRequest /z /time=1 url=url, method=get
		if(v_flag || !cmpstr(S_serverResponse,"Internal Server Error"))
			return ""
		endif
	endif
	string output = S_serverResponse
	return output

end

and call it using multithread with a large text wave of URLs (not showing the code that compiles these URLs) - these are alternating from the data and timestamp series for each monitor. Then pulling out the strings - converting them to byte waves, and redimensioning them into floating points

multithread /NT=(numpnts(list_of_files)) stream_data = fetch_string(urls[p],num2str(p))
	string list_of_monitor_waves = ""
	variable len1, len2
	for(i=0;i<numpnts(urls);i+=2) 
		len1 = strlen(stream_data[i])
		len2 = strlen(stream_data[i+1])
		if(len1==len2 && len1>1 )
			setdatafolder homedf
			newdatafolder /o/s $cleanupname(uids[i],0)
			streambase = streambases[i]
			wave stream_wave = StringToUnsignedByteWave(stream_data[i]) // convert each string received into a bytewave
			wave time_wave = StringToUnsignedByteWave(stream_data[i+1])
			redimension /d /E=1 /n=(len1/8) stream_wave, time_wave // recast the byte waves into floats (double precision)
			concatenate /o {stream_wave,time_wave}, $cleanupName(streambase,0) // combine up the time and data into a single wave
			wave stream = $cleanupName(streambase,0)
			list_of_monitor_waves+= getwavesDataFolder(stream,2)+";"
		endif
	endfor

this loads 30 bluesky runs (with 840 monitors total) in ~4.5 seconds, or on average about 0.15 seconds per run. It runs a single run (28 monitors) in about 0.5 seconds, which seems to be the minimum url fetch time right now.

[danielballan]

Key points that may be of interest to others:

• It is possible to get C-encoded arrays directly into Igor (without touching disk).
• As expected, it's much faster to avoid going through a file on disk.
• As expected, binary C-encoded arrays are faster than CSV.
• As @EliotGann mentioned, latency is the main overhead, and parallelization helps with this. In the the future we should explore requesting multiple columns at a time, but we may need formats that support tabular data (and potentially heterogenous data types across columns) and Igor is thin on options there. HDF5 looks like the winning one.

[jilavsky]

Since I was mentioned... I am pulling json from web interface by building the web query string in Igor and doing URLrequest. I keep the json in memory and using json.xop (https://www.wavemetrics.com/node/20976) parse into Igor objects. I am currently plotting small scans and speed of getting and displaying those in Igor is not issue, yet.

[danielballan]

Good to hear. If the straightforward approach is fast enough, then it's definitely a winner!

[EliotGann]

Here is the initial implementation. It seems to be able to browse pretty much any tiled data, although it was built for my beamline, I've tried to keep it general. https://github.com/NSLS-II-SST/TiledRSoXS

[danielballan]

Awesome!

[danielballan]

@jilavsky I just released Tiled v0.1.0a66, which changes the URLs for xarrays. I am not sure whether that will affect your use case, but I thought I'd give you a heads up.

We try not to make disruptive changes, but at this early stage we will do so if we can get a large simplification in exchange. :-)

If you upgrade and find that your Igor code breaks, I'd be happy to help you sort out what to change on your side, as I am doing for @EliotGann presently.

[jilavsky]

@danielballan : I have little expectations of stability of anything with release versions ~ 0.1.0... Keep going and do not worry, simple faster interface is important. FYI: My USAXS data reduction code development version is now distributed with Tiled browser code which works with specific version of Tiled installed on our instrument and specific scans we need to deal with. Thanks to our control of both Tiled version and our data reduction code, it is easy to keep in sync.

[danielballan]

Thanks for the encouragement @jilavsky. Sounds good.

[danielballan]

@jilavsky Just checking in on this to keep an eye on the state of the Tiled "ecosystem". Is this still being used? How frequently/actively is it used?

There may be more Igor developments in the near future, which is why I was curious to check in on the state of yours.

[jilavsky]

I have been comissioning new hardware for last 3 months and my Igor code has not been used. We do not have Tiled server running at this time, nothing to serve. Once we recommission, in month or so, I will need to get back to this and restart using Tiled to give easy access to our instrument alignment data.