Improved Export/Download support

[sk1p]

This is a continuation of #665, related to discussion in tickets #46, #168, #328

Now that the basic functionality is there, we can get first feedback from users; then it can be further improved:

• [ ] Support for CSV/Text
• [ ] Support for hspy
• [ ] Support for more formats?
• [ ] Better filenames, maybe including name from the GUI?
• [ ] Support for writing analysis parameters into the exported file, to support reproducing the results
• [ ] Find out: can other programs properly open multi-page TIFF documents? DM, HyperSpy, anything else?

Known issues

• [ ] Image formats (currently TIFF) don't support complex data
• [ ] Code quality in the GUI according to code climate can be improved, although most of the issues are not new...
• [ ] If an analysis decides that it is a good idea to put channels of different dimensions into a single AnalysisResultSet, at least the TIFF writer will crash
• [x] CoM export is broken: #723

[sk1p]

HDF5 export of CoM is broken:

HTTPServerRequest(protocol='http', host='localhost:3000', method='GET', uri='/api/compoundAnalyses/88354ca5-bba4-43ca-b026-a90f5aa91136/analyses/ebac42fd-180a-41c1-bdb9-e641215fdf1d/download/HDF5/', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/home/alex/.virtualenvs/libertem/lib/python3.7/site-packages/tornado/web.py", line 1699, in _execute
    result = await result
  File "/home/alex/source/LiberTEM/src/libertem/web/analysis.py", line 94, in get
    result_formatter.serialize_to_buffer(buf)
  File "/home/alex/source/LiberTEM/src/libertem/io/writers/results/formats.py", line 19, in serialize_to_buffer
    f[k] = self._result_set[k]
  File "/home/alex/.virtualenvs/libertem/lib/python3.7/site-packages/h5py/_hl/group.py", line 387, in __setitem__
    ds = self.create_dataset(None, data=obj, dtype=base.guess_dtype(obj))
  File "/home/alex/.virtualenvs/libertem/lib/python3.7/site-packages/h5py/_hl/group.py", line 136, in create_dataset
    dsid = dataset.make_new_dset(self, shape, dtype, data, **kwds)
  File "/home/alex/.virtualenvs/libertem/lib/python3.7/site-packages/h5py/_hl/dataset.py", line 83, in make_new_dset
    else base.guess_dtype(data)))
  File "/home/alex/.virtualenvs/libertem/lib/python3.7/site-packages/numpy/core/_asarray.py", line 85, in asarray
    return array(a, dtype, copy=False, order=order)
ValueError: object __array__ method not producing an array

Reason: .raw_data is a tuple, and that's what is returned as the __array__. Maybe need to exclude this from the export - users can still get the x and y components as separate layers.

[sk1p]

Note about multi-layered tiffs: they can be opened by Fiji, but are otherwise not widely supported. Float TIFF in general has bad support in normal image editors, same deal with multi-page/layer support. Might need an option to download channels separately, or as a zip of tiffs.

[matkraj]

@sk1p Download function is great! On my system any CoM result (x,y,RGB) goes into

500: Internal Server Error

and in console:

tornado.application - ERROR - Uncaught exception GET /api/compoundAnalyses/bb9adb50-1b7d-498d-830c-bc06cbe183e9/analyses/32c1c272-4d22-4435-b32a-f8e9698d4d52/download/TIFF/ (127.0.0.1)
HTTPServerRequest(protocol='http', host='localhost:9000', method='GET', uri='/api/compoundAnalyses/bb9adb50-1b7d-498d-830c-bc06cbe183e9/analyses/32c1c272-4d22-4435-b32a-f8e9698d4d52/download/TIFF/', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/home/mk/anaconda3/envs/libertem/lib/python3.7/site-packages/tornado/web.py", line 1703, in _execute
    result = await result
  File "/home/mk/APPS/LiberTEM/src/libertem/web/analysis.py", line 94, in get
    result_formatter.serialize_to_buffer(buf)
  File "/home/mk/APPS/LiberTEM/src/libertem/io/writers/results/formats.py", line 81, in serialize_to_buffer
    first_image = next(images)
  File "/home/mk/APPS/LiberTEM/src/libertem/io/writers/results/formats.py", line 76, in get_channel_images
    result = np.array(self._result_set[k]).astype(np.float32)
ValueError: object __array__ method not producing an array

[sk1p]

@matkraj thanks for the feedback and for trying out the new feature! Yeah, I still have to think about how to best support downloading CoM results. Would it work for you if the x and y components were available as separate "layers"? I also thought about returning complex values, but that won't work well for both tiff and raw export, as far as I can tell...

[matkraj]

@sk1p I would prefer multiframe TIFFs -> I just tested a 2 layer 32 bit TIFF and they are supported by GIMP (v2.10 up to date version at least, only trouble is that it does not stretch the contrast out of the box, Colors > Auto > Stretch contrast needs to be used). I guess complex images can be separated into 2 frames and imported like that.

[uellue]

Note about multi-layered tiffs: the can be opened by Fiji, but are otherwise not widely supported. Float TIFF in general has bad support in normal image editors, same deal with multi-page/layer support. Might need an option to download channels separately, or as a zip of tiffs.

A zip with layers as individual files sounds like a good idea!

[bdalevin]

I'm unable to save COM images in any format. When I try to do so, the server GUI becomes a blank page with the words "500: Internal Server Error" printed on the screen.

[uellue]

Temporarily assigned to 0.5 release to fix CoM 500 error in download. CC @sk1p

[sayandip18]

• [ ] Support for hspy

Is this the one? In that case, the link to the project is broken. It is described as a "python wrapper around the Citrix GoToWebinar REST API" in the description section.

However, the author does have a project named hspy which is described as "Django Middlewares to aide your HubSpot Marketplace development efforts".

The author has another project named gtwpy which is described as "python wrapper around the Citrix GoToWebinar REST API" (same as the one for hspy in PyPI).

I guess gtwpy is the one we need here.

[sk1p]

I guess gtwpy is the one we need here.

@sayandip18 not related to gotowebinar, this is about the file format of hyperspy