Matthew Bryan

Based on discussions surrounding #1170 , it might be useful to have a mechanism so that a user can unwrap the various LiberTEM data structures into their Python/Numpy/Dask equivalents in...

`PipelinedExecutor` shutdown errors (cloudpickle teardown)

3

I'm finding that `PipelinedExecutor` is failing to shutdown correctly (hangs when the interpreter exits) with the following MWE. Difficult to diagnose but the tracebacks after `KeyboardInterrupt` seem to imply that...

Frame-level progress bar updates (+ comms system to enable it)

6

Closes #1333. This modifies the progress bar to give more fine-grained progress updates when partitions are slow to compute: ``` Partitions 1(1)/8, Frames: 17%|███████ | 176/1024 [00:11

Multi-level progress bar for slow Datasets / UDFs

4

When running heavy UDFs or reading from slow data sources (e.g. over the network), the `ctx.run_udf` progress bar can be stuck for a long time with no feedback to the...

DM4 DataSet + Fortran Array reader

6

This adds a `DM4DataSet` class which is capable of detecting if the file is new-style C-ordering or old-style F/C-hybrid-ordering and reads it accordingly. The dataset has also been added to...

Plot interface cannot support line-profile datasets

1

Came across this when working with some user data: ``` File "C:\ProgramData\Miniconda3\envs\GMS_LIBERTEM\lib\site-packages\libertem\viz\base.py", line 158, in __init__ channel, eligible_channels ValueError: channel intensity not found or not plottable, have: [] ``` as...

UDF mode `process_frame_stack(stack)`

8

I seem to frequently encounter a situation where I am implementing a UDF which could quite easily be vectorized along a stack of whole frames. For example applying an FFT...

A consequence of the plotting logic contained in the following part of `_run_sync_wrap()` https://github.com/LiberTEM/LiberTEM/blob/ed7ab5c2e821aa84ea98cafd3491ad61f9b09b98/src/libertem/api.py#L1259-L1270 is that every live plot gets at least two updates, once with `force=True`, even if we...

We get a warning when running with `progress=True` in a notebook on a clean install. This originates from Tqdm because we don't require `ipywidgets` as part of a LiberTEM install....

Too many open files error while running slow tests

1

While running the slow tests on the server I get the following errors related to too many open files (I think probably processes), coming from distributed. ``` 2023-03-21 10:07:28,708 -...