Auspex icon indicating copy to clipboard operation
Auspex copied to clipboard

WIP: Feature/self aware filters

Open grahamrow opened this issue 3 years ago • 0 comments

Yikes.

Trying to solve a few problems here:

  1. Improved pipeline stability - @martin-gustafsson identified a lot of sketchy behavior of the pipeline. I moved to having all of the filer objects sharing Exit and now Panic events that the filter set upon Exceptions and the experiment periodically checks to see if it should bail. I have no real-world stability improvements to claim.
  2. Move to methods running actual filter code - using class methods allows Processes to be started without any explicit dependence on instance variables. Verbosity increases here since we now have to explicitly pass lots of important constructs, but in principle it is now much easier to write a fast filter in some external language. Currently configuration parameters live in a dictionary, but this could be spit out in ctypes struct later on.
  3. Finally allow things to work using spawn - this is a disaster as references to shared objects such as Queues and RawArrays are not properly synchronized without a multiprocessing Manager entity. Adding this still took a TON of hacks to fix numpy RawArray mappers, return values from IO buffers, etc. etc. etc. Using vanilla multiprocessing is

Still need to convert the following to using class methods:

  • [ ] Alazar listener
  • [ ] Filters other than averager and buffer

grahamrow avatar Apr 12 '21 17:04 grahamrow