Cross-actor, shm, (tick) ring buffer

[goodboy]

trafficstars

As part of the fsp subsystem design we're likely going to want to implement some shared memory systems for low latency multi-time frame calcs. I've started a couple design ideas about this in #98 but I do think for sure part of this system will require a fast ring buffer for numpy data:

Here's a starter list of projects to try after some very brief searching

---

committed to long ago

• 2016: ringbuffer it's multi-reader/writer, multi-proc aware and has an example with numpy
• 2015: pyringbuf a c-ext implementing a character buffer

---

committed recently

• 2020: pyring pure python impl and has a multi-proc c-types example
• 2020: ringbuf looks to be built for real-time audio but is single producer/consumer (which is actually fine for our use case afaik) but is built as a cython wrapper around boost (ala C++) libs; numpy example looks good
• 2021: redo of ringbuf in rust targeting some missing features with ctypes wrapping/integration for python
• 2020: circular_buffer_numpy seems kinda like a simple wrapper around an array with "pointer" index references kinda like what i've rolled many times before.

---

futher resources

• a scipy mailing list discussion from 09'
• comparison of implementing a circular buffer using 3 different data structures
• a random walk plot using none other then the great pyqtgraph and a numpy using np.roll().

It may just end up we take a lang from #106 and implement one or try out some designs from above on the new SharedMemory type which has an example using a numpy array. An example wrapper for an older version of this from the scipy cookbooks is here.

---

disruptor style

The "lurkers" said LMAX already did this best with their disruptor project. I think we can make a very cool variant of this with actors, numpy and numba:

• slickin paper
• pump vid
• there's more resources on the disruptor link :surfer:
• data_pipe is another one to check out (though not sure it'll have numpy support at all).
  • Looks to imply it has some disruptor style examples.

---

As always, lurkers please chime.

[goodboy]

The "lurkers" said LMAX already did this best with their disruptor project:

• slickin paper
• pump vid
• there's more resources on the disruptor link :surfer:

I think we can make a very cool variant of this with actors, numpy and numba.

[goodboy]

data_pipe is another one to check out (though not sure it'll have numpy support at all).

Looks to imply it has some disruptor style examples.

[goodboy]

This definitely can tie in with #192 @guilledk