The Structural Dynamics of Flow: finally a formalized modular transport layer -> `tractor.ipc` 😎

[goodboy]

trafficstars

Although most of this work transpired on an alt (and now quite distributed) set of (sometimes-private) git hosting services, we figured getting some lurker hype going again on the project couldn't hurt.

This initial tractor.ipc effort was spearheaded by our very own @guilledk surrounding their work on a new linux-only high-performance impl of a shared-mem ring-buffer with native trio async blocking support/semantics using eventfd :sunglasses:

The idea is to get a more generic and performant, soft-real-time targeting, ShmList-like interface which does not require read/write event signalling over another IPC transport (like TCP as was the case with the current state of things in https://github.com/goodboy/tractor/pull/338).

That ringbuf code will now land later on top of this history since I've made a buncha of reworks and refinements to that original attempt at modularizing our IPC transport layering.

Much more details below.

---

The grand .ipc-sub-system summary

Detailed writeup coming later.. get your tickets at the box office.

Maybe i'll try to get an "AI" to draft this up?

---

Outstanding feat requests this relates-to/resolves

• [ ] https://github.com/goodboy/tractor/issues/109
• [ ] https://github.com/goodboy/tractor/issues/136

---

Idealistic todos before this lands

Gosh it'd sure be nice but I'd rather get UDS in sooner then later sin unit-tests if it comes to it.

• [x] factor out the Actor._stream_handler() method to a mod-level func either in .ipc._server or similar.

  • done in 42cf9e1

• [x] also move over all the Actor._peers: dict[tuple, Channel] (and related state) tracking to the IPCServer and change all dependent code to match.

  • done in 112ed27

• [ ] fill out the IPCEndpoint.peer_tpt tables (implicitly) by pre-hooking into a new (likely) IPCEndpoint (class) method which first registers every MsgTransport/Channel prior to processing and handover to the RPC layer.

  • [ ] ideally in a way that let's us granular-ly know which peers are on which transports and which .stream_handler_tn: Nurserys so we can eventually do fancy stuff like connection resetting, peer filtering, dynamic transport-proto swapping etc.

• [ ] make all the .__repr__()s for various runtime primitives look real nice (and correct) even when we use our little "sclang" (see the strange long comment i put in ._child.nest_from_op() XD) ascii-syntax for conveying various states!

  • [ ] Actor
  • [ ] IPCServer
  • [ ] IPCEndpoint
  • [ ] Channel
  • [ ] MsgTransport

• [ ] a (couple?) suite(s) of tests to audit,

  • [ ] the new .ipc._server subsys in unit-style-isolation
  • [ ] multiple-transports in tandem (eventually per actor) by first ensuring the server layer is correct before exposing upward via enable_transports: list[str] to runtime-booting entrypoints (open_nursery()/open_root_actor()).

[goodboy]

Lol well first thing is getting uv in CI i suppose 😂

[goodboy]

The IPC server subsys is fully factored in close-to-original form as of now 🥳

I'd still like to get the per-IPCEndpoint peer-Channel tracking going along with a collections.ChainMap view into all per-actor instances for a similar read-interface to what (was Actor._peers) is IPCServer._peers: dict[tuple, Channel].

So that would be one_ring_to_rule_them_all branch found on other remotes/git-servers (not sure if it's been pushed here yet).

[goodboy]

There changed base to main now that i just landed #338.

[goodboy]

Welp just force pushed after a bunch of rebasing to get in as much of the final impl minus testing (which will come in a follow up patchset).

[goodboy]

Ehyyo, we green in CI with uv 😎

[goodboy]

See #381 for .__repr__() formatting follow up.

[goodboy]

New badge links added to the readme.. though let's see if it flips green 🙏🏼

[goodboy]

Eyyyo***2, guess she's ready B)