Consider optional integration with tower/tokio

[LegNeato]

For example, the adapter interface looks a lot like https://docs.rs/tokio-tower/latest/tokio_tower/ and it would be neat to use https://docs.rs/tokio-console/latest/tokio_console/ via https://github.com/tokio-rs/tracing to see what is going on in the circuits.

[LegNeato]

I see tracing is already covered in https://github.com/vmware/database-stream-processor/issues/243

[ryzhyk]

Thanks for the suggestion, @LegNeato! Can you elaborate a bit on the kind of tokio integration you think would be helpful? The current state of affairs is that DBSP is internally thread-based, but it can be used from tokio applications.

Not directly related to your question but since I brought this up ... the reason we use threads is to get the best performance by controlling concurrency, communication, and synchronization inside the circuit. This design also complicates things as we are starting to add support for persistence, and need to handle I/O-intensive workloads.

Be as it may, DBSP's datapath is thread-based, using a fixed pool of worker threads allocated on startup. But the control API used to build/destroy/pause/resume the circuit is runtime-agnostic and can be used from regular or tokio-based applications. In fact, we are building an HTTP API around the controller using actix.

As for monitoring the internals of the circuit, we have the beginnings of an event-based framework meant as a foundation for various tools for visualization, profiling, and tracing, but there's a lot more work we need to do to make it useful.