TileDB icon indicating copy to clipboard operation
TileDB copied to clipboard

Lums/sc 19401/general node function node

Open lums658 opened this issue 2 years ago • 1 comments

This PR includes the following:

  • Much improved explanation and proofs of correctness for the task graph finite state machine, including for three-stage (aka buffered). The files for the documentation have been moved to the subdirectory state_machine/doc
  • Implemented unit_proof.cc to check the predicates in the proof outline contained in fsm2.md.
  • Porting and benchmarking of the sieve example program to use task graph nodes. The current example uses the "bountiful" scheduler (every node gets its own thread). Even with the overhead of system threads and synchronization, this sieve implementation is only about 10%-20% slower than the TBB implementation.
  • Improvements to simple nodes, and creation of a file simple.h for their implementations. A simple node is one whose enclosed function creates a single output for every input.
  • Initial implementation of a general node. Currently, it supports multiple inputs and multiple outputs, as well as specialization for no inputs or no outputs (i.e., to emulate producer or consumer nodes, respectively). The general node includes scaffolding for a Duff's device scheduler, but this will likely change as real schedulers are implemented and use cases for stateful functions are incorporated.
  • Development of generator functions prng and Injector. The prng generator currently uses random number generators from the standard library. The Injector allows input to be sent into the graph via its member function try_put, put, and stop. Full support for a truly non-blocking try_put will require fairly significant changes to the entire mover-policy-state machine stack -- and will require its own story.
  • Implementation of a primitive user-defined node that can generate one output from multiple inputs on the same input port. The node is tested in the file state_machine/sched (though it may really belong under nodes/test or even execution/test). This example uses both a "bountiful" scheduler as well as a "stingy" scheduler (based on a fixed-size thread pool). Both schedulers use Duff's device to interact with the node.
  • Initial implementation of an alternative finite state machine (fxm) to experiment with alternative scheduling for user-defined stateful nodes. Certain limitations, that once addressed, obviate the need for this state machine relative to the existing state machine (fsm). As result fxm will likely be removed in a future PR. The unit test for fxm does not pass.
  • Initial implementation and demonstration of much more compact, type and value parameterized, testing. The unit tests under dag will be refactored in a future PR.
  • Implementation of Source initiated stopping for task graph, using stop_source from C++20 jthreads. The source for threads is currently under dag/execution. Primary testing for stopping in state_machine/test.

All unit tests (except for unit_fxm) compile and pass with no failures.

TYPE: FEATURE DESC: PR for sc-19401 -- general node

lums658 avatar Sep 11 '22 21:09 lums658

This pull request has been linked to Shortcut Story #19401: General Node (Function Node).