Dagger.jl
Dagger.jl copied to clipboard
Add Services/Actors API
Stateless tasks are nice from a scheduling point-of-view, since we can re-schedule and rearrange tasks in any valid order. However, stateful computation allows for memory reuse, data locality, gated resource access, and often is just a better model for a given computation or operation.
We can add a "Services" API which is focused around the creation, management, and deletion of persistent tasks executing on a fixed processor:
- A Dagger service, like a task, would execute in a task context.
- Services can be created by any task or service (and services can create tasks and services), and are accessed (for invoking or deleting) by a handle returned by the creation function. The handle will be serializable.
- A service can be created on a specific processor if needed, or otherwise left to the scheduler to decide.
- State would be stored within the function itself (which would be a callable struct), arguments would be passed as normal, and the result would be returned to the caller (which can be a task or service).
- The scheduler may choose to track the execution and resource usage of a service externally, and use that to control its execution priority, as well as control scheduling of other tasks/services onto the same processor.
- A service can be called from anywhere, so if the service wants to receive/send a reference to a non-serializable object, it must be wrapped in a Chunk.
- Some helpful services might be provided by Dagger, such as a service-name registry (allowing services to be associated with and located by a unique String).
- The service handle will be cached in the scheduler so that, in the event that a worker dies, its services will be automatically restarted and the cached handle will be reused.
- Advanced: Raw access to any service's function object will be available for any task scheduled on the same processor, to allow for building functionality on top of existing services.
Note: This is a first pass at specifying the requirements; I'm open to any changes to the above!