dffml
dffml copied to clipboard
intel
df: System local resource management
We need some way for operations and networks to manage system local resources (memory, disk space, threads).
This may end up being a property of the Orchestrator, which should probably be passed to all operations and networks upon instantiation.
Ideally we'd have the ThreadPoolExecutor be a property of the resource manager, and then submit threads via a dedicated method, that way we can cap the number of running threads if desired.
See https://github.com/intel/dffml/discussions/1369#discussioncomment-2639935
subflow will be called one time for each new system context yielded by the prioritizer. Prioritizer is also used in orchestrator main loop for dynamic reprioritization or drop. When prioritizter is in main loop run_operations_for_ctx(). Contexts can be dynamically reprioritized and that will effect the operations running within the context, for example could lead to opimp cancelation or whole context cancelation. On dynamic reprioritization prioritizer will rerun strategic plans (same as with LTMs we have output operations reading from cached state) and gatekeeper then re-run prioritization and re-score with knowledge of what other contexts in the overall set are running. Therefore, the pritoritizer MUST be told which system contexts map to which context that will be created for the InputSet passed to the flow. Gatekeepers would be aware of the context and aware of the outputs of the stratgic plans and their values relative to other system contexts being executed in the overall network to determine optimal dynamic reprioritization of executions of system contexts within the overall system contexts this allows us to efficivly reward trains of thought on demand.