TileDB icon indicating copy to clipboard operation
TileDB copied to clipboard

Adds functionality for creating and executing a task graph

Open stavrospapadopoulos opened this issue 4 years ago • 0 comments

This PR adds functionality for creating and executing a task graph. The functionality allows for tasks to dynamically generate sub task graphs while the graph is being executed.

All internal algorithms (reads, writes and the upcoming advanced computations) will be refactored to use the task graph functionality. The task graph stores logs and statistics that will make it very easy to fix both bugs and performance issues.

Minimal example:

// Create task graph
auto task_graph = tdb_make_shared(TaskGraph);
auto task0 = task_graph->emplace([]() {return Status::Ok();}, "0");
auto task1 = task_graph->emplace([]() {return Status::Ok();}, "1");
auto task2 = task_graph->emplace([]() {return Status::Ok();}, "2");
task_graph->succeeds(task2, task0, task1);
  
// Execute the task graph
ThreadPool tp;
tp.init(std::thread::hardware_concurrency());
TaskGraphExecutor tge(&tp, task_graph);
tge.execute(); // This is non-blocking
tge.wait(); // This is blocking 

TYPE: FEATURE DESC: Adds functionality for creating and executing a task graph.

stavrospapadopoulos avatar Nov 06 '21 21:11 stavrospapadopoulos

Closing in favor of #3328 #3366 #3453 and others in the series.

ihnorton avatar Aug 16 '22 04:08 ihnorton