blazingsql
blazingsql copied to clipboard
Finalize BlazingContext properly on exit
For the C++ side:
- If we want to finalize all the engine properly we need to implement first many parts: thread interruptions, free memory from incomplete threads, notify every ral node to shutdown the processing, etc.
- The threads/futures in c++ cannot be cancel by itself we need to program that logic: We should implement some sort of interruption logic for the kernel threads/futures and then clean its memory. For now we are using getExecuteGraphResult for the finalize caller and this will ensure to free all the memory
For the python side:
- We need to free the memory from all workers invoking the finalizer. We cannot just send the finalizeCaller on each worker in the BlazingContext.del because python garbage collector already deleted some objects related with dask, so the tornado loop from dask doesn't exists there.