blazingsql icon indicating copy to clipboard operation
blazingsql copied to clipboard

Finalize BlazingContext properly on exit

Open aucahuasi opened this issue 3 years ago • 0 comments

For the C++ side:

  • If we want to finalize all the engine properly we need to implement first many parts: thread interruptions, free memory from incomplete threads, notify every ral node to shutdown the processing, etc.
  • The threads/futures in c++ cannot be cancel by itself we need to program that logic: We should implement some sort of interruption logic for the kernel threads/futures and then clean its memory. For now we are using getExecuteGraphResult for the finalize caller and this will ensure to free all the memory

For the python side:

  • We need to free the memory from all workers invoking the finalizer. We cannot just send the finalizeCaller on each worker in the BlazingContext.del because python garbage collector already deleted some objects related with dask, so the tornado loop from dask doesn't exists there.

aucahuasi avatar Mar 02 '21 22:03 aucahuasi