texera
texera copied to clipboard
Texera
Create OperatorExecutor on Worker side
Our engine creates an OperatorExecutor upon the creation of a worker actor. However, the OperatorExecutor is first created by calling a (Int) => IOperatorExecutor lambda function on the controller, then serialized and passed to the worker side. This will cause unnecessary memory allocation. In the worst case, OperatorExecutor may establish some connections that cannot be serialized (this ideally should be done inside open(), but we are not restricting developers to do that in the constructor). So I propose we pass a () => IOperatorExecutor lambda to the worker's constructor, then let the worker execute it. So the OperatorExecutor will be created on worker side.
I noticed for some operators, we passed a java lambda function as (Int) => IOperatorExecutor. Although scala will convert it to a scala lambda function, it won't be marked as Serializable. So the akka serializer will complain. For now, I didn't find any good way to solve this issue.
