byzer-lang icon indicating copy to clipboard operation
byzer-lang copied to clipboard

[Enhancement] MLSQLRest should utilize executors to process http request and response

Open chncaesar opened this issue 2 years ago • 0 comments

MLSQLRest's code only runs in Driver, and executors stay idle. If users run Byzer-lang on K8S/Yarn/Standalone mode, processing power is limited to Driver. We should refactor code in load and save method, and partition the http paging request if possible, For instance, the auto increment paging strategy can be parallelised.

In our FLAG env, running MLSQLRest code on 16GB Local mode byzer-lang is much faster than 8GB driver + 2 ✖ 8GB executors K8S byzer-lang.

chncaesar avatar Apr 14 '22 12:04 chncaesar