byzer-lang
byzer-lang copied to clipboard
[Enhancement] MLSQLRest should utilize executors to process http request and response
MLSQLRest's code only runs in Driver, and executors stay idle. If users run Byzer-lang on K8S/Yarn/Standalone mode, processing power is limited to Driver.
We should refactor code in load
and save
method, and partition the http paging request if possible, For instance, the auto increment paging strategy can be parallelised.
In our FLAG env, running MLSQLRest code on 16GB Local mode byzer-lang is much faster than 8GB driver + 2 ✖ 8GB executors K8S byzer-lang.