scikit-learn-intelex icon indicating copy to clipboard operation
scikit-learn-intelex copied to clipboard

Distributed sklearn kmeans based on ray and modin

Open PivovarA opened this issue 4 years ago • 0 comments
trafficstars

  • Added one_ccl transceiver.
  • Rewrote dist_kmeans.h. The spark sample and daal4py used map_tree, but it is based on send and receive operations, which oneccl not support yet. oneDAL KMeans MPI sample didn't use map_tree, so I used it as a basis.
  • Added ray_partition_actor and ray context. At the moment ray_context is basic and serves more to extract some information from the ray cluster. This functionality is planned to be expanded in the future.
  • Added to build.sh the ability to work with oneccl. At the moment, for correct operation, you need a oneccl build: https://github.com/oneapi-src/oneCCL/tree/2021.1-beta07-1 And also set the variable CCLROOT. In addition, it needs to use oneccl_transceiver: export D4P_TRANSCEIVER = oneccl_transceiver

PivovarA avatar Dec 17 '20 09:12 PivovarA