elasticdl icon indicating copy to clipboard operation
elasticdl copied to clipboard

Feature Request: Support other database such as clickhouse

Open wingerted opened this issue 5 years ago • 1 comments

How to support clickhouse or mysql?

wingerted avatar Jul 13 '20 10:07 wingerted

In the current ElasticDL version, one way to support clickhouse or MySQL is using custom_data_reader_params.
Shard is defined as (shard_name, start_index, num_record). Task is defined as (shard_name, start_index, end_index) Thus, need to put all required info to access clickhouse or MySQL in shard_name, which is a string. Also need to implement data reading in read_records using (shard_name, start_index, end_index).

https://github.com/sql-machine-learning/elasticdl/blob/develop/docs/designs/custom_data_source_support.md

We are working on supporting more general data sources.

skydoorkai avatar Jul 15 '20 07:07 skydoorkai