elasticdl
elasticdl copied to clipboard
Feature Request: Support other database such as clickhouse
How to support clickhouse or mysql?
In the current ElasticDL version, one way to support clickhouse or MySQL is using custom_data_reader_params.
Shard is defined as (shard_name, start_index, num_record).
Task is defined as (shard_name, start_index, end_index)
Thus, need to put all required info to access clickhouse or MySQL in shard_name, which is a string.
Also need to implement data reading in read_records using (shard_name, start_index, end_index).
https://github.com/sql-machine-learning/elasticdl/blob/develop/docs/designs/custom_data_source_support.md
We are working on supporting more general data sources.