sling-cli icon indicating copy to clipboard operation
sling-cli copied to clipboard
verified publisher icon slingdata-io

StarRocks: Use FILES to bulk load

Open flarco opened this issue 1 year ago • 2 comments

https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files/

flarco avatar Feb 08 '24 14:02 flarco

Yes.. the issue is that files is only for parquet. Doesn't still work with JSON and CSV yet. It'll take some time so thats why I didn't want you to waste your engineering time and just do inserts for now and then when it gets all moved to files(), then we can refactor for performance.

alberttwong avatar Feb 08 '24 16:02 alberttwong

Also behind the scenes, when you do single inserts in the StarRocks front end, we batch them up and stream load it. So basically our MySQL layer is opinionated insert micro batches.

Files() is just a nice interface to stream load.

alberttwong avatar Feb 08 '24 21:02 alberttwong