starrocks-connector-for-apache-spark icon indicating copy to clipboard operation
starrocks-connector-for-apache-spark copied to clipboard

Published 20 hours ago • verified publisher icon StarRocks

support spark catalog

Open melin opened this issue 2 years ago • 7 comments

support spark catalog

melin avatar Dec 09 '22 06:12 melin

Hi melin. Would that feature allow to create table in the spark catalog (so far only temp table can be created, not real tables) ?

parisni avatar Dec 24 '22 10:12 parisni

You cannot create a table. It facilitates spark sql to access starrocks table

melin avatar Dec 24 '22 12:12 melin

What is the goal of the starrocks catalog if you can't put tables in it ? I thought you meant a starrocks catalogPlugin , same as iceberg, delta or hudi catalog

parisni avatar Dec 24 '22 16:12 parisni

spark catalog 是方便spark sql 直接访问starrocks 中的表。starrocks 是使用mysql 协议，直接使用spark jdbc catalog 也能通过sql 访问starrocks 表，但那样是走jdbc 读取数据，效率不高。参考pingcap 的tispark： https://github.com/pingcap/tispark/blob/master/docs/userguide_3.0.md#write-with-jdbc-datasource image

melin avatar Dec 25 '22 02:12 melin

I guess I got it. Once you setup a starrocks catalog in spark you can list/load all the tables in starroks

parisni avatar Dec 25 '22 09:12 parisni

I guess I got it. Once you setup a starrocks catalog in spark you can list/load all the tables in starroks

You are right to understand

melin avatar Dec 25 '22 14:12 melin

计划支持吗？

melin avatar Dec 28 '22 08:12 melin