starrocks-connector-for-apache-spark
starrocks-connector-for-apache-spark copied to clipboard
StarRocks
connector版本：1.1.2 spark版本：3.2.0 Spark报错日志： `Caused by: java.lang.RuntimeException: java.sql.Date is not a valid external type for schema of date at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.StaticInvoke_0$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Serializer.apply(ExpressionEncoder.scala:207)` 我的sr表的一个字段是date格式，spark在读取的时候报了上述错误
## What type of PR is this： - [ ] BugFix - [x] Feature - [ ] Enhancement - [ ] Refactor - [ ] UT - [ ] Doc...
我的spark版本的3.3.1 scala版本是2.12.14 starrocks的版本是3.2.0 spark-connector的版本是starrocks-spark-connector-3.3_2.12-1.1.2.jar mysql-jdbc的版本是6.0.6 当我查询表数据时，会报以下错误 starrocksSparkDF.show(10) Hive Session ID = 7e592fe6-75aa-4025-abac-4ac70fae390d 2024-03-28 14:39:16,623 Executor task launch worker for task 0.0 in stage 0.0 (TID 0) ERROR Recursive call to...
## What type of PR is this： - [ ] BugFix - [x] Feature - [ ] Enhancement - [ ] Refactor - [ ] UT - [ ] Doc...
`The MAP data type is supported from v3.1 onwards.` But until now, this connector does not supports spark map type to starrocks maptype
基于dataset创建出来的视图是有数据的，但是count数量却为0，最后导出的结果也是空  
1. check if BE node is active 2. if not, don't select this ## What type of PR is this： - [x] BugFix - [ ] Feature - [ ]...
When saving the data from delta table to starrocks with starrocks-connector-for-apache-spark operations breaks with `Value count does not match column count. Expect 6, but got 0.Column delimiter: 9,Row delimiter: 10..`...
类似doris spark connector，支持读取部分列： 
`23/11/16 17:51:48 WARN BackendClient: Connect to StarRocks StarRocks BE{host='192.168.57.75', port=9060} failed. com.starrocks.shade.org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: connect timed out at com.starrocks.shade.org.apache.thrift.transport.TSocket.open(TSocket.java:226) at com.starrocks.connector.spark.backend.BackendClient.open(BackendClient.java:82) at com.starrocks.connector.spark.backend.BackendClient.(BackendClient.java:67) at com.starrocks.connector.spark.rdd.ScalaValueReader.(ScalaValueReader.scala:51) at com.starrocks.connector.spark.sql.ScalaStarrocksRowValueReader.(ScalaStarrocksRowValueReader.scala:36) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at...
