amoro icon indicating copy to clipboard operation
amoro copied to clipboard

[spark2.3] support arctic table scan and mor

Open baiyangtx opened this issue 3 years ago • 0 comments

add spark conf

conf.set("arctic.catalog.url", "thrift://xxxx/catalog_name")
conf.set("arctic.resolve-all-identifier", "true" )
conf.set("spark.sql.extensions" , "com.netease.arctic.spark.ArcticSparkSessionExtensions")

note: new spark conf

  • arctic.catalog.url : default arctic catalog url for lookup table meta.
  • arctic.resolve-all-identifier : set to false, only resolve UnresolvedRelation to arctic Table. or resolve ALL LogicalRelation with tableIdentifier to arctic table

support merge on read for keyed table

baiyangtx avatar Jul 19 '22 12:07 baiyangtx