clickhouse-java icon indicating copy to clipboard operation
clickhouse-java copied to clipboard

Syntax error when use spark jdbc catalog

Open xiaoc024 opened this issue 1 year ago • 1 comments
trafficstars

Describe the bug

The same sql but not get same result between spark jdbc catalog and DBeave(by jdbc connection)

Steps to reproduce

spark = SparkSession.builder \
        .config("spark.jars", "https://repo1.maven.org/maven2/com/clickhouse/clickhouse-jdbc/0.3.2-patch11/clickhouse-jdbc-0.3.2-patch11-all.jar") \
        .appName("JDBCTableCatalogExample") \
        .getOrCreate()

    spark.conf.set("spark.sql.catalog.xx","org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog")
    spark.conf.set("spark.sql.catalog.xx.url","jdbc:clickhouse://xxx")
    spark.conf.set("spark.sql.catalog.xx.driver","com.clickhouse.jdbc.ClickHouseDriver")
    spark.conf.set("spark.sql.catalog.xx.user","xxx")
    spark.conf.set("spark.sql.catalog.xx.password","xxx")


    df = spark.sql("SELECT * FROM xx.xx_db.aa t final")

    df.show()

Expected behaviour

sql could run and return result

Code example

Error log

File "/Users/tianci.yu/Library/Python/3.8/lib/python/site-packages/pyspark/sql/utils.py", line 117, in deco raise converted from None pyspark.sql.utils.ParseException: extraneous input 'final' expecting {<EOF>, ';'}(line 1, pos 105) == SQL == SELECT * FROM xx.xx_db.aa t final

xiaoc024 avatar Mar 05 '24 10:03 xiaoc024

if i removed alais "t" ,the sql became "SELECT * FROM xx.xx_db.aa final", then it run normal

xiaoc024 avatar Mar 05 '24 10:03 xiaoc024

Why do you expect SparkSQL syntax to recognise Clicksouse's FINAL keyword?

dolfinus avatar Mar 27 '24 22:03 dolfinus