zeppelin icon indicating copy to clipboard operation
zeppelin copied to clipboard

[ZEPPELIN-5798] Add multiline sql statement support for Livy Spark SQL interpreter

Open zkytech opened this issue 3 years ago • 14 comments
trafficstars

What is this PR for?

  1. Add multiline sql statement support for Livy Spark SQL interpreter
  2. Bug Fix
  3. Bug Fix

What type of PR is it?

Feature & Bug Fix

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-5798

How should this be tested?

execute multiline sql statement with ";", for example:

show databases;
use xxxx;
show tables;

Questions:

  • Does the licenses files need to update? No
  • Is there breaking changes for older versions? No
  • Does this needs documentation? No

zkytech avatar Aug 14 '22 07:08 zkytech

I have found a new bug in this file.

message data like this will fail with exception com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected a string but was BEGIN_OBJECT at line 1 column 9 path $.

df: org.apache.spark.sql.DataFrame = [_id: struct<oid: string>, pv: bigint ... 2 more fields]
_id	pv	stats_date	uv	
{"_id":{"oid":"6295fdbd458288396d008163"},"pv":0,"stats_date":"20220530","uv":0}

which is caused by code in this place

I will fix this later.

zkytech avatar Aug 15 '22 07:08 zkytech

ping @zkytech

zjffdu avatar Aug 31 '22 14:08 zjffdu

ping @zkytech

What should I do next?

zkytech avatar Sep 01 '22 14:09 zkytech

@zkytech Have you resolved the issue in the above comment ?https://github.com/apache/zeppelin/pull/4443#issuecomment-1214710538

zjffdu avatar Sep 01 '22 14:09 zjffdu

@zkytech Have you resolved the issue in the above comment ?#4443 (comment)

Yes, It has been fixed in this commit https://github.com/apache/zeppelin/pull/4443/commits/c31945d85dd2add9021ed072904dd254efbe1a51

The url in commit message is wrong, it should be https://github.com/apache/zeppelin/pull/4443#issuecomment-1214710538,

zkytech avatar Sep 01 '22 15:09 zkytech

Sorry for late response @zkytech Could you retrigger the CI?

zjffdu avatar Sep 24 '22 03:09 zjffdu

Sorry for late response @zkytech Could you retrigger the CI?

ok

zkytech avatar Sep 24 '22 06:09 zkytech

@zkytech livy interpreter in CI is failed, could help fix it? LivySparkSQLInterpreter

zjffdu avatar Oct 10 '22 14:10 zjffdu

@zjffdu fixed

zkytech avatar Oct 12 '22 07:10 zkytech

Please rebase your PR to current master.

Reamer avatar Oct 12 '22 14:10 Reamer

@Reamer Done

zkytech avatar Oct 13 '22 12:10 zkytech

@Reamer Done

Let's wait for CI tests

Reamer avatar Oct 13 '22 12:10 Reamer

Unit test is required to update. Because JSON number is no longer treat as string in this place

zkytech avatar Oct 14 '22 04:10 zkytech

@zjffdu To fix this bug, I have made a change in this place, JSON number is no longer treat as string . Since JSON doesn't distinguish between integer and floating point fields, I have made a change to unit test, all number in json will be convert to float value.

zkytech avatar Oct 14 '22 05:10 zkytech