zeppelin icon indicating copy to clipboard operation
zeppelin copied to clipboard

[ZEPPELIN-5798] Add multiline sql statement support for Livy Spark SQL interpreter

Open zkytech opened this issue 2 years ago • 14 comments

What is this PR for?

  1. Add multiline sql statement support for Livy Spark SQL interpreter
  2. Bug Fix
  3. Bug Fix

What type of PR is it?

Feature & Bug Fix

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-5798

How should this be tested?

execute multiline sql statement with ";", for example:

show databases;
use xxxx;
show tables;

Questions:

  • Does the licenses files need to update? No
  • Is there breaking changes for older versions? No
  • Does this needs documentation? No

zkytech avatar Aug 14 '22 07:08 zkytech

I have found a new bug in this file.

message data like this will fail with exception com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected a string but was BEGIN_OBJECT at line 1 column 9 path $.

df: org.apache.spark.sql.DataFrame = [_id: struct<oid: string>, pv: bigint ... 2 more fields]
_id	pv	stats_date	uv	
{"_id":{"oid":"6295fdbd458288396d008163"},"pv":0,"stats_date":"20220530","uv":0}

which is caused by code in this place

I will fix this later.

zkytech avatar Aug 15 '22 07:08 zkytech

ping @zkytech

zjffdu avatar Aug 31 '22 14:08 zjffdu

ping @zkytech

What should I do next?

zkytech avatar Sep 01 '22 14:09 zkytech

@zkytech Have you resolved the issue in the above comment ?https://github.com/apache/zeppelin/pull/4443#issuecomment-1214710538

zjffdu avatar Sep 01 '22 14:09 zjffdu

@zkytech Have you resolved the issue in the above comment ?#4443 (comment)

Yes, It has been fixed in this commit https://github.com/apache/zeppelin/pull/4443/commits/c31945d85dd2add9021ed072904dd254efbe1a51

The url in commit message is wrong, it should be https://github.com/apache/zeppelin/pull/4443#issuecomment-1214710538,

zkytech avatar Sep 01 '22 15:09 zkytech

Sorry for late response @zkytech Could you retrigger the CI?

zjffdu avatar Sep 24 '22 03:09 zjffdu

Sorry for late response @zkytech Could you retrigger the CI?

ok

zkytech avatar Sep 24 '22 06:09 zkytech

@zkytech livy interpreter in CI is failed, could help fix it? LivySparkSQLInterpreter

zjffdu avatar Oct 10 '22 14:10 zjffdu

@zjffdu fixed

zkytech avatar Oct 12 '22 07:10 zkytech

Please rebase your PR to current master.

Reamer avatar Oct 12 '22 14:10 Reamer

@Reamer Done

zkytech avatar Oct 13 '22 12:10 zkytech

@Reamer Done

Let's wait for CI tests

Reamer avatar Oct 13 '22 12:10 Reamer

Unit test is required to update. Because JSON number is no longer treat as string in this place

zkytech avatar Oct 14 '22 04:10 zkytech

@zjffdu To fix this bug, I have made a change in this place, JSON number is no longer treat as string . Since JSON doesn't distinguish between integer and floating point fields, I have made a change to unit test, all number in json will be convert to float value.

zkytech avatar Oct 14 '22 05:10 zkytech