spark icon indicating copy to clipboard operation
spark copied to clipboard

Call skipChildren for nested object

Open tedyu opened this issue 1 year ago • 5 comments

What changes were proposed in this pull request?

In jsonObjectKeys, we should call skipChildren() properly for nested object.

Why are the changes needed?

Currently if jsonParser.currentName() is null, the loop is terminated. We should relocate the current name check inside the loop.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

No.

tedyu avatar Oct 13 '24 13:10 tedyu

cc @panbingkun @MaxGekk

tedyu avatar Oct 13 '24 13:10 tedyu

Test suite seems to pass - there was one error w.r.t. downloading which is unrelated.

[error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] https://maven-central.storage-download.googleapis.com/maven2/org/codehaus/jackson/jackson-jaxrs/1.9.2/jackson-jaxrs-1.9.2.jar: download error: Caught java.io.IOException (Server returned HTTP response code: 503 for URL: https://maven-central.storage-download.googleapis.com/maven2/org/codehaus/jackson/jackson-jaxrs/1.9.2/jackson-jaxrs-1.9.2.jar) while downloading https://maven-central.storage-download.googleapis.com/maven2/org/codehaus/jackson/jackson-jaxrs/1.9.2/jackson-jaxrs-1.9.2.jar
[error]
[error]         at lmcoursier.internal.shaded.coursier.Artifacts$.$anonfun$fetchArtifacts$9(Artifacts.scala:365)

tedyu avatar Oct 13 '24 14:10 tedyu

Do we have corresponding cases? You can use UT to cover it.

panbingkun avatar Oct 14 '24 05:10 panbingkun

@panbingkun @MaxGekk Please take another look.

tedyu avatar Oct 14 '24 18:10 tedyu

The following was not related to this PR

[info] - SPARK-43923: commands send events ((streaming_query_command {
[info]   query_id {
[info]     id: "89ea6117-1f45-4c03-ae27-f47c6aded093"
[info]     run_id: "89ea6117-1f45-4c03-ae27-f47c6aded093"
[info]   }
[info]   stop: true
[info] }
[info] ,None)) *** FAILED *** (10 seconds, 25 milliseconds)
[info]   The code passed to eventually never returned normally. Attempted 661 times over 10.00008105 seconds. Last failure message: VerifyEvents.this.listener.executeHolder.isDefined was false No events have been posted in Timeout(Span(10, Seconds)). (SparkConnectServiceSuite.scala:878)
[info]   org.scalatest.exceptions.TestFailedDueToTimeoutException:
[info]   at org.scalatest.enablers.Retrying$$anon$4.tryTryAgain$2(Retrying.scala:219)
[info]   at org.scalatest.enablers.Retrying$$anon$4.retry(Retrying.scala:226)
[info]   at org.scalatest.concurrent.Eventually.eventually(Eventually.scala:348)
[info]   at org.scalatest.concurrent.Eventually.eventually$(Eventually.scala:347)
[info]   at org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:457)
[info]   at org.apache.spark.sql.connect.planner.SparkConnectServiceSuite$VerifyEvents.executeHolder(SparkConnectServiceSuite.scala:878)
[info]   at org.apache.spark.sql.connect.planner.SparkConnectServiceSuite$VerifyEvents.onCompleted(SparkConnectServiceSuite.scala:898)
[info]   at org.apache.spark.sql.connect.planner.SparkConnectServiceSuite.$anonfun$new$27(SparkConnectServiceSuite.scala:536)
[info]   at 

tedyu avatar Oct 14 '24 18:10 tedyu

This looks like a bug fix?

@tedyu @panbingkun Will it be continued?

@tedyu BTW, please file a Jira for this, as it doesn't seem to be a minor issue. Thanks

LuciferYang avatar Jan 05 '25 03:01 LuciferYang

@LuciferYang I have created https://issues.apache.org/jira/browse/SPARK-50727 for this PR.

@MaxGekk Please review.

tedyu avatar Jan 05 '25 12:01 tedyu

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

github-actions[bot] avatar Apr 16 '25 00:04 github-actions[bot]