spark-druid-connector icon indicating copy to clipboard operation
spark-druid-connector copied to clipboard

Update some codes for spark version 2.4.4

Open fwani opened this issue 4 years ago • 0 comments

When I tested original code with spark(2.4.4), I got some trouble. So, I updated 4 difference steps.

  1. Change spark, scala version and dependency modules version in build.sbt.
  2. Add case to check field like Alias(Cast(....))
  3. Add case to check LogicalPlan in scanPlan.
  • before
    • df.select('fieldA').take(1) --> checked case Project(projectList, _)
    • df.take(1) --> checked case _
  • after
    • df.select('fieldA').take(1) --> checked case Project(projectList, _)
    • df.take(1) --> checked case a if a.isInstanceOf[LogicalPlan]
  1. Change log level when the druid client requests some thing to druid server.

fwani avatar Oct 28 '20 04:10 fwani