Takeshi Yamamuro

icon indicating a website link https://twitter.com/maropu icon indicating an email [email protected]

icon location Tokyo/Japan icon location OSS engineer@R&D, Ph.D. in CS (Database Systems) - Apache Spark PMC&committer, Apache Hivemall PPMC, PostgreSQL enthusiast - LLVM/C/C++11/Java/Scala/Rust/Python

Results 28 comments of Takeshi Yamamuro

VM-like optimization in decoding loops - Direct Threading, or something

Reference to https://github.com/rui314/threading/blob/master/direct.c

VM-like optimization in decoding loops - Direct Threading, or something

Reference to http://jp.rubyist.net/magazine/?0008-YarvManiacs

A correction of decodeArray(..) in every one algorithm class

Yep, you're right, and my code's slightly wrong..., so I'll fix it in a few days following your points. Thx :))

Lineage Data can be directly output in JSON format？

Ah, it might be worth doing it, but is there a common JSON format to represent a directed acyclic graph?

Lineage Data can be directly output in JSON format？

INFO: I added an interface to define an output format for exporting a generated graph into other systems in https://github.com/maropu/spark-sql-flow-plugin/commit/abb1f0f2f9591d3f511948a57ae8fde0032e25ff: https://github.com/maropu/spark-sql-flow-plugin/blob/abb1f0f2f9591d3f511948a57ae8fde0032e25ff/src/test/scala/org/apache/spark/sql/SQLFlowSuite.scala#L175-L190

Supported Insert as select sql

Yea, but it is not easy to implement it because Spark does not hold the relationship in its catalog. Probably, we need a custom SQL listener to track `INSERT` query...

Supported Insert as select sql

I don't have it because I don't have a smart idea for implementing it. Any plan to use the feature if it's implemented?

Calling Java from <other language>

Currently, no. But, the feature sounds good to me. That's a kind of future work.

Unsupported types in alloca: Type=StructTyID

This project doesn't target a full-fledge transformation, so I feel its difficult to support the conversion in the example above....

build fail of "error: scala.reflect.internal.MissingRequirementError: object java.lang.Object in compiler mirror not found."

Thanks for the report. You hit this issue when calling `./build/mvn clean package -DskipTests`?