Simeon Simeonov
Simeon Simeonov
> Does the slowness have more to do with the verbosity of show table extended (contextual information we don't strictly need)? Yes. `SHOW TABLES` runs quickly.
This aligns with our experience @jtcohen6. Forgive my unfamiliarity with DBT architecture: how do capabilities "degrade" based on the quality of metadata available about assets a data production depends on?...
Then the path forward seems to be the implementation of your original idea: to generate the `like 'table_one|table_two|...'` from DBT's project information, is that right?
@jtcohen6 @TalkWIthKeyboard yes, Spark will use `database` or `namespace` based on the Spark version. `dbt` code should expect and support either. The change was made for ANSI SQL compatibility reasons.
Per @smtaylorslc @theromsh @boxysean I'll add that our team has moved away from dbt Cloud because of this problem.
@xyu thanks for doing all this work and preparing a PR: this is awesome and we'd love to evaluate merging the changes. That said, this PR is difficult to review...
@xyu I created the https://github.com/swoop-inc/elasticsearch-statsd-plugin/tree/v1x branch for your PRs to target. Once we are done with the integration there we can switch master to the new version.
+1 for this issue. A (very) common use case is running the equivalent of a project-specific `spark-shell` (the REPL for Spark, which I think is currently the most popular open-source...
I'd be happy to write a blog post and share with the Spark community once this is done. :-)
@willb the issue I am referring to has nothing to do with Spark per se (it exists in v1.4.1). I have no problem using SparkContext and common RDD operations via...