Update Presto to accept MultiValues Insert

[danfrankj]

Currently without this change inserts are done line by line which can be horrifically slow.

Documentation: https://prestodb.io/docs/current/sql/insert.html FYI @matthewwardrop @lchircus

[codecov-io]

Codecov Report

Merging #183 into master will increase coverage by <.01%. The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #183      +/-   ##
==========================================
+ Coverage   95.71%   95.71%   +<.01%     
==========================================
  Files          14       14              
  Lines        1422     1423       +1     
  Branches      147      147              
==========================================
+ Hits         1361     1362       +1     
  Misses         40       40              
  Partials       21       21

Impacted Files	Coverage Δ
pyhive/sqlalchemy_presto.py	91.45% <100%> (+0.07%)	:arrow_up:

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update b68e1a8...8beba44. Read the comment docs.

[danfrankj]

@jingw would you be able to take a look at this change?

[danfrankj]

@matthewwardrop you don't happen to be a collaborator on for this repo?

[matthewwardrop]

@danfrankj Alas, no. While I have contributed, I don't have commit rights.

[danfrankj]

@KevinFarrell would you be able to take a look at this change

[naoyak]

@gsilk would you mind taking a look at this one?

[eddies]

Anything (technical) holding up this getting reviewed & merged? The Travis failure looks like a network failure that's unrelated to the PR itself.

[naoyak]

@gsilk do you mind taking a look?

On Apr 11, 2019, at 6:32 PM, Edwin Shin [email protected] wrote:

Anything (technical) holding up this getting reviewed & merged? The Travis failure looks like a network failure that's unrelated to the PR itself.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[naoyak]

@bkyryliuk @Fokko can you PTAL and rerun the CI on this PR?

[bkyryliuk]

Thanks @danfrankj World is small :) @naoyak or @danfrankj Would you be open to add a test case with the insert to make sure it works?

[naoyak]

@bkyryliuk I tried to set up a test case using Presto to create a table / insert values, but keep running into a permissions issue (noticed that none of the other Presto tests involve writing to tables). Log here: https://travis-ci.org/github/naoyak/PyHive/jobs/669167402

[bkyryliuk]

@naoyak one row is created in hive: https://github.com/dropbox/PyHive/blob/master/scripts/make_one_row.sh that maybe a reason of the failed write. for the test I would suggest to create a table in presto and ran the inserts against that table.

[naoyak]

@bkyryliuk I tried that approach too (creating a table from the Presto SQLAlchemy connector), which ran into the same problem: https://travis-ci.org/github/naoyak/PyHive/jobs/669143204

error

E   pyhive.exc.DatabaseError: {'message': 'Failed to create directory: file:/user/hive/warehouse/pyhive_test_database.db/insert_multivalue_test', 'errorCode': 16777232, 'errorName': 'HIVE_FILESYSTEM_ERROR', 'errorType': 'EXTERNAL', 'failureInfo': {'type': 'com.facebook.presto.spi.PrestoException', 'message': 'Failed to create directory: file:/user/hive/warehouse/pyhive_test_database.db/insert_multivalue_test', 'cause': {'type': 'java.io.IOException', 'message': 'mkdirs returned false', 'suppressed': [], 'stack': ['com.facebook.presto.hive.metastore.MetastoreUtil.createDirectory(MetastoreUtil.java:630)', 'com.facebook.presto.hive.metastore.SemiTransactionalHiveMetastore$Committer.prepareAddTable(SemiTransactionalHiveMetastore.java:1165)', 'com.facebook.presto.hive.metastore.SemiTransactionalHiveMetastore$Committer.access$300(SemiTransactionalHiveMetastore.java:1073)', 'com.facebook.presto.hive.metastore.SemiTransactionalHiveMetastore.commitShared(SemiTransactionalHiveMetastore.java:964)', 'com.facebook.presto.hive.metastore.SemiTransactionalHiveMetastore.commit(SemiTransactionalHiveMetastore.java:908)', 'com.facebook.presto.hive.HiveMetadata.commit(HiveMetadata.java:2861)', 'com.facebook.presto.hive.HiveConnector.commit(HiveConnector.java:211)', 'com.facebook.presto.transaction.InMemoryTransactionManager$TransactionMetadata$ConnectorTransactionMetadata.commit(InMemoryTransactionManager.java:671)', 'java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)', 'com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)', 'com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)', 'com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)', 'com.facebook.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)', 'java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)', 'java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)', 'java.lang.Thread.run(Thread.java:748)']}, 'suppressed': [], 'stack': ['com.facebook.presto.hive.metastore.MetastoreUtil.createDirectory(MetastoreUtil.java:634)', 'com.facebook.presto.hive.metastore.SemiTransactionalHiveMetastore$Committer.prepareAddTable(SemiTransactionalHiveMetastore.java:1165)', 'com.facebook.presto.hive.metastore.SemiTransactionalHiveMetastore$Committer.access$300(SemiTransactionalHiveMetastore.java:1073)', 'com.facebook.presto.hive.metastore.SemiTransactionalHiveMetastore.commitShared(SemiTransactionalHiveMetastore.java:964)', 'com.facebook.presto.hive.metastore.SemiTransactionalHiveMetastore.commit(SemiTransactionalHiveMetastore.java:908)', 'com.facebook.presto.hive.HiveMetadata.commit(HiveMetadata.java:2861)', 'com.facebook.presto.hive.HiveConnector.commit(HiveConnector.java:211)', 'com.facebook.presto.transaction.InMemoryTransactionManager$TransactionMetadata$ConnectorTransactionMetadata.commit(InMemoryTransactionManager.java:671)', 'java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)', 'com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)', 'com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)', 'com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)', 'com.facebook.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)', 'java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)', 'java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)', 'java.lang.Thread.run(Thread.java:748)']}}

[danfrankj]

Hi @bkyryliuk ! good to see a familiar face out here. We don't use this lib very much anymore so I'll let @naoyak look into the tests.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}