Making a INSERT query synchronous

[kylase]

Currently when an INSERT statement is run, the request will return a PrestoResult immediately, is there a way to make the query synchronous?

[ggreg]

@kylase can you please share the code you're running?

[kylase]

The background of this issue is that we are using presto-python-client in Airflow to do INSERT query into Hive tables instead of using pyhive.

We were looking at changing Airflow's Presto hook to use presto-python-client instead of pyhive but realised the lack of SQLAlchemy compatibility which will affect pandas-related APIs in the hook.

So currently we are running the queries in PythonOperator as

 with prestodb.dbapi.connect(
        host=presto_conn.host,
        port=presto_conn.port,
        user=presto_conn.login,
        catalog=presto_conn.schema,
        schema=<schema>,
        isolation_level=transaction.IsolationLevel.AUTOCOMMIT,
) as conn:
        cur = conn.cursor()
        cur.execute(query)

where query is an INSERT INTO <table> SELECT ...

When this code is run in the operator, it exits immediately.

I would think that introducing SQLAlchemy compatibility would also help for #56?

[ggreg]

SQL Alchemy compatibility is definitely a feature that this client will support. That is one of the goal of implement DBAPI2.

In your example, when the code exits the with blocks it should commit the transaction and wait for the query to complete. If it does not, it is a bug.