snowflake-connector-python icon indicating copy to clipboard operation
snowflake-connector-python copied to clipboard

SNOW-610798: Bulk INSERT works incorrectly with OVERWRITE

Open littleK0i opened this issue 3 years ago • 0 comments

Please answer these questions before submitting your issue. Thanks!

  1. What version of Python are you using?
Python 3.9.5 (default, May 18 2021, 12:31:01) 
[Clang 10.0.0 ]
  1. What operating system and processor architecture are you using?
macOS-10.16-x86_64-i386-64bit
  1. What are the component versions in the environment (pip freeze)?
...
snowflake-connector-python==2.7.8
...

SnowflakeCursor object uses the following regular expression to detect INSERT statements and use BindUploadAgent for bulk inserts:

    INSERT_SQL_RE = re.compile(r"^insert\s+into", flags=re.IGNORECASE)

It does work well for normal INSERT INTO, but it does not match INSERT OVERWRITE INTO, so the BindUploadAgent is not being used.

Instead, connector executes INSERT OVERWRITE INTO once for each row, and it ends up with only one row in the final table.

The expected behaviour is to insert all rows in one go and apply OVERWRITE using the single statement, similar to normal INSERT INTO.

Proposed fix: add optional (overwrite)? part to regexp.

Thank you.

littleK0i avatar Jun 13 '22 19:06 littleK0i