snowflake-sqlalchemy icon indicating copy to clipboard operation
snowflake-sqlalchemy copied to clipboard

SNOW-280650: CopyIntoStorage will not load data from an ExternalStage

Open ajmarks opened this issue 5 years ago • 0 comments

Please answer these questions before submitting your issue. Thanks!

  1. What version of Python are you using (python --version)? Python 3.8.5

  2. What operating system and processor architecture are you using (python -c 'import platform; print(platform.platform())')? Windows-10-10.0.19041-SP0

  3. What are the component versions in the environment (pip list)?

...
snowflake-connector-python 2.3.9            py38haa244fe_0    conda-forge
snowflake-sqlalchemy      1.2.4              pyh9f0ad1d_0    conda-forge
...
  1. What did you do? Attempt to COPY INTO from an external stage
from snowflake.sqlalchemy import CopyIntoStorage, ExternalStage

qry = CopyIntoStorage(
    into=table,
    from_=ExternalStage(stage_name),
    formatter=sa.text(format_spec),
)
conn.execute(qry)
  1. What did you expect to see? A working query. Instead, the compiler wraps the stage name in parens, so the query is, e.g., COPY INTO public.test_table_65234d02 FROM (@my_stage) ... instead of COPY INTO public.test_table_65234d02 FROM @my_stage.

The issue is in base.py:190:

        elif isinstance(copy_into.from_, AWSBucket) or isinstance(copy_into.from_, AzureContainer):
            from_ = copy_into.from_._compiler_dispatch(self, **kw)
        # everything else (selects, etc.)
        else:
            from_ = '({})'.format(copy_into.from_._compiler_dispatch(self,
                                                                     **kw))

This should be

        elif isinstance(copy_into.from_, (AWSBucket, AzureContainer, ExternalStage)):
            from_ = copy_into.from_._compiler_dispatch(self, **kw)
        # everything else (selects, etc.)
        else:
            from_ = '({})'.format(copy_into.from_._compiler_dispatch(self,
                                                                     **kw))

Even better would be to have all three inherit from a common ancestor and check on that so that it can be easily extended to include GCP or other providers.

ajmarks avatar Feb 08 '21 18:02 ajmarks