snowflake-connector-python

snowflake-connector-python copied to clipboard

Please answer these questions before submitting your issue. Thanks!

1. What version of Python are you using?

   Python 3.9.2

2. What operating system and processor architecture are you using?

   Windows-10-10.0.19041-SP0

3. What are the component versions in the environment (pip freeze)?

agate==1.6.1
alabaster==0.7.12
appdirs==1.4.4
arrow==1.1.0
asn1crypto==1.4.0
astroid==2.5.1
async-generator==1.10
atomicwrites==1.4.0
attrs==20.3.0
autopep8==1.5.5
azure-common==1.1.27
azure-core==1.20.1
azure-storage-blob==12.9.0
Babel==2.9.0
backcall==0.2.0
bcrypt==3.2.0
binaryornot==0.4.4
black==20.8b1
bleach==3.3.0
boto3==1.18.4
botocore==1.21.4
cached-property==1.5.2
cachetools==4.2.4
certifi==2020.12.5
cffi==1.14.5
chardet==4.0.0
charset-normalizer==2.0.12
click==7.1.2
cloudpickle==1.6.0
colorama==0.4.4
cookiecutter==1.7.3
cryptography==3.4.6
dbt==0.21.0
dbt-bigquery==0.21.0
dbt-core==0.21.0
dbt-extractor==0.4.0
dbt-postgres==0.21.0
dbt-redshift==0.21.0
dbt-snowflake==0.21.0
decorator==4.4.2
defusedxml==0.7.1
diff-match-patch==20200713
docutils==0.16
entrypoints==0.3
flake8==3.8.4
google-api-core==1.31.4
google-auth==1.35.0
google-cloud-bigquery==2.30.1
google-cloud-core==1.7.2
google-crc32c==1.3.0
google-resumable-media==2.1.0
googleapis-common-protos==1.53.0
greenlet==1.1.0
grpcio==1.41.1
helpdev==0.7.1
hologram==0.0.14
idna==2.10
imagesize==1.2.0
importlib-metadata==3.7.2
inflection==0.5.1
intervaltree==3.1.0
ipykernel==5.5.0
ipython==7.21.0
ipython-genutils==0.2.0
isodate==0.6.0
isort==5.7.0
jedi==0.17.2
Jinja2==2.11.3
jinja2-time==0.2.0
jmespath==0.10.0
json-rpc==1.13.0
jsonschema==3.1.1
jupyter-client==6.1.11
jupyter-core==4.7.1
jupyterlab-pygments==0.1.2
keyring==21.8.0
lazy-object-proxy==1.5.2
leather==0.3.4
Logbook==1.5.3
lxml==4.6.4
MarkupSafe==1.1.1
mashumaro==2.5
mccabe==0.6.1
minimal-snowplow-tracker==0.0.2
mistune==0.8.4
msgpack==1.0.2
msrest==0.6.21
mypy-extensions==0.4.3
nbclient==0.5.3
nbconvert==6.0.7
nbformat==5.1.2
nest-asyncio==1.5.1
networkx==2.6.3
numpy==1.20.1
numpydoc==1.1.0
oauthlib==3.1.1
oscrypto==1.2.1
packaging==20.9
pandas==1.2.3
pandocfilters==1.4.3
paramiko==2.7.2
parsedatetime==2.6
parso==0.7.0
pathspec==0.8.1
pexpect==4.8.0
pickleshare==0.7.5
pluggy==0.13.1
poyo==0.5.0
prompt-toolkit==3.0.16
proto-plus==1.19.8
protobuf==3.19.1
psutil==5.8.0
psycopg2-binary==2.9.2
ptyprocess==0.7.0
pyarrow==6.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.6.0
pycparser==2.20
pycryptodomex==3.11.0
pydocstyle==5.1.1
pyflakes==2.2.0
Pygments==2.8.1
PyJWT==2.3.0
pylint==2.7.2
pyls-black==0.4.6
pyls-spyder==0.3.2
PyMySQL==1.0.2
PyNaCl==1.4.0
pyOpenSSL==20.0.1
pyparsing==2.4.7
PyQt5==5.12.3
PyQt5-sip==12.8.1
PyQtWebEngine==5.12.1
pyrsistent==0.17.3
pysftp==0.2.9
python-dateutil==2.8.1
python-jsonrpc-server==0.4.0
python-language-server==0.36.2
python-slugify==5.0.2
pytimeparse==1.1.8
pytz==2021.1
pywin32==300
pywin32-ctypes==0.2.0
PyYAML==6.0
pyzmq==22.0.3
QDarkStyle==3.0.2
qstylizer==0.2.0
QtAwesome==1.0.2
qtconsole==5.1.0
QtPy==1.9.0
regex==2020.11.13
requests==2.25.1
requests-oauthlib==1.3.0
requests-toolbelt==0.9.1
rope==0.18.0
rsa==4.7.2
Rtree==0.9.7
s3transfer==0.5.0
simplejson==3.17.5
six==1.15.0
snowballstemmer==2.1.0
snowflake-connector-python==2.7.4
sortedcontainers==2.3.0
Sphinx==3.5.2
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==1.0.3
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.4
spyder==5.0.3
spyder-kernels==2.0.3
SQLAlchemy==1.4.21
sqlparse==0.4.2
testpath==0.4.4
text-unidecode==1.3
textdistance==4.2.1
three-merge==0.1.1
tinycss2==1.1.0
toml==0.10.2
tornado==6.1
traitlets==5.0.5
typed-ast==1.4.2
typing-extensions==3.7.4.3
ujson==4.0.2
urllib3==1.26.3
watchdog==1.0.2
wcwidth==0.2.5
webencodings==0.5.1
Werkzeug==2.0.2
workday==0.4.0
wrapt==1.12.1
yapf==0.30.0
zeep==3.4.0
zipp==3.4.1

4. What did you do? I'm using concurrent.futures.ThreadPoolExecutor(max_workers=15) to call an API, load the result to a dataframe, then I'm using write_pandas(). The program the calls API thousands of times with different parameters. In VS Code debugger I can see my threads running as they are clearly labeled 0-14. I can also see python-connector-snowflake is starting some threads, as those are labeled with higher numbers, like 107_0, etc. Everything works great, except the last few of the snowflake threads never close.
   At the end when I list my results, it shows an error: cannot import name 'SnowflakeOCSPAsn1Crypto' from partially initialized module 'snowflake.connector.ocsp_asn1crypto. (most likely due to a circular import)'

It seems like all the data made it to snowflake, but I would have no way to check if one of the write_pandas() failed. There are no failed copy commands in the snowflake query history, so I assume it actually worked.

Below is a sample of my code to help you understand what I'm doing. It is not the entire program, but the important part.

def donwload_reports(urls):
    '''
    urls is a list of lists in the form of
    [[url_to_download,file_name],...]
    '''
    with concurrent.futures.ThreadPoolExecutor(max_workers=15) as e:
        results = e.map(download_one_url,urls,itertools.repeat(download_path))
    list(results)
    # the way concurrent.futures works is the error will never materialize unless you iterate through the results.  list(results) is an easy way to do that to raise any errors that happened.   At this step, it raises the error "cannot import name 'SnowflakeOCSPAsn1Crypto' from partially initialized module 'snowflake.connector.ocsp_asn1crypto' (most likely due to a circular import) "

def download_one_url(uri,download_path):
    '''
    used to multi-thread the calling urls and writing to Snowflake
    '''
    url = uri[0]  # the thing that was passed in is a list of list... 
    file_name = uri[1]
    print(f'calling API for {file_name}')
    data = requests.get(url, auth=HTTPBasicAuth(user,password)).content
    data = pd.read_csv(io.StringIO(data.decode('utf-8')),low_memory=False)
    if len(data) > 1:
        data = enhance_df(data,url)
        if output_target == 'local_files':
            data.to_csv(os.path.join(download_path,file_name))
        if output_target == 'snowflake':
            with snowflake_connection.connection() as con:
                write_pandas(con,data,'MY_TABLE')
    return file_name

The traceback is complaining about the connection, here:

def connection():
    '''
    returns the connection object only, not the cursor... after setting role, database, warehouse, and schema.
    used with the write_pandas function
    '''
    ctx = snowflake.connector.connect(
        user=snowflake_user,
        password=snowflake_password,
        account=snowflake_account
        )
    cs = cs = ctx.cursor()
    cs.execute(f'use role {role}')
    cs.execute(f'use database {database}')
    cs.execute(f'use warehouse {warehouse}')
    cs.execute(f'use schema {schema}')
    return ctx

5. What did you expect to see?

   Final threads close? Not sure what is driving this error. I've done this design patter many times.

6. Can you set logging to DEBUG and collect the logs?

This will reveal too much confidential information to post on the public internet. I'm happy to share logs via support portal, email, etc.

I have the same problem

  File "/appenv/lib/python3.10/site-packages/snowflake/connector/__init__.py", line 50, in Connect
    return SnowflakeConnection(**kwargs)
  File "/appenv/lib/python3.10/site-packages/snowflake/connector/connection.py", line 284, in __init__
    self.connect(**kwargs)
  File "/appenv/lib/python3.10/site-packages/snowflake/connector/connection.py", line 494, in connect
    self.__open_connection()
  File "/appenv/lib/python3.10/site-packages/snowflake/connector/connection.py", line 733, in __open_connection
    self._authenticate(auth_instance)
  File "/appenv/lib/python3.10/site-packages/snowflake/connector/connection.py", line 994, in _authenticate
    self.__authenticate(self.__preprocess_auth_instance(auth_instance))
  File "/appenv/lib/python3.10/site-packages/snowflake/connector/connection.py", line 1014, in __authenticate
    auth.authenticate(
  File "/appenv/lib/python3.10/site-packages/snowflake/connector/auth.py", line 257, in authenticate
    ret = self._rest._post_request(
  File "/appenv/lib/python3.10/site-packages/snowflake/connector/network.py", line 700, in _post_request
    ret = self.fetch(
  File "/appenv/lib/python3.10/site-packages/snowflake/connector/network.py", line 790, in fetch
    ret = self._request_exec_wrapper(
  File "/appenv/lib/python3.10/site-packages/snowflake/connector/network.py", line 912, in _request_exec_wrapper
    raise e
  File "/appenv/lib/python3.10/site-packages/snowflake/connector/network.py", line 833, in _request_exec_wrapper
    return_object = self._request_exec(
  File "/appenv/lib/python3.10/site-packages/snowflake/connector/network.py", line 1086, in _request_exec
    raise err
  File "/appenv/lib/python3.10/site-packages/snowflake/connector/network.py", line 988, in _request_exec
    raw_ret = session.request(
  File "/appenv/lib/python3.10/site-packages/snowflake/connector/vendored/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/appenv/lib/python3.10/site-packages/snowflake/connector/vendored/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/appenv/lib/python3.10/site-packages/snowflake/connector/vendored/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/appenv/lib/python3.10/site-packages/snowflake/connector/vendored/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/appenv/lib/python3.10/site-packages/snowflake/connector/vendored/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/appenv/lib/python3.10/site-packages/snowflake/connector/vendored/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/appenv/lib/python3.10/site-packages/snowflake/connector/vendored/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/appenv/lib/python3.10/site-packages/snowflake/connector/ssl_wrap_socket.py", line 81, in ssl_wrap_socket_with_ocsp
    from .ocsp_asn1crypto import SnowflakeOCSPAsn1Crypto as SFOCSP
ImportError: cannot import name 'SnowflakeOCSPAsn1Crypto' from partially initialized module 'snowflake.connector.ocsp_asn1crypto' (most likely due to a circular import) (/appenv/lib/python3.10/site-packages/snowflake/connector/ocsp_asn1crypto.py)

Are you importing the module through any special means? How could the module be partially initialized if you are opening a connection already?

I have found running it on another machine solves the issue. The new machine has: • snowflake-connector-python==2.7.6 • Python 3.7.9

As a reminder, the machine with the issue has: • snowflake-connector-python==2.7.4 • Python 3.9.2

Are you importing the module through any special means?

No, not importing through any special means.

How could the module be partially initialized if you are opening a connection already? Not sure. That's why I created the bug!

Any Updates on this? I am facing the same error on python 3.9.2 with snowflake related packages

snowflake-connector-python==2.7.12
snowflake-sqlalchemy==1.3.2

We had a user that was experiencing this error, the machine had the following: Python Vers: 3.9.0 snowflake-connector-python: 2.7.12

The fix was upgrading to Python version 3.9.10 on the same snowflake conn version.

I've experienced this on a lot of machines now. There are many combinations of Python interpreter / snowflake connector version that conflict. That's why it's important to control the versions used with something like pipenv.

I even experienced this with dbt recently. Their own installation guide to install dbt 1.0 installed a conflicting version for me. Then I had to upgrade the snowflake connector after getting this error. Craziness.

FWIW, I've had that happening to me when running dbt on Debian Bullseye, Python 3.9.2, dbt-core==1.3.1, dbt-snowflake==1.3.0 when I ran dbt multithreaded. Single threaded - no problem (apart from things of course taking a long time.)

What happens when you upgrade your snowflake-connector-python in that environment?

On Fri, Dec 9, 2022, 11:59 AM Tomáš Heran @.***> wrote:

FWIW, I've had that happening to me when running dbt on Debian Bullseye, Python 3.9.2, dbt-core==1.3.1, dbt-snowflake==1.3.0 when I ran dbt multithreaded. Single threaded - no problem (apart from things of course taking a long time.)

— Reply to this email directly, view it on GitHub https://github.com/snowflakedb/snowflake-connector-python/issues/1073#issuecomment-1344538549, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJGCUAEGFPLOCSTCODN6GBTWMNQNTANCNFSM5QIIOARQ . You are receiving this because you authored the thread.Message ID: @.***>

What happens when you upgrade your snowflake-connector-python in that environment?

I've upgraded from 2.7.12 to 2.8.3 and no change, still getting this error:

  cannot import name 'SnowflakeOCSPAsn1Crypto' from partially initialized module 'snowflake.connector.ocsp_asn1crypto' 
(most likely due to a circular import) (/go/src/github.com/Pexeso/analytics/disco_new/pyvenv/lib/python3.9/site-packages/snowflake/connector/ocsp_asn1crypto.py)

FWIW, I've switched to Python 3.9.15 on a different machine and all seems to work just fine even multithreaded.

For completeness, this is the list of the pieces that work for me:

• Debian x86 BullsEye 11.5
• Python 3.9.15
• dbt-core==1.3.0
• dbt-snowflake==1.3.0
• snowflake-connector-python==2.7.12

... the environment where it doesn't work for me differs only in the version of Python - there it's 3.9.2.

FWIW, I believe what this is about is essentially this issue in Python https://bugs.python.org/issue43517. I've verified that the fix (and the fix alone) in 3.9.2 resolves the issues.

It seems like this issue went away for many users by upgrading Python. Please reopen this issue if you are still having this issue after upgrading Python