version function in importlib.metadata returns None and causes TypeError

[sblumin]

Driver version

JupyterLab IPython : 7.32.0 ipykernel : 6.5.1 ipywidgets : 7.6.5 jupyter_client : 7.1.0 jupyter_core : 4.9.1 jupyter_server : 1.12.1 jupyterlab : 3.2.4 nbclient : 0.5.9 nbconvert : 6.3.0 nbformat : 5.1.3 notebook : 6.4.6 qtconsole : 5.2.1 traitlets : 5.1.1

Redshift version

N/A

Client Operating System

JupyterLab conda_python3 kernel

Python version

3.8.12

Table schema

N/A

Problem description

Executing redshift_connector.connect() in python 3.8 results in TypeError: expected string or bytes-like object. This is due to get_pkg_version() inside redshift_connector/idp_auth_helper.py. In python versions 3.6 and 3.7 from importlib.metadata import version as version returns ModuleNotFoundError and the version is correctly derived using pkg_resources.get_distribution(module_name).version. However, on python 3.8 the importlib.metadata module can be imported, but version(module_name) returns None. This is why the TypeError is returned when executing line match = self._regex.search(version) as it expects a version string but None is given instead.

1. Expected behaviour: redshift_connector.connect() completes successfully and dataframes can be loaded into jupyter
2. Actual behaviour: redshift_connector.connect() returns TypeError: expected string or bytes-like object
3. Error message/stack trace:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_18887/103128458.py in <cell line: 15>()
     13 
     14 # Connect to DB
---> 15 conn = redshift_connector.connect(
     16     access_key_id=creds['AccessKeyId'],
     17     secret_access_key=creds['SecretAccessKey'],

~/anaconda3/envs/python3/lib/python3.8/site-packages/redshift_connector/__init__.py in connect(user, database, password, port, host, source_address, unix_sock, ssl, sslmode, timeout, max_prepared_statements, tcp_keepalive, application_name, replication, idp_host, db_user, app_id, app_name, preferred_role, principal_arn, access_key_id, secret_access_key, session_token, profile, credentials_provider, region, cluster_identifier, iam, client_id, idp_tenant, client_secret, partner_sp_id, idp_response_timeout, listen_port, login_url, auto_create, db_groups, force_lowercase, allow_db_user_override, client_protocol_version, database_metadata_current_db_only, ssl_insecure, web_identity_token, role_session_name, role_arn, iam_disable_cache, auth_profile, endpoint_url, provider_name, scope, numeric_to_float, is_serverless, serverless_acct_id, serverless_work_group, group_federation)
    330 
    331     if not redshift_native_auth:
--> 332         IamHelper.set_iam_properties(info)
    333 
    334     _logger.debug(make_divider_block())

~/anaconda3/envs/python3/lib/python3.8/site-packages/redshift_connector/iam_helper.py in set_iam_properties(info)
     91         provider_type: IamHelper.IAMAuthenticationType = IamHelper.IAMAuthenticationType.NONE
     92         # set properties present for both IAM, Native authentication
---> 93         IamHelper.set_auth_properties(info)
     94 
     95         if info._is_serverless and info.iam:

~/anaconda3/envs/python3/lib/python3.8/site-packages/redshift_connector/idp_auth_helper.py in set_auth_properties(info)
     80         #     )
     81         if info.iam is True:
---> 82             _logger.debug("boto3 version: {}".format(IdpAuthHelper.get_pkg_version("boto3")))
     83             _logger.debug("botocore version: {}".format(IdpAuthHelper.get_pkg_version("botocore")))
     84 

~/anaconda3/envs/python3/lib/python3.8/site-packages/redshift_connector/idp_auth_helper.py in get_pkg_version(module_name)
     47             return Version(pkg_resources.get_distribution(module_name).version)
     48 
---> 49         return Version(version(module_name))
     50 
     51     @staticmethod

~/.local/lib/python3.8/site-packages/packaging/version.py in __init__(self, version)
    262 
    263         # Validate the version and parse it into pieces
--> 264         match = self._regex.search(version)
    265         if not match:
    266             raise InvalidVersion(f"Invalid version: '{version}'")

TypeError: expected string or bytes-like object

4. Any other details that can be helpful:

Python Driver trace logs

N/A

Reproduction code

Run in python 3.8 JupyterLab

!pip install redshift_connector
import redshift_connector
import boto3
import os

conn = redshift_connector.connect(
    access_key_id=creds['AccessKeyId'],
    secret_access_key=creds['SecretAccessKey'],
    session_token=creds['SessionToken'],
    
    cluster_identifier=cluster_id, # Cluster name, replace if connecting to a different cluster
    database=db_name, # Database name, replace if using a different database
    region=region,

    db_user='user',
    db_groups=['sais_users'],
    password='',
    auto_create=True,
    user='',
    iam=True,
)

[Brooke-white]

@sblumin , what version of redshift_connector are you using?

[Brooke-white]

Tthis looks to be the latest driver release. I just tried to repro in Pycharm and am unable to see this issue using Python 3.8.2. I will continue to look into this, but in the meantime I'd recommend downgrading the version of redshift_connector you're using to avoid this issue.

[Brooke-white]

In JupyterLab with Python3.8.5 and the following, I was unable to repo

jupyter core     : 4.7.1
jupyter-notebook : 6.3.0
qtconsole        : 5.1.1
ipython          : 7.29.0
ipykernel        : 6.5.0
jupyter client   : 6.1.12
jupyter lab      : not installed
nbconvert        : 6.0.7
ipywidgets       : 7.6.5
nbformat         : 5.1.3
traitlets        : 5.1.1

will get my env inline with yours and continue looking

[sblumin]

Output of !pip show redshift_connector in conda_python3 kernel in JupyterLab

Name: redshift-connector
Version: 2.0.909
Summary: Redshift interface library
Home-page: https://github.com/aws/amazon-redshift-python-driver
Author: Amazon Web Services
Author-email: [email protected]
License: Apache License 2.0
Location: /home/ec2-user/anaconda3/envs/python3/lib/python3.8/site-packages
Requires: beautifulsoup4, boto3, botocore, lxml, packaging, pytz, requests, scramp, setuptools
Required-by:

[sblumin]

Hi team, any update here? Additionally, do you have any recommendation on which version of redshift_connector to downgrade to?

[Brooke-white]

Hi @sblumin, the team was unable to reproduce this issue when configuring our environment inline with yours. We introduced the use of importlib.metadata in 2.0.909. Could you please try to use 2.0.908 and let us know if you still see the issue?

[concavegit]

Hello Brooke, we have narrowed down the issue to our own custom packaging of boto3 that does not have metadata files. This is not an issue with the python driver.