databricks
Max retries exceeded with url (NewConnectionError) - Upgrading from 2.7.0 to 2.9.3
Description
I encountered this very consistent error when using the dbt-databricks adapter when upgrading from dbt-databricks==1.5.5 to dbt-databricks==1.6.6. For a dbt run with sufficiently enough models, this error always occurs eventually - sometimes immediately and sometimes after many models have successfully built:
Runtime Error
Runtime Error
Database Error
HTTPSConnectionPool(host='<HOST>', port=443): Max retries exceeded with url: /sql/1.0/warehouses/<URL>
(Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x16d7d5390>:
Failed to establish a new connection: [Errno 12] Cannot allocate memory'))
I isolated the issue to be with this databricks-sql-python package when upgrading from databricks-sql-connector==2.7.0 to databricks-sql-connector==2.9.3.
The fact that it's a NewConnectionError isn't surprising but that it's failing because of "memory allocation" is very odd.
What Works
Using dbt-databricks==1.5.5 and databricks-sql-connector==2.7.0 works perfectly. Even dbt-databricks==1.6.6 then downgrading to databricks-sql-connector==2.7.0, forcing an incompatible package version, results in an environment that still works fine.
What Does Not Work
An environment with dbt-databricks==1.6.6 and databricks-sql-connector==2.9.3.
Other Troubleshooting
$ python --version
Python 3.11.5
$ dbt --version
Core:
- installed: 1.6.6
- latest: 1.6.6 - Up to date!
Plugins:
- databricks: 1.6.6 - Up to date!
- spark: 1.6.0 - Up to date!
I have attempted to adjust all of the profiles.yml parameters for retries and connections, with no success in getting 2.9.3 to work properly, ex:
threads: 4
retry_all: True
connect_retries: 5
connect_timeout: 180
I will add the full list of packages in my python environments below in the comments.
Perhaps these issues will be resolved in the 3.0.0 release, but I wanted to raise this issue just in case!
Packages in working environment:
# Name Version Build Channel
agate 1.7.0 pypi_0 pypi
alembic 1.11.1 pypi_0 pypi
attrs 23.1.0 pypi_0 pypi
babel 2.12.1 pypi_0 pypi
bzip2 1.0.8 h620ffc9_4
ca-certificates 2023.05.30 hca03da5_0
certifi 2023.7.22 pypi_0 pypi
cffi 1.15.1 pypi_0 pypi
charset-normalizer 3.2.0 pypi_0 pypi
click 8.1.3 pypi_0 pypi
colorama 0.4.6 pypi_0 pypi
databricks-sdk 0.1.7 pypi_0 pypi
databricks-sql-connector 2.7.0 pypi_0 pypi
dbt-core 1.5.3 pypi_0 pypi
dbt-databricks 1.5.5 pypi_0 pypi
dbt-extractor 0.4.1 pypi_0 pypi
dbt-spark 1.5.0 pypi_0 pypi
et-xmlfile 1.1.0 pypi_0 pypi
future 0.18.3 pypi_0 pypi
hologram 0.0.16 pypi_0 pypi
idna 3.4 pypi_0 pypi
importlib-metadata 6.8.0 pypi_0 pypi
isodate 0.6.1 pypi_0 pypi
jaraco-classes 3.3.0 pypi_0 pypi
jinja2 3.1.2 pypi_0 pypi
jsonschema 4.18.4 pypi_0 pypi
jsonschema-specifications 2023.7.1 pypi_0 pypi
keyring 24.2.0 pypi_0 pypi
leather 0.3.4 pypi_0 pypi
libffi 3.4.4 hca03da5_0
logbook 1.5.3 pypi_0 pypi
lz4 4.3.2 pypi_0 pypi
mako 1.2.4 pypi_0 pypi
markupsafe 2.1.3 pypi_0 pypi
mashumaro 3.6 pypi_0 pypi
minimal-snowplow-tracker 0.0.2 pypi_0 pypi
more-itertools 10.0.0 pypi_0 pypi
msgpack 1.0.5 pypi_0 pypi
ncurses 6.4 h313beb8_0
networkx 2.8.8 pypi_0 pypi
numpy 1.25.1 pypi_0 pypi
oauthlib 3.2.2 pypi_0 pypi
openpyxl 3.1.2 pypi_0 pypi
openssl 3.0.9 h1a28f6b_0
packaging 23.1 pypi_0 pypi
pandas 1.5.3 pypi_0 pypi
parsedatetime 2.4 pypi_0 pypi
pathspec 0.11.1 pypi_0 pypi
pip 23.2.1 py311hca03da5_0
protobuf 4.23.4 pypi_0 pypi
pyarrow 12.0.1 pypi_0 pypi
pycparser 2.21 pypi_0 pypi
python 3.11.4 hb885b13_0
python-dateutil 2.8.2 pypi_0 pypi
python-slugify 8.0.1 pypi_0 pypi
pytimeparse 1.1.8 pypi_0 pypi
pytz 2023.3 pypi_0 pypi
pyyaml 6.0.1 pypi_0 pypi
readline 8.2 h1a28f6b_0
referencing 0.30.0 pypi_0 pypi
requests 2.28.2 pypi_0 pypi
rpds-py 0.9.2 pypi_0 pypi
setuptools 68.0.0 py311hca03da5_0
six 1.16.0 pypi_0 pypi
sqlalchemy 1.4.49 pypi_0 pypi
sqlite 3.41.2 h80987f9_0
sqlparams 5.1.0 pypi_0 pypi
sqlparse 0.4.4 pypi_0 pypi
text-unidecode 1.3 pypi_0 pypi
thrift 0.16.0 pypi_0 pypi
tk 8.6.12 hb8d0fd4_0
typing-extensions 4.7.1 pypi_0 pypi
tzdata 2023c h04d1e81_0
urllib3 1.26.16 pypi_0 pypi
werkzeug 2.3.6 pypi_0 pypi
wheel 0.38.4 py311hca03da5_0
xz 5.4.2 h80987f9_0
zipp 3.16.2 pypi_0 pypi
zlib 1.2.13 h5a0b063_0
Packages in NON-working environment:
# Name Version Build Channel
agate 1.7.1 pypi_0 pypi
alembic 1.12.0 pypi_0 pypi
attrs 23.1.0 pypi_0 pypi
babel 2.13.1 pypi_0 pypi
bzip2 1.0.8 h620ffc9_4
ca-certificates 2023.08.22 hca03da5_0
certifi 2023.7.22 pypi_0 pypi
cffi 1.16.0 pypi_0 pypi
charset-normalizer 3.3.1 pypi_0 pypi
click 8.1.7 pypi_0 pypi
colorama 0.4.6 pypi_0 pypi
databricks-sdk 0.12.0 pypi_0 pypi
databricks-sql-connector 2.9.3 pypi_0 pypi
dbt-core 1.6.6 pypi_0 pypi
dbt-databricks 1.6.6 pypi_0 pypi
dbt-extractor 0.4.1 pypi_0 pypi
dbt-semantic-interfaces 0.2.2 pypi_0 pypi
dbt-spark 1.6.0 pypi_0 pypi
et-xmlfile 1.1.0 pypi_0 pypi
hologram 0.0.16 pypi_0 pypi
idna 3.4 pypi_0 pypi
importlib-metadata 6.8.0 pypi_0 pypi
isodate 0.6.1 pypi_0 pypi
jaraco-classes 3.3.0 pypi_0 pypi
jinja2 3.1.2 pypi_0 pypi
jsonschema 4.19.1 pypi_0 pypi
jsonschema-specifications 2023.7.1 pypi_0 pypi
keyring 24.2.0 pypi_0 pypi
leather 0.3.4 pypi_0 pypi
libffi 3.4.4 hca03da5_0
logbook 1.5.3 pypi_0 pypi
lz4 4.3.2 pypi_0 pypi
mako 1.2.4 pypi_0 pypi
markupsafe 2.1.3 pypi_0 pypi
mashumaro 3.8.1 pypi_0 pypi
minimal-snowplow-tracker 0.0.2 pypi_0 pypi
more-itertools 8.14.0 pypi_0 pypi
msgpack 1.0.7 pypi_0 pypi
ncurses 6.4 h313beb8_0
networkx 3.2 pypi_0 pypi
numpy 1.26.1 pypi_0 pypi
oauthlib 3.2.2 pypi_0 pypi
openpyxl 3.1.2 pypi_0 pypi
openssl 3.0.11 h1a28f6b_2
packaging 23.2 pypi_0 pypi
pandas 2.1.1 pypi_0 pypi
parsedatetime 2.6 pypi_0 pypi
pathspec 0.11.2 pypi_0 pypi
pip 23.3 py311hca03da5_0
pip-system-certs 4.0 pypi_0 pypi
protobuf 4.24.4 pypi_0 pypi
pyarrow 13.0.0 pypi_0 pypi
pycparser 2.21 pypi_0 pypi
pydantic 1.10.13 pypi_0 pypi
python 3.11.5 hb885b13_0
python-dateutil 2.8.2 pypi_0 pypi
python-slugify 8.0.1 pypi_0 pypi
pytimeparse 1.1.8 pypi_0 pypi
pytz 2023.3.post1 pypi_0 pypi
pyyaml 6.0.1 pypi_0 pypi
readline 8.2 h1a28f6b_0
referencing 0.30.2 pypi_0 pypi
requests 2.31.0 pypi_0 pypi
rpds-py 0.10.6 pypi_0 pypi
setuptools 68.0.0 py311hca03da5_0
six 1.16.0 pypi_0 pypi
sqlalchemy 1.4.49 pypi_0 pypi
sqlite 3.41.2 h80987f9_0
sqlparams 5.1.0 pypi_0 pypi
sqlparse 0.4.4 pypi_0 pypi
text-unidecode 1.3 pypi_0 pypi
thrift 0.16.0 pypi_0 pypi
tk 8.6.12 hb8d0fd4_0
typing-extensions 4.8.0 pypi_0 pypi
tzdata 2023.3 pypi_0 pypi
urllib3 1.26.18 pypi_0 pypi
wheel 0.41.2 py311hca03da5_0
wrapt 1.15.0 pypi_0 pypi
xz 5.4.2 h80987f9_0
zipp 3.17.0 pypi_0 pypi
zlib 1.2.13 h5a0b063_0
I tested several other versions of this databricks-sql-connector package, including the latest unreleased changes from main. Version 2.8.0 still works as expected, but anything >=2.9.0 fails with the same NewConnectionError from the original issue above:
Runtime Error
Runtime Error
Database Error
HTTPSConnectionPool(host='<HOST>', port=443): Max retries exceeded with url: /sql/1.0/warehouses/<URL>
(Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x16d7d5390>:
Failed to establish a new connection: [Errno 12] Cannot allocate memory'))
Versions tested:
2.7.0 - pass
2.8.0 - pass
2.9.0 - fail (DatabricksRetryPolicy bypass)
2.9.1 - fail (yanked package version, DatabricksRetryPolicy bypass)
2.9.2 - fail (yanked package version)
2.9.3 - fail
main@9b48cd49ac55b434b6daa80ef07e2f4f4b0b3173 - fail
Hi @nrichards17! Does this issue still reproduce if you upgrade to latest dbt-databricks and databricks-sql-connector?
