SNOW-728151: [Question / Bug] Allow modifying results processor by reading `asdecimal: bool` attribute

Open YYYasin19 opened this issue 2 years ago • 1 comments

Please answer these questions before submitting your issue. Thanks!

What version of Python are you using? Python 3.11.0 | packaged by conda-forge | (main, Oct 25 2022, 06:21:25) [Clang 14.0.4 ]
What operating system and processor architecture are you using? macOS-13.1-arm64-arm-64bit
What are the component versions in the environment (pip freeze)?

alabaster 0.7.12
anyio 3.6.2
appnope 0.1.3
argon2-cffi 21.3.0
argon2-cffi-bindings 21.2.0
arrow 1.2.3
arrow-cpp 9.0.0
asn1crypto 1.5.1
asttokens 2.2.0
attrs 22.1.0
aws-c-auth 0.6.21
aws-c-cal 0.5.20
aws-c-common 0.8.5
aws-c-compression 0.2.16
aws-c-event-stream 0.2.15
aws-c-http 0.6.27
aws-c-io 0.13.11
aws-c-mqtt 0.7.13
aws-c-s3 0.2.1
aws-c-sdkutils 0.1.7
aws-checksums 0.1.14
aws-crt-cpp 0.18.16
aws-sdk-cpp 1.9.379
babel 2.11.0
backcall 0.2.0
backports 1.0
backports.functools_lru_cache 1.6.4
bcrypt 3.2.2
beautifulsoup4 4.11.1
black 22.10.0
bleach 5.0.1
brotlipy 0.7.0
bzip2 1.0.8
c-ares 1.18.1
ca-certificates 2022.9.24
certifi 2022.9.24
cfgv 3.3.1
charset-normalizer 2.1.1
click 8.1.3
colorama 0.4.6
colour 0.1.5
coverage 6.5.0
cryptography 38.0.4
debugpy 1.6.4
decorator 5.1.1
defusedxml 0.7.1
distlib 0.3.6
docutils 0.19
entrypoints 0.4
exceptiongroup 1.0.4
executing 1.2.0
filelock 3.8.0
flake8 6.0.0
flake8-docstrings 1.6.0
flit-core 3.8.0
furl 2.1.3
furo 2022.12.7
gettext 0.21.1
gflags 2.2.2
glib 2.74.1
glib-tools 2.74.1
glog 0.6.0
gmp 6.2.1
greenlet 2.0.1
gst-plugins-base 1.21.1
gstreamer 1.21.1
icu 70.1
identify 2.5.9
idna 3.4
imagesize 1.4.1
importlib-metadata 5.1.0
importlib_resources 5.10.0
infinity 1.5
iniconfig 1.1.1
intervals 0.9.2
ipykernel 6.17.1
ipython 8.6.0
ipython_genutils 0.2.0
ipywidgets 8.0.2
isort 5.11.2
jedi 0.18.2
jinja2 3.1.2
jpeg 9e
jsonschema 4.17.3
jupyter 1.0.0
jupyter_client 7.4.7
jupyter_console 6.4.4
jupyter_core 5.1.0
jupyter_server 1.23.3
jupyterlab_pygments 0.2.2
jupyterlab_widgets 3.0.3
krb5 1.19.3
libabseil 20220623.0
libblas 3.9.0
libbrotlicommon 1.0.9
libbrotlidec 1.0.9
libbrotlienc 1.0.9
libcblas 3.9.0
libclang 15.0.6
libclang13 15.0.6
libcrc32c 1.1.2
libcurl 7.86.0
libcxx 14.0.6
libedit 3.1.20191231
libev 4.33
libevent 2.1.10
libffi 3.4.2
libgfortran 5.0.0
libgfortran5 11.3.0
libglib 2.74.1
libgoogle-cloud 2.5.0
libgrpc 1.51.1
libiconv 1.17
liblapack 3.9.0
libllvm15 15.0.6
libnghttp2 1.47.0
libogg 1.3.4
libopenblas 0.3.21
libopus 1.3.1
libpng 1.6.39
libpq 15.1
libprotobuf 3.21.12
libsodium 1.0.18
libsqlite 3.40.0
libssh2 1.10.0
libthrift 0.16.0
libutf8proc 2.8.0
libvorbis 1.3.7
libxml2 2.10.3
libzlib 1.2.13
llvm-openmp 15.0.5
lz4-c 1.9.3
make 4.3
markdown-it-py 2.1.0
markupsafe 2.1.1
matplotlib-inline 0.1.6
mccabe 0.7.0
mdit-py-plugins 0.3.1
mdurl 0.1.0
mistune 2.0.4
msodbcsql17 17.9.1.1
mypy 0.991
mypy_extensions 0.4.3
mysql-common 8.0.31
mysql-libs 8.0.31
myst-parser 0.18.1
nbclassic 0.4.8
nbclient 0.7.2
nbconvert 7.2.5
nbconvert-core 7.2.5
nbconvert-pandoc 7.2.5
nbformat 5.7.0
nbsphinx 0.8.10
ncurses 6.3
nest-asyncio 1.5.6
nodeenv 1.7.0
notebook 6.5.2
notebook-shim 0.2.2
nspr 4.35
nss 3.78
numpy 1.23.5
openssl 3.0.7
orc 1.8.1
orderedmultidict 1.0.1
oscrypto 1.2.1
packaging 21.3
pandas 1.5.2
pandoc 2.19.2
pandocfilters 1.5.0
parquet-cpp 1.5.1
parso 0.8.3
passlib 1.7.4
pathspec 0.10.2
pcre2 10.40
pendulum 2.1.2
pexpect 4.8.0
phonenumbers 8.13.2
pickleshare 0.7.5
pip 22.3.1
pkgutil-resolve-name 1.3.10
platformdirs 2.5.2
pluggy 1.0.0
ply 3.11
pre-commit 2.20.0
prometheus_client 0.15.0
prompt-toolkit 3.0.33
prompt_toolkit 3.0.33
psutil 5.9.4
ptyprocess 0.7.0
pure_eval 0.2.2
pyarrow 9.0.0
pycodestyle 2.10.0
pycparser 2.21
pycryptodomex 3.16.0
pydocstyle 6.1.1
pyflakes 3.0.1
pygments 2.13.0
pyjwt 2.6.0
pyodbc 4.0.35
pyopenssl 22.1.0
pyparsing 3.0.9
pyqt5-sip 12.11.0
pyrsistent 0.19.2
pysocks 1.7.1
pytest 7.2.0
pytest-console-scripts 1.3.1
pytest-cov 4.0.0
python 3.11.0
python-dateutil 2.8.2
python-dotenv 0.21.0
python-fastjsonschema 2.16.2
python_abi 3.11
pytz 2022.6
pytzdata 2020.1
pyyaml 6.0
pyzmq 24.0.1
qtconsole-base 5.4.0
qtpy 2.3.0
re2 2022.06.01
readline 8.1.2
requests 2.28.1
send2trash 1.8.0
setuptools 65.5.1
setuptools-scm 7.0.5
sip 6.7.5
six 1.16.0
snappy 1.1.9
sniffio 1.3.0
snowballstemmer 2.2.0
snowflake-connector-python 2.8.3
snowflake-sqlalchemy 1.4.5
soupsieve 2.3.2.post1
sphinx 5.3.0
sphinx-autodoc-typehints 1.19.5
sphinx-basic-ng 1.0.0b1
sphinx-copybutton 0.5.0
sphinx-inline-tabs 2022.1.2b11
sphinxcontrib-applehelp 1.0.2
sphinxcontrib-devhelp 1.0.2
sphinxcontrib-htmlhelp 2.0.0
sphinxcontrib-jsmath 1.0.1
sphinxcontrib-qthelp 1.0.3
sphinxcontrib-serializinghtml 1.1.5
sqlalchemy 1.4.45
sqlalchemy-utils 0.38.3
sqlalchemy-utils-arrow 0.38.3
sqlalchemy-utils-babel 0.38.3
sqlalchemy-utils-base 0.38.3
sqlalchemy-utils-color 0.38.3
sqlalchemy-utils-encrypted 0.38.3
sqlalchemy-utils-intervals 0.38.3
sqlalchemy-utils-password 0.38.3
sqlalchemy-utils-pendulum 0.38.3
sqlalchemy-utils-phone 0.38.3
sqlalchemy-utils-timezone 0.38.3
sqlalchemy-utils-url 0.38.3
sqlite 3.40.0
stack_data 0.6.2
tabulate 0.9.0
terminado 0.17.0
tinycss2 1.2.1
tk 8.6.12
toml 0.10.2
tomli 2.0.1
tornado 6.2
tqdm 4.64.1
traitlets 5.6.0
types-tabulate 0.9.0.0
typing-extensions 4.4.0
typing_extensions 4.4.0
tzdata 2022g
ukkonen 1.0.1
unixodbc 2.3.10
urllib3 1.26.13
virtualenv 20.17.1
wcwidth 0.2.5
webencodings 0.5.1
websocket-client 1.4.2
wheel 0.38.4
widgetsnbextension 4.0.3
xz 5.2.6
yaml 0.2.5
zeromq 4.3.4
zipp 3.11.0
zlib 1.2.13
zstd 1.5.2

What did you do?

I am working on a framework where we want to compare the outputs of multiple DBMS. Snowflake, however, returns Decimal as a result of some queries. I initially believed that this code was the reason for that, though I can't reproduce the time it get's called. Searching around in sqlalchemy I found out that there are multiple ways to tell the engine/dialect to convert incoming numbers into float instead of Decimal values. Either by setting the asdecimal attribute or my modifying the results_processor altogether, I assume.

However, setting snowflake.sqlalchemy.custom_types._CUSTOM_DECIMAL.asdecimal=False has no effect on the outcome.

My goal would be to modify the SnowflakeDialect at startup time as such that all values are automatically parsed as float values instead of having to process them myself at every possible location.

Is that somehow possible? Is this missing in the snowflake-sqlalchemy part?

I am happy to contribute if pointed towards the right direction :)

What did you expect to see?

I expected all values to be returned as float after setting asdecimal to False

Jan 18 '23 13:01 YYYasin19

thank you for raising this ! from the looks of it, it looks like an enhancement rather a bug; adding a capability to automatically parse return values as float - and the PR is more than welcome, if that's within the possibilities.

if you think it's more a bug than an enhancement, could you please provide a runnable piece of code (and perhaps, a sample dataset if that makes sense) which leads to the issue you're experiencing? thank you in advance!

Mar 15 '24 12:03 sfc-gh-dszmolka