snowflake-connector-python
snowflake-connector-python copied to clipboard
snowflakedb
support for asyncio
The Snowflake connector for python seems to be implemented essentially as API calls over HTTP. Using aiohttp, companion subclasses to SnowflakeConnector, SnowflakeCursor, SnowflakeRestful etc, could be created that implement the key methods as asynchronous coroutines. Then asyncio tools could be used to run Snowflake connection routines alongside other I/O-centric or API-driven tasks.
Has this been considered? Is it a viable addition to the Snowflake Connector? If so I'm happy to contribute, I'd love to hear any requirements you folks might have in mind. Or on the other hand, is it more appropriate as a fork, or as a separate project altogether in the style of aiobotocore?
Thanks for suggestion. Yes, we considered aiohttp as well as any async feature in 3.4+. There are two concerns: 1) since the driver needs to support both 2.7+ and 3.4+, it may need to have a branch for 3.4+, 2) it is not clear how to handle OCSP check in aiohttp.
OCSP check is a requirement from our security team, though the standard Python's SSL library doesn't support it (and even most other clients don't care about the certificate revocation status!). We use a monkey patch on the top of https://github.com/requests/requests to intercept SSL handshake and add the OCSP checks.
So definitely all contribution would be pleased along with the above concerns addressed.
Checking if there has been any progress with asyncio support for this package, the last update seems to be over 2 years ago..
sorry we didn't have enough bandwidth to work on this. A plan is add async support after dropping python2.
Python 2 will stop being supported by the Python Foundation on Jan 1, 2020. Will Snowflake also drop support at that time? Just curious if any timeline has been established on when Snowflake will EOL Python 2 support.
If it helps anyone interested, as a temporary workaround, I've been running Snowflake queries inside a concurrent.futures.ThreadPoolExecutor using what basically amounts to await asyncio.get_event_loop().run_in_executor(...).
This has been just fine for my purposes (internal reporting) and allows me to still use the other neat asyncio features while not blocking the main thread.
For Python 2 support, feel free to discuss in #107. We have not decided when we are going to drop Python 2 support yet. Good new is the tread of Python 2 download is down: https://pypistats.org/packages/snowflake-connector-python
We'll keep eyes on the usage metrics on our end to determine the timing of Python2 drop.
We want this for Snowflake: https://github.com/MagicStack/asyncpg https://github.com/aio-libs/aiomysql
What's the progress on aio support? Anything that uses it, such as sqlalchemy-aio cannot take advantage of this dialect+driver without it...
Thanks for suggestion. Yes, we considered aiohttp as well as any async feature in 3.4+. There are two concerns: 1) since the driver needs to support both 2.7+ and 3.4+, it may need to have a branch for 3.4+, 2) it is not clear how to handle OCSP check in aiohttp.
Python 2.7 will be deprecated since Jan 1 2020. ANy progress on async SnowFlake connector?
Commenting here to re-emphasise the issue. We are going to use asyncio in ongoing projects involving Python, and not having support for async is a bit frustrating. Are there any news regarding this? Thanks.
It is very important for snowflake to support the async connector. Otherwise, we have to workaround. It is a big bottleneck for performance optimization. Thanks.
Hey Everyone, just wanted to bump this feature request. With FastAPI becoming more and more utilized in the python ecosystem, support for an Asyncio continues to become more important. Thanks!
I think what Snowflake would really benefit from is some sort of publicly documented HTTP/REST API. It would be a lot easier to build an asyncio community library on top of something that is documented than trying to reverse engineer it or come up with sub-optimal solutions.
Snowflake is the primary database for a project I am working on. It has been an absolute struggle dealing with all of the gotchas. Having some visibility would be a good idea - however I am going to assume the security team would struggle with this. Definitely sold to our CTO as very much compatible with existing workflows, tech stacks, etc... and showed a lot of promise by offering an sqlalchemy driver.
I have been mostly using usql (https://github.com/xo/usql) to fill in a lot of gaps with the web ui, the almost unusable cli offered by snowflake, and to act as a system call when I need something done async without having to deal with busted multiprocessing solutions.
Next bump. Building a FastAPI application which needs to perform sync and async queries would be much better if the snowflake connector would support asyncio. Yes the sqlalchemy connector is supporting this but lacking snowflake exclusive features like execute_async.
@keller00 are there any updates?
I'm sorry, but this is not planned for anytime soon.
Our codebase is built upon using urllib3 and other dependencies that use it under the hood (boto3 comes to my mind immediately). We also monkey patch our own OCSP verification into urllib3 for extra security.
Last time I checked urllib3 said that they will not support asyncio ever, so to support it we'd need a complete rewrite of the library, which we have tried, but the benchmarks didn't live up to our standards unfortunately.
However; we do as of recently support our own Async execution feature, see documentation here: https://docs.snowflake.com/en/user-guide/python-connector-example.html#label-python-connector-asynchronous-query-examples I hope that this could be useful for some of you!
While the async feature is neat.. it does not offer async socket support.
Perhaps the team at snowflake can look into using httpx as a base.
On Thu, Mar 18, 2021 at 10:05 PM Mark Keller @.***> wrote:
I'm sorry, but this is not planned for anytime soon. Our codebase is built upon using urllib3 and other dependencies that use it under the hood (boto3 comes to my mind immediately). We also monkey patch our own OCSP verification into urllib3 for extra security. Last time I checked urllib3 said that they will not support asyncio ever, so to support it we'd need a complete rewrite of the library, which we have tried, but the benchmarks didn't live up to our standards unfortunately.
However; we do as of recently support our own Async execution feature, see documentation here: https://docs.snowflake.com/en/user-guide/python-connector-example.html#label-python-connector-asynchronous-query-examples I hope that this could be useful for some of you!
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/snowflakedb/snowflake-connector-python/issues/38#issuecomment-802581703, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKRFPLDZ7QPXZYNR3SGXTTELSRLANCNFSM4EAS4B5A .
--
[image: --]
Shane R. Spencer [image: https://]about.me/ShaneSpencer https://about.me/ShaneSpencer?promo=email_sig
Why support python at all if you can't invest the time and effort to support modern Python?
Async Python is Future Python, not Modern Python. A lot of widely used libraries are not (yet) async compatible and most Python running in the real world is not async. Don't be silly.
this is a MUST we use snowflake on top of async python to provide customer facing dashboards and it basically slows down our service dramatically there's no new python project that starts without asyncio please prioritize this!
hi guys! for what its worth I've hacked together a nodejs server that I use as proxy with python's aiohttp it works very well https://github.com/varos-io/snowflake-proxy/
+1 This is pretty critical for us, our entire backend is python asyncio
Looking at writing a threading based async/await wrapper similar to https://github.com/aio-libs/aioodbc
Hey, I'm from the future 👾, we live on Mars, ETH is the main currency, we resurrected dinosaurs. But there is still no support for the async SF connector...
Any updates? 👀
