ibis icon indicating copy to clipboard operation
ibis copied to clipboard

feat(backend): snowflake backend

Open hammer opened this issue 1 year ago • 4 comments

I could not find an issue to track work on adding a Snowflake backend to Ibis. Feel free to close this one if I missed it!

hammer avatar Aug 09 '22 11:08 hammer

Google is maintaining enterprise backends here: https://github.com/GoogleCloudPlatform/professional-services-data-validator/tree/develop/third_party/ibis one of which is a Snowflake backend.

There's an exciting Pandora's Box of logistics here:

  1. Should these backends be part of ibis-project?
  2. How would we test these systems in our CI? I believe all of these backends are proprietary and require some kind of registration and/or credentials. This makes automation and environment setup more burdensome.
  3. How can the ibis project better serve developers of third party backends? Should we work towards a more stable internal API and provide a push-button way to run the backend test suite against a new backend? This seems like a must-have at some point, if we stick to the third party model, but see the next question.
  4. Should we go the monorepo(ish) route and bring all backends into the ibis-project/ibis repository? There was an effort a while ago to move in the opposite direction. It's hard to tell whether this was net positive move. @tswast maintains BigQuery (which lives under ibis-project as a separate third party backend) and I think he'll have a more informed take than I on how it's been to maintain a third party backend.
  5. Do we have the capacity to maintain 7 (!) additional backends, even if the maintenance is shared?

cpcloud avatar Aug 09 '22 12:08 cpcloud

I did see that repo mentioned in https://github.com/ibis-project/ibis/issues/2803.

I think prioritizing backends from that repo to promote to either this repo or a separate repo within this org would be wise.

For the backends that remain in that repo it might be worth mentioning in the docs their existence and how to make use of them.

hammer avatar Aug 09 '22 12:08 hammer

Should we go the monorepo(ish) route and bring all backends into the ibis-project/ibis repository? There was an effort a while ago to move in the opposite direction. It's hard to tell whether this was net positive move. @tswast maintains BigQuery (which lives under ibis-project as a separate third party backend) and I think he'll have a more informed take than I on how it's been to maintain a third party backend.

The move to https://github.com/ibis-project/ibis-bigquery has been good in the sense that I can support some fixes across versions of Ibis. I wasn't able to support 1.x and 2.x+ unfortunately, due to the extent of the refactors though.

The not-so-great part is that it means that whenever there's a refactor, it means some additional PRs to fix the BigQuery backend after-the-fact. The github workflow means we catch these in tests, but knowing something is broken doesn't always mean I know how to fix it.

tswast avatar Aug 17 '22 21:08 tswast

How would we test these systems in our CI? I believe all of these backends are proprietary and require some kind of registration and/or credentials. This makes automation and environment setup more burdensome.

Indeed, this was a big pain in that repo as well. Teradata was the worst of the bunch, since it doesn't even have a proper open source client library.

tswast avatar Aug 17 '22 21:08 tswast

Closed by #4568.

cpcloud avatar Sep 29 '22 23:09 cpcloud