ibis-project
feat(flink): temporal join support
Description of changes
Explorative PR towards addressing https://github.com/ibis-project/ibis/issues/8247.
This is my first time getting hands-on with sqlglot changes :) So, particularly seeking suggestions on
- Temporal join API
- The newly added op
VersionedDatabaseTableand expressionTemporalJoin.
Note: Added support only in the Flink backend for now. Did not spend time on how to error out when the user calls tempora_join() on a backend that does not support temporal join. Plan to address these once we reach an agreement on the API.
Issues closed
Is there another ibis backend which supports this kind of temporal join? I would like to avoid introducing global APIs highly tailored towards certain backends since it would defeat the purpose of ibis. Having it implemented for another backend could validate the API and IR design we have here.
Just had a closer look at the issue description which you have nicely collected the necessary information in, apparently the options are RisingWave and MySQL. How much work would it be to support risingwave here?
Is there another ibis backend which supports this kind of temporal join? I would like to avoid introducing global APIs highly tailored towards certain backends since it would defeat the purpose of ibis. Having it implemented for another backend could validate the API and IR design we have here.
Just had a closer look at the issue description which you have nicely collected the necessary information in, apparently the options are RisingWave and MySQL. How much work would it be to support risingwave here?
Adding support for RisingWave would have been great, but RisingWave seems to support only processing-time temporal join. This PR adds support for only event-time temporal join. In event-time temporal join, rows are joined with their right-table versions specified by a given time-attribute (at_time in this PR). In processing-time temporal join, rows are always joined with their most-recent version in the right-table.
For MySQL, I could not find enough information on whether it supports temporal join or not. At the time, I could find only this article that discusses support for performing joins against temporal tables, which does not have the same semantics as Flink's temporal join. In my understanding, it is for joining against a particular version of the table that is specified by a given timestamp rather than a time attribute. So it is like first performing a time travel on the table(s) and then joining them.
Closing, as I don't think I can maintain this code at the moment. We can always revisit later!
