sqrl
sqrl copied to clipboard
DataSQRL
Compiler for streaming data pipelines and data microservices with configurable engines.
Add this to track external limitations of Snowflake / flink iceberg connector. Currently snowflake does not support equality or positional deletes but support is planned. The flink iceberg connector does...
We can automatically add flink metrics to udf functions, such as invoke rates and counters. This can give greater visibility into flink pipelines with little effort. This can be extended...
When we add a query federation layer, we may be able to automatically detect batching scenarios and create the correct query for them. Some queries are not-trivial or impossible to...
It is best practice now to use relay pagination. It can also help with compatibility. This could be driven by configuration to use different pagination models. https://graphql.org/learn/pagination/
This is a low hanging fruit for showing some impressive performance numbers. 1. Add caching as a engine type and stage after database. 2. Only support time-expiration caching for now...
Since we changed our function system, we need to update the logic that tests if a particular engine can execute a given function/operator.
Ensure converted functions have semantically identical results. Not all functions map, and some functions map to functions with different names, for example `TRUNC` in flink and `TRUNCATE` in postgres.
Users can define their own function translations between different engines (e.g. postgres, duckdb, etc) by creating either com.datasqrl.calcite.function.RuleTransform for hep rule planning transformations, or com.datasqrl.function.translations.SqlTranslation for simple transformations. These interfaces...
There is an extra build folder in the build directory. This should be removed.
The logging configuration is hard-coded in the vertx server. This should be made configurable, as well as add the ability to do structured logging output.