optimus
optimus copied to clipboard
raystack
Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
Jobs and Resources are connected, the connections to be presented to the users both static and runtime view which aids in better debuggability of any issues.
Even though it is not directly has to be fixed in Optimus, there is a need for providing the correctness, availability of the data through which the pipelines can be...
There is an inbuilt notification mechanism in place already but it has to be leveraged to alert the users if an upstream has changed after the downstream runs or if...
Optimus relies on data availability that brings the need of supporting various sensors when data is not managed within the system. - Http Sensor, Optimus Sensor, BQ Sensor, GCS Sensor...
Currently there is no mechanism to test the jobs before deploying, if provided a capbility to verify the jobs through dry run and sample run it would be a good...
There are few limitations in the current automated dependency resolution, where the dependencies are resolved by just keeping one data source in mind, but if there are multiple sources with...
WIP - this needs a further discussion on the usecases it will solve
There are situations especially when rerunning the past pipelines the pipelines should be executed with the previous configs and assets of that time.
The current implementation of bigquery datastore ([here](https://github.com/odpf/optimus/blob/54f0784be99bd128a197ff0a1e4395c598d35efd/ext/datastore/bigquery/bigquery.go)) has methods with similar implementations, mainly on `CreateResource` and `UpdateResource`. These methods are similar, with only difference is how the parameters being sent...
**Description** Once we have init container to load assets and the super kubernetes operator to provide the needed environment variables we don't need to have wrapper images Refer [RFC :...