optimus
optimus copied to clipboard
raystack
Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
* deploy optimus & airflow on local kubernetes cluster * use optimus odpf helm chart
while specifying the dependency user should just provide job_name if with in the same project & if it is depending on other projects user can specify project_name/job_name as the dependency....
As part of epic around using persisted dependencies for job deployment, dependencies are not recomputed for all jobs every time as it is time consuming. On Resource deployment if there...
On a job creation or deployment, a new or modified job will be stored in the job table and then will proceed to other processes such as resolving dependency resolution,...
Current approach of replay interferes withe the scheduled jobs, there is a need for the replayed jobs should n't interfere with the scheduled jobs https://github.com/odpf/optimus/blob/main/docs/docs/rfcs/20220525_replay_rate_limiting.md
Observability is a key in any mature product, as part of this we would improve various aspects of observability logging, tracing and metrics and provide the relevant dashboards.