optimus
optimus copied to clipboard
raystack
Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
Bigquery external table API supports auto schema detection for external table source by a boolean property - [ref](https://cloud.google.com/go/docs/reference/cloud.google.com/go/bigquery/latest#cloud_google_com_go_bigquery_ExternalDataConfigOptions). If the target field names are the same as the source fields...
Currently, the BigQuery view is created using SQL query. Although we need a SQL query to create a schema, we should support updating an existing schema with descriptions in the...
Changes : * init-container: * airflow deploys - init container + executor container * `fetches config` : airflow to load config onto init-container & executor container * default_env + fetched_env...
**Description** Currently the code for cli to read the yaml configuration related to job and resource are present along with server side code. We need to move these to client...
This is to address [this issue](https://github.com/odpf/optimus/issues/548). Changes: * remove `ProjectJobSpecRepository` and `NamespaceJobSpecRepository` * refactor job spec repository to handle job spec only * remove `ResolveStaticDependencies` because: * error in its...
Currently the configuration inheritance will create a task or hook to be added to all the jobs if added in `this.yaml`, this should inherit only the configuration. This should help...
Prototyping of atomicity property on create / update resource
**Describe the bug** If the plugin process fails due to any error then it does not come back up again, and results in connection refusals for Optimus on the socket...
**Description** This is part of [this discussion](https://github.com/odpf/optimus/discussions/479). There are some todos in airflow scheduler implmentations. This issue is to address such todos: - [ ] method ListJobs can get record...