airflow-provider-lakeFS
airflow-provider-lakeFS copied to clipboard
treeverse
lakeFS airflow operator
Objective: Allow users that want to periodically import existing data from an object store into a lakeFS repository How: Using Airflow as an orchestrator, users can create DAGs that trigger...
Apache Airflow v2.7.0 was released on Friday 18th of August with news about [built-in integration with OpenLineage](https://airflow.apache.org/blog/airflow-2.7.0/#openlineage-built-in-integration). Blog post from OpenLineage about the same: https://openlineage.io/blog/airflow-provider This would add great value...
@arielshaqed, should I add some provider tests to this PR to confirm that the following operators, `LakeFSCommitOperator`, `LakeFSGetCommitOperator`, `LakeFSMergeOperator`, and `LakeFSUploadOperator` - now work as expected inside of `TaskGroup`s? _Originally...
### Deadline: "_End of June 2023_" ## What [docker-compose is deprecated](https://docs.docker.com/compose/compose-v2/) and users are encouraged to switch to Docker compose v2. This may be as simple as deleting the hyphen...
The integration test runs on lakefs:latest, which is really nice: that is the most important target to support. Unfortunately it only runs on PRs and pulls to main. So we...