kedro-plugins
kedro-plugins copied to clipboard
kedro-org
First-party plugins maintained by the Kedro team.
## Description Since Spark 3.4, [spark-connect](https://spark.apache.org/docs/latest/spark-connect-overview.html) (and the equivalent [databricks-connect v2](https://docs.databricks.com/en/dev-tools/databricks-connect-ref.html)) were introduced for seamless development with remote Spark sessions. This is extremely useful for interactive debugging of kedro pipelines...
## Introduction How are we going to release when certain libraries are not compatible? i.e. if tensorflow has no support for Python3.11, how do we handle this in our CI?...
## Description Searching for the plugins living in this monorepo yields old or unrelated links. For example, `kedro-airflow`. Ecosia:  Google: 
[TBC] Running `kedro docker run` the first time gives me an error like this: ``` docker: Error response from daemon: error while creating mount source path '/Users/juan_cano/Projects/QuantumBlack Labs/tmp/spaceflights/results': chown /Users/juan_cano/Projects/QuantumBlack...
## Description & context The `upsert` method in `ManagedTableDataSet` is using string-interpolation to pass the table name in the SQL request, see [here](https://github.com/kedro-org/kedro-plugins/blob/main/kedro-datasets/kedro_datasets/databricks/managed_table_dataset.py#L354). The string interpolation is not using regular...
## Description Already existing dataset databricks.ManagedTableDataset doesn't allow to specify the location of the stored files, which in some setups is crucial. There's already PR #251 for it, but it...
## Description Spark 3.4.0 was released in April. Our `databricks` and `spark` datasets should support this newer version of Spark, though it currently causes many tests to fail. Also with...
When Python 3.10 support was added to Kedro 1.5 years ago in https://github.com/kedro-org/kedro/commit/74301c70c31eef71dd372fd3b13afb91b6f297d3, HoloViews tests were skipped on 3.10 due to some Matplotlib incompatibility. However, the HoloViews dependency is tightly...
## Description When checking if a Spark dataset exists, the method always raise instead of returning `False` in case it does not exists. ## Context My pipeline failed when I...
## Context 1. There is currently no clear consistency in what a dataset does; it loads (or, in cases like Spark, connects to) data in some format, and then you...