Miguel Rodriguez Gutierrez
Miguel Rodriguez Gutierrez
- Removed some references to Unity Catalog being in preview as [Unity Catalog is GA](https://docs.databricks.com/release-notes/product/2022/august.html#unity-catalog-is-ga) - Fixed order of container and storage account in ADLS2 path
Currently importing from Azure Blob storage or ADLS only supports paths using `https://` protocol. However, for many Azure applications, the `abfss://` protocol is preferred and has a different syntax (or...
## Description Since Spark 3.4, [spark-connect](https://spark.apache.org/docs/latest/spark-connect-overview.html) (and the equivalent [databricks-connect v2](https://docs.databricks.com/en/dev-tools/databricks-connect-ref.html)) were introduced for seamless development with remote Spark sessions. This is extremely useful for interactive debugging of kedro pipelines...
# Environment - Linux - python 3.10.10 - deltalake==0.10.2 **Environment**: - **Cloud provider**: Azure Databricks *** # Bug **What happened**: I am trying to replicate this example from the [documentation](https://delta-io.github.io/delta-rs/python/usage.html#loading-a-delta-table)...
## Description I want to discuss how we could improve the way dependencies are managed for `SparkDataSet` and similar ## Context Currently `kedro-datasets[sparks]` installs pyspark, which is a >300MB monster...
## Context Currently, the preferred method of authentication with a datalake or cloud storage when using Databricks is via [Unity Catalog and external locations](https://docs.databricks.com/en/sql/language-manual/sql-ref-external-locations.html), not directly authenticating to the storage....