Riccardo Iacomini

Results 7 comments of Riccardo Iacomini

[FEATURE] Increase task limit for workflows to 200

Wont't fix this as it is a Databricks limit

[FEATURE] Ensure that we can support DBR 14.3LTS

I have added details in here related to foreachbatch function: https://github.com/Nike-Inc/koheesio/issues/56 If you prefer collecting everything here I will copy paste the comment and close the issue. One additional point...

[FEATURE] - Generic file writer

@dannymeijer Hope now it renders well

[FEATURE] - DBR 14.3 support - foreachbatch impacts

I have found only one reference to DataFrame.foreachbatch function in the `StreamWriter` class (spark.writers.stream.py:292) One additional consideration after reading the [DOC](https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/streaming/readwriter.html#DataStreamWriter.foreachBatch) This function behaves differently in Spark Connect mode. See...

[FEATURE] Databricks Secrets support

There are two ways to retrieve the secrets: - dbutils - databricks API For dbutils, you are forced to run on a notebook, which is not always the case, and...

[BUG] Workflow.health parameter is incorrectly typed

I'll take this one ✋

feature - Cast to target transformation

> How does it work with nested fields ? I will try to make this support also nested structures 💪 > How does it take into account NOT NULL constraint...