Riccardo Iacomini

[FEATURE] - DBR 14.3 support - foreachbatch impacts

1

## Is your feature request related to a problem? Please describe. This issue is related to https://github.com/Nike-Inc/koheesio/issues/33. Goal is to investigate the impacts on foreachbatch calls in the code.

Updates to CONTRIBUTING ## Description - removed poetry references - updated make command to run tests ## Related Issue https://github.com/Nike-Inc/koheesio/issues/53 ## Motivation and Context ## How Has This Been Tested?...

[FEATURE] - Generic file writer

1

Generic file writer ## Description This PR provides a generic FileWriter class for several file formats and can be configured with any required option ## Related Issue https://github.com/Nike-Inc/koheesio/issues/52 ## Motivation...

## Describe the bug The contribution guidelines in CONTRIBUTING, referenced in the documentation, require an update: - poetry is listed as package manager - update the make command required to...

## Is your feature request related to a problem? Please describe. Koheesio is currently shipping with the _FileLoader_ which allows to read any file format specifying any needed option. The...

[FEATURE] Databricks Secrets support

1

## Is your feature request related to a problem? Please describe. I want to be able to retrieve secrets stored in an existing scope. ## Describe the solution you'd like...

**Is your feature request related to a problem? Please describe.** Tags that are assigned to the Workflow cannot be currently retrieved inside tasks. **Cloud Information** - [x] AWS - [x]...

**Is your feature request related to a problem? Please describe.** Brickflow context allows setting task values through BrickflowTaskComs: `brickflow_ctx.task_coms.put(task_id="some_task_id", key="some_key", value="some_value"))` This will get by default serialized through pickle +...

## Description Adds support for clustering ## Related Issue https://github.com/Nike-Inc/koheesio/issues/214 ## Motivation and Context This will allow users to configure clustering when writing to a delta table that it does...

feature - Cast to target transformation

4

## Description This PR adds a new spark transformation **CastToTarget** that can be used to cast the input dataframe to a specified target. This transformation takes care of: - adding...