Pari
PR - https://github.com/Nike-Inc/brickflow/pull/138
This is indeed very handy feature. However, we should enable this feature only for local development. for higher environments, it does not make sense to deploy only set of workflows...
> For local: we will default it to None, but will give an option to override and schedule it For dev, qa & prod: Folks should be able to decide...
assign it to muhammadwaqas0505
addressed it as part of for each functionality
Primary goal of brickflow is orchestration. Box integration or any such integrations is part of koheesio. e.g. https://github.com/Nike-Inc/koheesio/blob/main/src/koheesio/integrations/box.py
https://docs.databricks.com/en/dev-tools/databricks-connect/python/limitations.html
check the affected/reference code within Koheesio Not available on Databricks Connect for Databricks Runtime 13.3 LTS and below: - [ ] Streaming foreachBatch ==> @riccamini -- NA -- works for...
Run the unit test locally with spark-connect remote instant.
Check how to manage SparkSession and DatabricksSession.