[ISSUE] Feature - DLT Schedules for Pipelines

[kurtmaile]

trafficstars

Hi,

I would like to request a feature for databricks_pipeline resource.

To add the ability to use the new UI feature to 'schedule' on a recurring basis.

This is new in the UI, would be great to get that into the TF provider too :)

Thanks heaps! Kurt

[alexott]

This button just creates a databricks job resource that triggers that DLT pipeline. Look for job documentation for pipeline task block: https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job#pipeline_task-configuration-block

[kurtmaile]

thanks @alexott , super helpful ! :)

[kurtmaile]

One question if I may - have you seen any examples of customers using databricks provider, but building custom TF functionality (e.g null resource?) to be able to trigger side effect behaviours rather then resource state? e.g trigger a run of a pipeline, do that with diff params, stop a pipleine etc. We dont have an expensive CI system, which could probably do as such, but wondering if there is any examples of a TF only option for some simple needs (essentially call a API command with side effets?). thoughts? Thanks heaps, keep up the great work :) !!

[nfx]

@kurtmaile

always_running - (Optional) (Bool) Whenever the job is always running, like a Spark Streaming application, on every update restart the current active run or start it again, if nothing it is not running. False by default. Any job runs are started with parameters specified in spark_jar_task or spark_submit_task or spark_python_task or notebook_task blocks.

Can you try and check if this solves your need? Probably we need to document it more clearly and perhaps even rename it.

[kurtmaile]

Thanks - so applies to DLT pipelines if its wrapped in job/workflow and is running? Inc if its running as part of a scheduled run (or would it wait for this run to finish, in which case it wouldnt be effected until the next schedule run).

Also is there any specific rules that dictate what a 'update' constitutes? e.g updating a pipeline tag v updating a notebook SQL are quite different changes? e.g DLT table metadata metadata, adding more constraints to a particular step.

Does sound in the area of what is needed though so thanks for directing me to this as an option! :)

[nfx]

so applies to DLT pipelines if its wrapped in job/workflow and is running?

@kurtmaile it should, let me know if it doesn't. There's no integration test for that one. curious if it solves your use-case.

Also is there any specific rules that dictate what a 'update' constitutes?

anything that triggers update of terraform resource - either tag value, or notebook path, or any spark config of a task - anything.

Logic is very straightforward https://github.com/databricks/terraform-provider-databricks/blob/master/jobs/resource_job.go#L581-L583 & https://github.com/databricks/terraform-provider-databricks/blob/master/jobs/resource_job.go#L350-L375

[nfx]

@kurtmaile how did your testing with always_running go?

[kurtmaile]

@nfx thanks for replying will let you know re: always running when we test it, appreciate the feedback!

[nfx]

Following up - is this issue still relevant?