[MLOPS-596] Add Lakehouse Monitor Resource

[aravind-segu]

trafficstars

Changes

Adds Lakehouse Monitor Resource to the Terraform Provider

Tests

• [x] make test run locally
• [x] relevant change in docs/ folder
• [x] covered with integration tests in internal/acceptance
• [x] relevant acceptance tests are passing
• [ ] using Go SDK

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 78.57143% with 15 lines in your changes are missing coverage. Please review.

Project coverage is 83.47%. Comparing base (d4812c5) to head (ad9481d). Report is 4 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3238   +/-   ##
=======================================
  Coverage   83.46%   83.47%           
=======================================
  Files         174      176    +2     
  Lines       16067    16165   +98     
=======================================
+ Hits        13410    13493   +83     
- Misses       1845     1854    +9     
- Partials      812      818    +6     

Files	Coverage Δ
provider/provider.go	94.65% <100.00%> (+0.02%)	:arrow_up:
catalog/resource_lakehouse_monitor.go	78.26% <78.26%> (ø)

... and 2 files with indirect coverage changes

[alexott]

@aravind-segu integration tests are passing (at least on AWS)

[alexott]

Integration tests are failing - I fixed incorrect template (please pull changes), but even after that, it looks like Update method doesn't send the correct payload:

2024-02-24T11:02:07.276Z [DEBUG] databricks: PUT /api/2.1/unity-catalog/tables/sandbox$tdeaabhaalaji.things$tabikdcljiild.bar$tabikdcljiild_inference/monitor
> {
>   "inference_log": {
>     "granularities": [
>       "1 hour"
>     ],
>     "model_id_col": "model_id",
>     "prediction_col": "prediction",
>     "problem_type": "PROBLEM_TYPE_REGRESSION",
>     "timestamp_col": "timestamp"
>   },
>   "output_schema_name": "sandbox$tdeaabhaalaji.things$tabikdcljiild",
>   "snapshot": {}
> }
< HTTP/2.0 400 Bad Request
< {
<   "error_code": "INVALID_PARAMETER_VALUE",
<   "message": "The given `info` is invalid for the following reason(s):\n - `assets_dir` is required but missing... (154 more bytes)"
< }: tf_req_id=e6f4373c-29d5-83ef-c7fd-ea412a3c369d tf_provider_addr=registry.terraform.io/hashicorp/databricks tf_rpc=ApplyResourceChange tf_resource_type=databricks_lakehouse_monitor
Error: -24T11:02:07.276Z [ERROR] sdk.proto: Response contains error diagnostic: diagnostic_detail="" tf_req_id=e6f4373c-29d5-83ef-c7fd-ea412a3c369d tf_provider_addr=registry.terraform.io/hashicorp/databricks tf_proto_version=5.4 tf_rpc=ApplyResourceChange tf_resource_type=databricks_lakehouse_monitor diagnostic_severity=ERROR
  diagnostic_summary=
  | cannot update lakehouse monitor: The given `info` is invalid for the following reason(s):
  |  - `assets_dir` is required but missing.
  |  - `table_name` is required but missing.
  |  - `profile_metrics_table_name` is required but missing.
  |  - `drift_metrics_table_name` is required but missing.

[alexott]

Integration test is still failing:

    init_test.go:236: Step 1/2 error: After applying this test step, the plan was not empty.
        stdout:
        
        
        Terraform used the selected providers to generate the following execution
        plan. Resource actions are indicated with the following symbols:
          ~ update in-place
        
        Terraform will perform the following actions:
        
          # databricks_lakehouse_monitor.testMonitorInference will be updated in-place
          ~ resource "databricks_lakehouse_monitor" "testMonitorInference" {
              - drift_metrics_table_name   = "sandbox$tdkkfakkkfadj.things$tdkkfakkkfadj.bar$tdkkfakkkfadj_inference_drift_metrics" -> null
                id                         = "sandbox$tdkkfakkkfadj.things$tdkkfakkkfadj.bar$tdkkfakkkfadj_inference"
              - monitor_version            = "0" -> null
              - profile_metrics_table_name = "sandbox$tdkkfakkkfadj.things$tdkkfakkkfadj.bar$tdkkfakkkfadj_inference_profile_metrics" -> null
              - status                     = "MONITOR_STATUS_PENDING" -> null
                # (3 unchanged attributes hidden)
        
                # (1 unchanged block hidden)
            }
        
        Plan: 0 to add, 1 to change, 0 to destroy.

[alexott]

after fixing computed fields, another error in update test:

    init_test.go:236: Step 2/2 error: Error running apply: exit status 1
        
        Error: cannot update lakehouse monitor: Data Monitor 'sandbox$tijgeledgkhhb.things$tejcfjihaeblg.bar$tejcfjihaeblg_inference' does not exist.
        
          with databricks_lakehouse_monitor.testMonitorInference,
          on terraform_plugin_test.tf line 36, in resource "databricks_lakehouse_monitor" "testMonitorInference":
          36:         resource "databricks_lakehouse_monitor" "testMonitorInference" {
        

[aravind-segu]

Integration tests are passing: https://github.com/databricks-eng/eng-dev-ecosystem/actions/runs/8147735527

[aravind-segu]

in general good, pending decision on if we should wait for OpenAPI spec changes for wait command

I checked with Miles, and he is ok to push this for now. I will wait for his approval as well

[samuhepp]

Hey! Will this be included in the next release? If so, when is the approximate target date for that? Keen to use this instead of the notebook approach.