[FEATURE] Support for Feature Spec Creation for Online Table Serving

[drewipsonhq]

Use-cases

I would like the Databricks Terraform Provider to support the creation of a feature_spec object/function within the Unity Catalog. This is essential for serving lookup tables in online serving endpoints, specifically for model serving scenarios. The feature_spec should support versioning and updates to reflect changes in the source online table schema dynamically.

Attempted Solutions

I’ve attempted to address this need by running a workflow using a Python notebook to manually create the feature_spec. However, this approach is not integrated with Terraform and lacks the ability to automatically update when the source online table schema changes.

Proposal

To address this requirement through the Databricks Provider for Terraform, I propose adding support for managing feature_spec objects within the Unity Catalog. This could be achieved by introducing new Terraform resource types or extending existing ones to allow specification of feature_spec details and its association with the source online table schema. Here’s an example of how this might look in Terraform configuration:

resource "databricks_feature_spec" "example_feature_spec" {
  name        = "example_feature_spec"
  description = "Feature spec for model serving lookup tables"
  # pull features from source table
  source_table {
    database = "example_db"
    table    = "online_table"
  }
  
  # Other feature_spec configurations as needed
}

References

Currently, there are no specific GitHub issues related to this feature request.

[kolomietsdv]

Here at JetBlue we also really miss this feature