dbx icon indicating copy to clipboard operation
dbx copied to clipboard

TypeError: dbx.api.adjuster.mixins.sql_properties.DashboardsList() argument after ** must be a mapping, not list

Open lubrunn opened this issue 2 years ago • 0 comments
trafficstars

Expected Behavior

When defining an Alert SQL task through the deployment.yml like this:

environments:
  default:
    workflows:
      - name: "alert-job"
          - task_key: "alert_task"
            sql_task:
              alert:
                alert_id: "alert://alert1"
              warehouse_id: "warehouse://sql-warehouse"

it should automatically determine the alert_id through the alert name like it does for warehouses and queries etc.

Current Behavior

Currently I get the following error in my azure pipeline:

╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /__w/1/s/prj-venv/lib/python3.9/site-packages/dbx/commands/deploy.py:154 in  │
│ deploy                                                                       │
│                                                                              │
│   151 │   │   if pipelines:                                                  │
│   152 │   │   │   dbx_echo("Found DLT pipelines definition, applying them fi │
│   153 │   │   │   for elements in [pipelines, workflows]:                    │
│ ❱ 154 │   │   │   │   adjuster.traverse(elements)                            │
│   155 │   │   │   │   wf_manager = WorkflowDeploymentManager(api_client, ele │
│   156 │   │   │   │   wf_manager.apply()                                     │
│   157 │   │   else:                                                          │
│                                                                              │
│ /__w/1/s/prj-venv/lib/python3.9/site-packages/dbx/api/adjuster/adjuster.py:1 │
│ 87 in traverse                                                               │
│                                                                              │
│   184 │   │   dbx_echo("Starting the traversal process")                     │
│   185 │   │   self.property_adjuster.library_traverse(workflows, self.additi │
│   186 │   │   self.property_adjuster.file_traverse(workflows, self.file_adju │
│ ❱ 187 │   │   self.property_adjuster.property_traverse(workflows)            │
│   188 │   │   self.property_adjuster.cluster_policy_traverse(workflows)      │
│   189 │   │   dbx_echo("Traversal process finished, all provided references  │
│   190                                                                        │
│                                                                              │
│ /__w/1/s/prj-venv/lib/python3.9/site-packages/dbx/api/adjuster/adjuster.py:1 │
│ 45 in property_traverse                                                      │
│                                                                              │
│   142 │   │   │   │   │   self._adjust_dashboard_ref(element, parent, index) │
│   143 │   │   │   │                                                          │
│   144 │   │   │   │   elif element.startswith("alert://"):                   │
│ ❱ 145 │   │   │   │   │   self._adjust_alert_ref(element, parent, index)     │
│   146 │                                                                      │
│   147 │   def cluster_policy_traverse(self, workflows: WorkflowList):        │
│   148 │   │   """                                                            │
│                                                                              │
│ /__w/1/s/prj-venv/lib/python3.9/site-packages/dbx/api/adjuster/mixins/sql_pr │
│ operties.py:116 in _adjust_alert_ref                                         │
│                                                                              │
│   113 │                                                                      │
│   114 │   def _adjust_alert_ref(self, element: str, parent: Any, index: Any) │
│   115 │   │   alert_name = element.replace("alert://", "")                   │
│ ❱ 116 │   │   _relevant = DashboardsList(                                    │
│   117 │   │   │   **self.api_client.perform_query("GET", path="/preview/sql/ │
│   118 │   │   )                                                              │
│   119 │   │   _id = _relevant.get(alert_name).id                             │

TypeError: dbx.api.adjuster.mixins.sql_properties.DashboardsList() argument 
after ** must be a mapping, not list

Steps to Reproduce (for bugs)

  1. Create deployment.yml with the following:
environments:
  default:
    workflows:
      - name: "alert-job"
          - task_key: "alert_task"
            sql_task:
              alert:
                alert_id: "alert://alert1"
              warehouse_id: "warehouse://sql-warehouse"
  1. Deploy.

Context

Your Environment

  • dbx version used: 0.8.18

On a first glance I think the reason for the error might be that in the _adjust_alert_refmethod of the SqlPropertiesAdjusterclass (line 116) in dbx/api/adjuster/mixins/sql_properties.py it should reference the class AlertsList instead of DashboardsList:

.
.
.

class DashboardsList(FlexibleModel, ResultsListGetterMixin):
    results: List[DashboardInfo] = []

    @property
    def object_type(self) -> str:
        return "dashboard"


class AlertsList(FlexibleModel, ResultsListGetterMixin):
    results: List[AlertInfo] = []

    @property
    def object_type(self) -> str:
        return "alert"

class SqlPropertiesAdjuster(ApiClientMixin, ElementSetterMixin):
    # TODO: design of this class is a terrible copy-paste. It must be rewritten.

    @functools.cached_property
    def _warehouses(self) -> WarehousesList:
        return WarehousesList(**self.api_client.perform_query("GET", path="/sql/warehouses/"))

    def _adjust_warehouse_ref(self, element: str, parent: Any, index: Any):
        _id = self._warehouses.get(element.replace("warehouse://", "")).id
        self.set_element_at_parent(_id, parent, index)

    def _adjust_query_ref(self, element: str, parent: Any, index: Any):
        query_name = element.replace("query://", "")
        _relevant = QueriesList(
            **self.api_client.perform_query("GET", path="/preview/sql/queries", data={"q": query_name})
        )
        _id = _relevant.get(query_name).id
        self.set_element_at_parent(_id, parent, index)

    def _adjust_dashboard_ref(self, element: str, parent: Any, index: Any):
        dashboard_name = element.replace("dashboard://", "")
        _relevant = DashboardsList(
            **self.api_client.perform_query("GET", path="/preview/sql/dashboards", data={"q": dashboard_name})
        )
        _id = _relevant.get(dashboard_name).id
        self.set_element_at_parent(_id, parent, index)

    def _adjust_alert_ref(self, element: str, parent: Any, index: Any):
        alert_name = element.replace("alert://", "")
        _relevant = AlertsList(
            **self.api_client.perform_query("GET", path="/preview/sql/alerts", data={"q": alert_name})
        )
        _id = _relevant.get(alert_name).id
        self.set_element_at_parent(_id, parent, index)

However, I haven't confirmed this. (When I provide the explicit alert_id the deployment goes through without any problems btw.)

lubrunn avatar Aug 04 '23 13:08 lubrunn