marquez icon indicating copy to clipboard operation
marquez copied to clipboard

Migrate `namespace` to be optional in `API` and `WEB`

Open phixMe opened this issue 6 months ago • 0 comments

Currently, the endpoints for marquez all look along the lines of: http://localhost:5000/api/v1/namespaces/{namespace}/datasets/{dataset}

This means that namespaces are effectively mandatory when users are working with the api and thus the web contains a dropdown allowing users to select a namespace. This is not ideal for a few reasons:

  1. We either need to default a namespace (1st in the list) or force a user to select a namespace before anything else can be rendered.
  2. A user does not always know the namespace of a job or dataset they are looking for because many of the OL integrations (Spark, Airflow, etc...) assign these.
  3. A long list of namespaces is not going to be ideal since a MQ user with thousands of namespaces will need to scroll a long way and have application rendering degraded.

Therefore, we need to handle namespaces a little differently across the board to rectify these problems listed above.

Proposal

Migrate the format of the endpoints to use query parameters like below and then forward old formats to new formats for a few releases (or forever)

# OLD
http://localhost:5000/api/v1/namespaces/{namespace}/datasets/{dataset}
# NEW
http://localhost:5000/api/v1/datasets/{dataset}?namespace={namespace}

Update the web to just render all the jobs and datasets, paginated of course without regard to namespace. Include a namespace filtering mechanism with a typeahead select so that users with many namespaces can find there known namespaces and filter.

phixMe avatar Aug 20 '24 17:08 phixMe