GNIP 93: Allowing changes to the dataset attributes table via REST API

[mwallschlaeger]

GNIP 93 - Allowing changes to the dataset attributes table via REST API

Overview

I guess this might be to small for a GNIP, but it might require a discussion of the PSC.

Currently making changes to the attribute table of a dataset is only possbile via the Advanced Metadata Editor form. To further restify GeoNode it would be nice have the possibility to change attributes via the REST API.

Proposed By

@mwallschlaeger : Marcel Wallschlaeger

Assigned to Release

This proposal is for GeoNode 4.1.

State

• [x] Under Discussion
• [ ] In Progress
• [ ] Completed
• [ ] Rejected
• [ ] Deferred

Motivation

The main motivation for us is requirement to upload data into our GeoNode instance using an external service only communicating to GeoNode through the REST API. Therefor to set or change attributes of a dataset this workflow requires the attribute manipulation via REST interface.

Proposal

Currently the Dataset REST endpoint has a read_only field for the attributes. In my opinion I would leave this construct as it is. and add another endpoint below the Dataset. Something like:

router.register(r"datasets/(?P<pk>\d+)/attributes",views.DatasetAttributesViewSet, "attributes")

Which handles the changes to the attributes of a dataset. This Endpoint would allow to change the "description", "attribute_label", "visable", "featureinfo_type, "display_order".

Backwards Compatibility

I dont think so.

Future evolution

For our specific usecase we further want to add fields to the Attributes models which might be interesting for the whole GeoNode. This would be "unit" and "keywords" for a column so in a later implementation users could find dataset including a specific type of attribute inside of a dataset. This would allow to make dashboards and stories to compare same units and keyword tags of different datasets more easy.

Feedback

Update this section with relevant feedbacks, if any.

Voting

Project Steering Committee:

• Alessio Fabiani:
• Francesco Bartoli:
• Giovanni Allegri:
• Simone Dalmasso:
• Toni Schoenbuchner:
• Florian Hoedt:

Links

[gannebamm]

I see the need for these capabilities. We are researching how an attribute value update would be possible via DRF and REST. @kilichenko-pixida will post some of his findings soon.

[kilichenko-pixida]

I see the need for these capabilities. We are researching how an attribute value update would be possible via DRF and REST. @kilichenko-pixida will post some of his findings soon.

As mentioned, I am working on this in this issue. As of now, I am not adding a separate endpoint for the attributes, I think I found a way to update them through the PATCH api/v2/dataset/<dataset_id>.

I didn't figure out what kind of JSON payload might just work out of the box with the DRF serizlizers for this type of nested field, so instead of passing it as {"attribute_set": [{...}, ...]} (would cause at exception), it would have to be passed as {"data": {"attribute_set": [{...}, ...]}}}. This is later read and parsed in the update method of the DatasetSerializer class (overriding the default from its parent class) and attributes are going to be directly updated from there as well.

I am also preparing a geonodectl command which would allow to pass a JSON file like {"attribute_set": [{...}, ...]} to update the attributes.

[kilichenko-pixida]

@gannebamm @ridoo @mwallschlaeger

Here is a summary of the relevant changes that already happened. I been working on this issue and it resulted in two already merged PRs: one to geonode to add the attrtibute processing ability and another to geonodectl to make patching from JSON files possible in geonodectl as mentioned in this issue. As per testing in the dev environment, patching of attributes is now possible, but current implementation might not be a good solution for a direct use of the API as described in this GNIP.

The reason is that currently the attribute_set values needs to be passed wrapped in a "data" field which allows to bypass internal DRF validation and access it from the update method within the DatasetSerializer class. E.g. {"data": {"attribute_set": [{...}, ...]}}}.

This "wrapping" is done automatically when patching attributes from the geonodectl, but that makes patching directly through the API not obvious (though now at least possible). Details on why it was done this way could be found in the discussion on the first PR to geonode - TLDR We don't know how to do it, but it might turn out to be trivial.

[mattiagiupponi]

Hi @gannebamm @kilichenko-pixida @ridoo @mwallschlaeger

I was checking the issue and I have a couple of notes about the implementation.

attributes as a model

The field attributes serve a purpose and come from Geoserver. Therefore, we must handle them with care. For example, we retrieve the attribute name and type from Geoserver through a function:

https://github.com/GeoNode/geonode/blob/cdeb2d95d6224cb364c1dfc52e3a6addf18c0b2c/geonode/geoserver/helpers.py#L991-L1105

and later set with this function:

https://github.com/GeoNode/geonode/blob/cdeb2d95d6224cb364c1dfc52e3a6addf18c0b2c/geonode/geoserver/helpers.py#L913-L988

in the end, they must always be coherent with Geoserver and the original dataset

Api implementation

Sometimes working with dynamic rest is a headache, I agree. The field always expects a list of IDs rather than a payload to update the values.

The reason is that currently, the attribute_set values need to be passed wrapped in a "data" field which allows us to bypass internal DRF validation and access it from the update method within the DatasetSerializer class. E.g. {"data": {"attribute_set": [{...}, ...]}}}.

This function seems like a workaround solution, which might not be the best approach in case we need to work with another field in the future.

We define a specific action and extend_schema in the viewset, allowing the user to add a schema to the API without creating a new URL.

For example, the extra_metadata:

https://github.com/GeoNode/geonode/blob/cdeb2d95d6224cb364c1dfc52e3a6addf18c0b2c/geonode/base/api/views.py#L1390-L1481

In this function, all the code required to add, delete, update, and handle the extra_metadata attribute for the resource is defined. The corresponding endpoint will be /api/v2/resources/{pk}/extra_metadata.

I would suggest having a similar approach for handling the attribute. This will have several benefits:

• It will limit the scope of the API to a specific usage.
• It will keep the serializer simple by avoiding any additional logic related to a particular field update.
• It will make maintenance easier.

[ridoo]

@mattiagiupponi thanks for your feedback. I am getting more and more into the rest_framework stuff. There is so much "magic" underneath .. once you have cases which diverge from the "normal" (tm) ones, you'd need much deeper understanding of how all works.

In this specific case, I would be curious how you plan to do edits via the REST API when dropping the legacy metadata editor templates. If you plan to re-use the API v2 (which I assume you will) we should revise it with regard to such changes. To my current understanding, the API v2 is too limited to accept all necessary metadata changes of a dataset, right?

[mattiagiupponi]

In this specific case, I would be curious how you plan to do edits via the REST API when dropping the legacy metadata editor templates. If you plan to re-use the API v2 (which I assume you will) we should revise it with regard to such changes. To my current understanding, the API v2 is too limited to accept all necessary metadata changes of a dataset, right?

We are still evaluating how change it, but for sure some API changes might be required

[giohappy]

@ridoo we're speaking of dataset attributes, not metadata in general here, right?