Add `field_mask` to webhooks and webhook templates

[happyRip]

Summary

Closes #5597

Changes

• Add field_mask field to webhook templates and entities
• Apply the field mask (if present) before marshalling the uplink
• Console integration

Testing

Tested manually by adding field_mask to a webhook with

$ ttn-lw-cli app webhooks set appname webhookname --field-mask received_at

When no field_mask is set all fields are sent.

When there is a field_mask it filters the fields to be sent.

---

Added FieldMask field to relevant unit tests.

Regressions

n/a

Notes for Reviewers

No frontend integration of the new field_mask field was added.

Checklist

• [x] Scope: The referenced issue is addressed, there are no unrelated changes.
• [x] Compatibility: The changes are backwards compatible with existing API, storage, configuration and CLI, according to the compatibility commitments in README.md for the chosen target branch.
• [ ] Documentation: Relevant documentation is added or updated.
• [ ] Changelog: Significant features, behavior changes, deprecations and fixes are added to CHANGELOG.md.
• [x] Commits: Commit messages follow guidelines in CONTRIBUTING.md, there are no fixup commits left.

[adriansmares]

Don't forget to update the documentation at https://github.com/TheThingsIndustries/lorawan-stack-docs/tree/master/doc/content/integrations/webhooks/webhook-templates to also include the field masks.

[kschiffer]

When trying to set the field mask via the API, I get:

{
  "code": 3,
  "message": "json: cannot unmarshal array into Go value of type map[string]json.RawMessage"
}

Here's the request:

curl 'http://localhost:8080/api/v3/as/webhooks/232342/rstrst' \
  -X 'PUT' \
  -H 'Accept: application/json, text/plain, */*' \
  -H 'Connection: keep-alive' \
  -H 'Content-Type: application/json' \
  … [redacted for brevity]
  --data-raw '{"webhook":{"service_data":null,"location_solved":null,"downlink_queue_invalidated":null,"downlink_queued":null,"downlink_failed":null,"downlink_sent":null,"downlink_nack":null,"downlink_ack":null,"join_accept":null,"downlink_api_key":"","field_mask":["rstrst"],"headers":{}},"field_mask":{"paths":["downlink_ack","downlink_api_key","downlink_failed","downlink_nack","downlink_queue_invalidated","downlink_queued","downlink_sent","field_mask","headers","join_accept","location_solved","service_data"]}}' \
  --compressed

[adriansmares]

Good catch - since it is a google.protobuf.FieldMask, we need to provide it as

		"field_mask": {
			"paths": ["rstst"]
		},

instead.

[kschiffer]

I've checked Console as well and setting field masks via templates is working. I've also added functionality to add field masks for custom webhooks. While doing that I also fixed a bug which lead to the webhook form not displaying certain errors.

[philippdzm]

Hi @kschiffer

I've also added functionality to add field masks for custom webhooks.

This does not work for me.

I think, I am the first wanting to use field-mask with custom webhook template. Can you try updating the dezem.yml and test this? Here is my change: https://github.com/deZemLabs/lorawan-webhook-templates/commit/189b2d99a2456e6cfa5cd990d84be89ba8940f4b

My tests showed that the lorawan-stack seem to ignore field-mask. I Assume this is a bug.

Can you check that? That'd be nice

Edit: To be more precise on what is not working for me: When applying a webhook template in the console, I expect to see entries in "Filter event data". This was not the case.

[kschiffer]

Hi @philippdzm! By "custom webhooks" I meant the ones that are set in the Console when using the "Custom Webhook" button, meaning webhooks without template. If field masking in your custom webhook is not working, then @happyRip might want to take another look at it. Iirc though, some fields cannot be filtered out from the webhook data at all, but I might be wrong. In any case, @happyRip / @adriansmares can assist you better with this.

[philippdzm]

Thanks @kschiffer . I edited my question to be more precise on what was not working. Hopefully your colleagues can have a look and help.

[kschiffer]

I actually identified the problem, which was indeed coming from the Console. I have opened a PR and it should land in the next version next week.

#5655

Thanks for notifying us of this issue!

[philippdzm]

@kschiffer amazing! I assume, you meant https://github.com/TheThingsNetwork/lorawan-stack/pull/6204 instead of https://github.com/TheThingsNetwork/lorawan-stack/pull/5655 (which is this PR) 👍

[philippdzm]

Another question. How come that the paths in field-mask diverge from the paths in the Webhook's payload?

Example for the spreading factor

field-mask
    - mask up.uplink_message.settings.data_rate.modulation.lora.spreading_factor

and in the payload of the Webhook uplink_message->settings->data_rate->lora->spreading_factor

Based on the documentation, I was not able to understand this.

[adriansmares]

Field masks are centered around the protobuf format used to actually pass these messages around inside The Things Stack.

Fields such as the data_rate have oneof fields such as modulation - you cannot have more than one modulation at the same time. The wrapper (modulation) is not rendered in JSON, but it exists at protobuf level.