Default JSON Template for HTTP integrations missing hotspots and dc objects

[arha]

The default template for HTTP integrations - the one you can edit with your custom fields - is not the one you receive when there's no template. The problem is that with the available mustachejs tools offered by the console, you can't recreate the original template, so you can never have a valid JSON, and the "default template" you can edit misses the hotspots, dc and metadata objects.

How to reproduce:

1. Create a new HTTP integration. Console → Integrations → HTTP → Save it.

2. Check original template: Send a RF message that ends up using this integration, on your HTTP endpoint you will obtain a JSON roughly equivalent to: {"app_eui": ..., "hotspots": [...], "dc": [...], "metadata": [...], ...}

3. Edit your JSON template for this integration. Console → Integrations → HTTP. "ADVANCED - JSON MESSAGE TEMPLATE (OPTIONAL)" → "Template Body" → "Apply default template"

4. Check how this "default template" formats data by sending another RF packet. The fields for hotspots, metadata and dc are missing. It is expected for these fields to be present, since we're using the "default" template. We'll call this template the "faux-default"

Expected behavior: clicking the "Apply default template" should generate code that does not discard the hotspots, metadata and dc fields, just like the original template does, but will still allow you to make changes.

Let's attempt to fix it by hand. For example, we can use the following template:

{
  "id": "{{id}}",
  "name": "{{name}}",
  "app_eui": "{{app_eui}}",
  "dev_eui": "{{dev_eui}}",
  "devaddr": "{{devaddr}}",
  "downlink_url": "{{downlink_url}}",
  "fcnt": "{{fcnt}}",
  "port": "{{port}}",
  "payload": "{{payload}}",
  "reported_at": "{{reported_at}}",
  "payload_hex": "{{base64_to_hex(payload)}}",
  "hotspots": [ {{#hotspots}} 
    {
      "channel": {{channel}}
      "frequency": {{frequency}}, 
      "id": "{id}}",
      "lat": {{lat}},
      "long": {{long}},
      "name": "{{name}}",
      "reported_at": {{reported_at}},
      "rssi": {{rssi}},      
      "snr": {{snr}},
      "spreading": "{{spreading}}",
      "status": "{{status}}",
      "asdf": {{@index}},
    },{{/hotspots}}
  ],
  "dc": { "balance": {{dc.balance}}, "nonce": {{dc.nonce}} }
}

which yields results very similar to the real default template (unlike the faux-default). The problem is that the comma separating the hotspots will always be added, thus creating an invalid JSON, unable to be validated by jq or parsed by libs with default settings. While many libs silently ignore it, JSON specs don't allow trailing commas

I am assuming the backend uses mustache for rendering these fields, since any error on the JSON template will send my integration mustache template render failed. Using mustache tools you are unable to remove this comma and create a valid JSON, for example by using {{#@index !== @last}}, {{/}} after each hotspot (like so: } {{#@index !== @last}}, {{/}} {{/hotspots}}).

Nor can you use indexing on your hotspots: "hotspots": [ {{#hotspots:i}} because Nova will again send your integration mustache template render failed.

So integration messages:

• can contain custom formatting, be a valid JSON, but not contain hotspots and DC
• can be a valid JSON and contain hotspots or dc, but can not contain custom formatting
• can contain custom formatting and hotspots, but not be a valid JSON

My original issue (so we don't get into an XY problem) and why this might be needed: I want hotspots, dc, and the following custom field "payload_hex": "{{base64_to_hex(payload)}}", - for example, when quickly glancing over ingest logs - ideally without extra processing on ingress.