Review NGSI to Crate translation corner cases

[c0c0n3]

We need to review how we turn incoming NGSI data into Crate tabular format to identify corner cases we are unaware of and possibly change some of the existing translation logic.

For example, at the moment we accept empty attributes, storing them as NULL strings in Crate:

$ curl -v -X POST http://0.0.0.0:8668/v2/notify -H 'Content-Type: application/json' -d'
{
    "data": [
        {
            "id": "d:1",
            "type": "d",
            "a1": { }
        }
    ]
}
'

"Notification successfully processed"

$ curl -v http://0.0.0.0:8668/v2/entities/d:1

{
    "data": {
    "attributes": [
      {
        "attrName": "a1",
        "values": [
          null
        ]
      }
    ],
    "entityId": "d:1",
    "index": [
      "2019-01-30T13:34:28.043"
    ]
  }
} 

(We have a test for this too: test_store_empty_attribute_as_null_text) As discussed in #142, we should probably not store the attribute at all in this case so that if the same attribute comes in the next notification with a type that isn't text, we can still save it without issues.

For the record, here's a simplified version of what's going on with NSGI to Crate type translation of an attribute A (pseudo code):

if A looks like { type: …, …}: t = A[type]
else:                          t = ‘Text’

if t in NGSI_TO_CRATE:
    if t = ‘StructuredValue’ && A[value] is list:
        crate_type = array(string)
    else:
        crate_type = NGSI_TO_CRATE[t]
else:
    if A looks like { value: {…}, …}:
        crate_type = object
    else:
        crate_type = string

So in particular if A = {} (no type, no value) then we assign it a string type and because of #126 we end up saving it as a NULL string in Crate. Also note that if t = […], or t = {…}, or any other unhashable value, then the test if t in NGSI_TO_CRATE fails with a runtime exception. There might be more corner cases I haven't thought of...

[c0c0n3]

this issue goes hand-in-hand with #141.

[github-actions[bot]]

Stale issue message