Ignore NULL values

[husqD]

Is your feature request related to a problem? Please describe. The NULL values in the Fiware entities have this syntax:

  "<property_name>": {
       "type" : "Property",
       "value": {
          "@type":  "Intangible",
          "@value":  null
      }
    }

I have a sensor with 4 fields: Humidity1, Humidity2, Humidity3 and Battery. If one of the measurements fails when I create the sensor, this field has a null value in Orion and quantumLeap saves it as an object (see syntax above) and therefore crateDB has this error:

ProgrammingError('SQLActionException[ColumnValidationException: Validation failed for batterylevel: Cannot cast {"@type"=\'Intangible\', "@value"=NULL} to type bigint]')

Describe the solution you'd like I think it's a good idea for quantumLeap to ignore the value when it has the syntax described above

Describe alternatives you've considered Another alternative could be saved the value as Null

[c0c0n3]

@husqD thanks for reporting this. I've got one question though what's the type of the BatteryLevel prop? Looking at the error code it was supposed to be an integer but in your example you've got a type of Intangible. We don't support multiple types for attributes/properties. The first time QL receives an attribute it tries to determine its type and then creates a DB column accordingly. If later on the prop type changes, on receiving the new prop value you'll get a DB type mismatch which causes the error you reported. Could it be that what happened in your tests is this:

1. Connect device.
2. Device sends battery level of integer type and value x, e.g. 10.
3. DB column batterylevel gets created w/ type of bigint and a row gets inserted with value of x.
4. Device sends battery level of Intangible type and value of null.
5. DB insert fails b/c of type mismatch.

Notice how the type changed from step 2 to step 4. Like I said we don't support this. Also, in principle I don't think these two NGSI-LD properties should be considered the same?

"batteryLevel": {
    "type" : "Property",
    "value": {
        "@type":  "Integer",
        "@value":  10
    }
}

"batteryLevel": {
    "type" : "Property",
    "value": {
        "@type":  "Intangible",
        "@value":  null
    }
}

i.e. they just happen to share the same name but probably have a different LD context expansion?

[husqD]

Hi @c0c0n3,

I use the Iot-Agent for Lorawan and when I create a device I define humidity measurements in this way:

{
          "object_id": "h1",
          "name": "soilMoisture1",
          "type": "Property",
          "metadata": { "unitCode": {"type": "Text", "value": "P1" }}
 },

My problem is that in my application the user can manage his own dataloggers. When he defines a sensor, he can choose if the datalogger has 1, 2 or 3 soil moisture sensors. So, if the user defines a datalogger with two moisture sensors but the datalogger sends only the measurements from one moisture sensor, the iot-agent gives the intangible value to the second sensor.

This is an example of a datalogger with two humidity measurements defined, but one has not been sent yet (due to sensor failure or human error), so the iot-agent gives it an intangible value:

{
    "@context": <MyContext>,
    "type": "Device",
    "batteryLevel": {
        "value": 3670,
        "type": "Property",
        "observedAt": "2021-04-26T06:45:52.876Z",
        "unitCode": "2Z"
    },
    "soilMoisture1": {
        "value": 12,
        "type": "Property",
        "observedAt": "2021-04-26T06:45:52.876Z",
        "unitCode": "P1"
    },
    "soilMoisture2": {
        "type": "Property",
        "value": {
            "@type": "Intangible",
            "@value": null
        },
        "unitCode": "P1"
    },
    "hasAgriParcel": {
        "object": "urn:ngsi-ld:AgriParcel:001",
        "type": "Relationship",
        "observedAt": "2021-04-26T06:45:52.876Z"
    }
}

This only happens if the measurement has never been sent, so the iot-agent can't give it a value type. And as a consequence QuantumLeap saves the value as an object.

[c0c0n3]

@husqD thanks so much for the example and for clarifying what's going on. Like I said QL doesn't support changing attribute/property type over time. I realise this is a bit of a bummer, but we don't have any easy way to support that at the moment. But maybe there's a workaround...

So, if the user defines a datalogger with two moisture sensors but the datalogger sends only the measurements from one moisture sensor, the iot-agent gives the intangible value to the second sensor.

I've never come across this Intangible type in the IoT agents and if I search for it in the Lorawan agent repo it doesn't look like it's anywhere in the code base:

• https://github.com/Atos-Research-and-Innovation/IoTagent-LoRaWAN/search?q=Intangible

Which Lorawan agent version are you on? Could it be that it's actually the device using that type? i.e.

device algo

if no soilMoistureN: 
    send (soilMoistureN, Intangible, null)
else
    send (soilMoistureN, Property, valueOf(soilMoistureN))

Or maybe there's some custom mapping defined in the agent config that maps null to Intangible? Is there any way you could change that so soilMoistureN always has a type of Property? Given your setup I'm not sure you can, but I'm asking just in case...

[husqD]

@c0c0n3 thanks for your reply,

The intangible type is an iotagennt-node-lib thing. You can check it in the metadata support section of this documentation: https://iotagent-node-lib.readthedocs.io/en/latest/advanced-topics/index.html

This error is caused by a human error in the lorawan backend. If a user defines a datalogger with two soil moisture measurements, but only sends one value, the iotagent-node-lib gives intangible to the missing value, because it does not know what kind of value it is.

It's not really a big problem, I can try to check the value in the IoT-Agent, but I think it could be useful if quantumLeap could check if the value is intangible to ignore it.

[c0c0n3]

@husqD

The intangible type is an iotagennt-node-lib thing

Holy moly! It's one of those new NGSI-LD changes I missed. Thank you sooo much for the reference!

I think it could be useful if quantumLeap could check if the value is intangible to ignore it.

Agreed! This should be part of the changes we're slowly trying to implement to complete support for NGSI-LD. Well, strictly speaking it's not part of the spec, but for practical purposes we should take it into account I reckon since it made it into the agents lib, so it's going to affect all agents under the sun!

[husqD]

Thank you @c0c0n3!!

I am glad to know that it will be considered. It is appreciated that there are developers with your patience and good humor.

[c0c0n3]

@husqD well, thank you for taking the time to help us out and for patiently explaining things, it usually takes me a bit to catch on to what peeps are saying---don't judge me, old age you know :-)

[github-actions[bot]]

Stale issue message

[SBlechmann]

Hey there,

just to get this straight... there is no way of excluding Null values from QL's response? QL does not yet support simple query language, does it? Thanks as always :)

[chicco785]

Hey there,

just to get this straight... there is no way of excluding Null values from QL's response? QL does not yet support simple query language, does it? Thanks as always :)

@SBlechmann i think your request is unrelated to the one from @husqD so i recommend to open up a specific issue for it

at the time being there is no way to strip null values as you said, but in some queries it may be possible to do so, without creating a mess in the data structure.

[SBlechmann]

@chicco785 yea, I guess you are right, I'm gonna open a seperate issue. Thanks for the answer!