BHoM_Engine: Elegantly handle nested lists within JSONs

[tg359]

Description:

Deserialisation from JSON into BHoM does not handle nested lists elegantly.

Steps to reproduce:

Try loading this JSON and converting it into a CustomObject:

{
    "Data": [
        [1, 2, 3],
        [4, 5, 6],
        [7, 8, 9],
        [10, 11, 12],
        [
            [1, 2, 3],
            [4, 5, 6],
            [7, 8, 9],
            [10, 11, 12]
        ]
    ]
}

Expected behaviour:

I'd like to be able to load from JSON containing this type of nested data structure. A recursive cast to List

Test file(s):

[IsakNaslundBh]

Hi @tg359 , I wonder if this is not a UI issue rather than a serialisation issue? It looks like the output from the json is nested lists of numbers to the level at least I would expect?

as a quick fix for now you can add two c# components after each other with a simple A=x; in them. That make sure all the data is untangled without raising errors and returning nulls.

[FraserGreenroyd]

Ok so I've investigated this extensively today and have found the following.

First, this is an issue with BHoM_UI and not BHoM_Engine/deserialisation of lists. The deserialisation works fine, but the issue described is a UI problem.

Secondly, the issue is with nested lists - in the example given by @tg359 the data is contained as a List<List<double>> effectively. This is deserialised to List<List<object>> and the BHoM_UI doesn't handle it appropriately. This is because the overall type is List<object>, so when calling BHoM_Engine/Query/UnderlyingType, we get back object and a depth of 1, telling the output accessor to be SetDataList rather than SetDataTree. This is because the UnderlyingType, which is object, is actually a List<object> - but the reflection queries don't know that because obviously List<object> and object are both valid and basically the same.

All of that said, I did manage to tweak the BHoM_UI to make this work, and get the output in the screenshot below.

This is a far nicer UX than having to set down C# components with A = x; as needed.

However, I was originally going to say this wasn't something we could/should aim to resolve, because nested lists are notoriously cumbersome within BHoM, until I was reminded of this issue, whereby @adecler and myself debated supporting List<List<List<T>>> vs List<List<T>>. Although the discussion is limited, both @adecler and I agree (by virtue of my comment and his thumbs up) that supporting List<List<T>> is ok, and anything nested greater than that shouldn't be.

Within BHoM itself, we don't have many objects in the core oM with List<List<T>> properties, but some do exist within toolkit level objects, and these can be handled fine with the explode functionality. Which comes onto the final point, as far as I can ascertain, this is an issue with Exploding Custom Objects which contain a List<List<T>> property. Our key objects (anything defined code side) work fine (tested with a couple of objects at least), but CustomObjects do not.

I think it's therefore fair to say, based on this aforementioned issue, and existing objects, that we should be able to handle exploding List<List<T>> properties on CustomObjects.

Therefore I am going to get this tackled with an update to BHoM_UI for this specific edge case.

[FraserGreenroyd]

Just to finalise comments on this, following discussion with @adecler after prototyping up this PR, we agreed it was actually better to try and resolve the problem at source with the Serialiser rather than visually with the UI. The issue was the deserialisation process was not accounting for nested lists, and when this was resolved via #3160 the visualisation and use within Grasshopper was also resolved.

This gives a stronger deserialisation process rather than hiding the issue with a bandaid on the UI and is a more appropriate solution - but wanted to capture that here for future reference just in case.

We have implemented the support for List<List<T>> level nesting, which as discussed in the previous comment, is the level of nesting we currently wish to support.