Getting path for nested in nested Get

[wiardvanrij]

I currently have some JSON in which I do the following Get on:

gjson.Get(json, "panels.#.targets.#.expr")

Next I just do a recursive function to basically update the value of it. I use the logic of the depth of the function so set the path of the element. This works great and fine.

However, I came across a use-case where the JSON has a part where it did not have the targets element. This then ruins my logic as it then shifts all my data. Appending the result from an element which did have the full path I query on, to that part which did not have it. (tl;dr I'm using the index from the result, but this is not index on which I actually should update).

Obviously, that's my flaw so I checked if there was a solution for this. I saw there was the option to use Path/Paths. However when I have those two wildcards, I does not seem to give me any result. Also adding a ForEach and then using the path, seem to not make a difference as the output is just empty for Path/Paths.

Relevant issues I checked. However these seem not to work for me (or I implement it wrongly)

https://github.com/tidwall/gjson/issues/199 https://github.com/tidwall/gjson/issues/206

[wiardvanrij]

My "workaround" - or perhaps this should just be the way you should do it, is to go from:

gjson.Get(json, "panels.#.targets.#.expr")

Towards

gjson.Get(json, "panels.#.targets")

and then do a ForEach over it, and then a new .Get on expr. This works for me. Although it would be super cool to not having to loop over all the elements.

[tidwall]

I was playing around with this how this would work. The issue is that two '#' will return an array of an array, but the Paths require that the number of paths that are returned are the same as the number of items in the outer array.

Basically for this json:

{
  "panels": [
    { "id": 1, "targets": [{"expr":"x1"},{"expr":"x2"}] }, 
    { "id": 2, "targets": [{"expr":"x3"},{"expr":"x4"}] },
    { "id": 3, "targets": [{"expr":"x5"},{"expr":"x6"}] }
  ]
}

This path:

panels.#.targets.#.expr

Returns:

[["x1","x2"],["x3","x4"],["x5","x6"]]

Running a Paths on the result yields an array of three empty paths because the there are only three outer items (["x1","x2"] + ["x3","x4"] + ["x5","x6"]), and those are not directly from the original json, but rather derived from the nested #.

To know that the desired paths are nested would require Paths to dig a little deeper. But how many paths are expected by the user? And how do we know how to map them to the internal values?

I whipped up a branch of gjson that digs deeper and returns all of the correct paths. In this case:

panels.0.targets.0.expr
panels.0.targets.1.expr
panels.1.targets.0.expr
panels.1.targets.1.expr
panels.2.targets.0.expr
panels.2.targets.1.expr

Which seems right, but when I use a weirder json doc like:

{
  "panels": [
    { "id": 1, "targets": [{"expr":["x1"]},{"expr":"x2"}] }, 
    { "id": 2, "targets": [{"expr":"x3"},{"expr":["x4"]}] },
    { "id": 3, "targets": [{"expr":"x5"}] }
  ]
}

[[["x1"],"x2"],["x3",["x4"]],["x5"]]

panels.0.targets.0.expr.0
panels.0.targets.1.expr
panels.1.targets.0.expr
panels.1.targets.1.expr.0
panels.2.targets.0.expr.0

And I got a little confused with the ordering.

I wonder it if wouldn't be better to have a "mapping" style Paths that returns a real json doc that does nested mapping. Where the above would look like:

[
  [["panels.0.targets.0.expr.0"],"panels.0.targets.1.expr"],
  ["panels.1.targets.0.expr",["panels.1.targets.1.expr.0"]],
  ["panels.2.targets.0.expr.0"]
]

I dunno. More thinking is needed. 🤔

[wiardvanrij]

Thanks! I'm also not sure if it's a valid use-case. In the end I've solved it. Granted it doesn't look that neat.

[lammel]

I'd say it is a valid use case ;-) We are quite often in the need to cleanup IDs from deeply structured JSON objects. Mixed with maps and arrays.

We needed to write a helper that will split up our paths at ".#." and traverse the array of path segemnts deeper to gather all paths matching a list of fields. The list is then fed to sjson to delete (or update) those paths.

Having support for multiple # operators would be very valuable for us. Although we have implemented our workaround for now, having direct support would be great.

The approach above (using an array of array for Indexes) was exactly what I had in mind. @tidwall Could you turn your branch into a draft PR? I'd be willing to join in and work on that too, to get that into gsjon

[tanmaymishu]

My "workaround" - or perhaps this should just be the way you should do it, is to go from:

gjson.Get(json, "panels.#.targets.#.expr")

Towards

gjson.Get(json, "panels.#.targets")

and then do a ForEach over it, and then a new .Get on expr. This works for me. Although it would be super cool to not having to loop over all the elements.

Hi, @wiardvanrij and @lammel could you please share a gist or code snippet of your workaround?

[mirobertod]

I'd say it is a valid use case ;-) We are quite often in the need to cleanup IDs from deeply structured JSON objects. Mixed with maps and arrays.

We needed to write a helper that will split up our paths at ".#." and traverse the array of path segemnts deeper to gather all paths matching a list of fields. The list is then fed to sjson to delete (or update) those paths.

Having support for multiple # operators would be very valuable for us. Although we have implemented our workaround for now, having direct support would be great.

The approach above (using an array of array for Indexes) was exactly what I had in mind. @tidwall Could you turn your branch into a draft PR? I'd be willing to join in and work on that too, to get that into gsjon

Hi @lammel , could you please share a gist or code snippet of your workaround? Thanks!

[manifestori]

I'm using this lib indirectly, as part of go-snaps, ofc I need to query deeply nested array of array (of arrays..) values. I don't have control over how this is implemented.

I tried digging into the sources to fix it, but I need some directions. nested "#" would be extremely helpful and useful.