Call a function on every element of a complex value

[philrz]

Repro is with Zed commit efdcd27.

Imagine you have an complex value such as the array of numbers [1, 4, 9, 16] and you want to execute some function on each element to produce a new array, such as multiplying each element by two. To accomplish this in Zed today requires a multi-part idiom:

$ zq -version
Version: v1.8.0-5-gefdcd272

$ echo '[1, 4, 9, 16]' | zq -z 'over this | yield this*2 | collect(this)' -
[2,8,18,32]

Compare this to something like JavaScript's map() (not to be confused with Zed's map type) which can perform the equivalent in one shot. We've recognized Zed should offer a similar shorthand.

While in JavaScript we see this approach applied to array types, it seems that in Zed it could be used for not only arrays but also sets, maybe maps, and perhaps into records (recursively?) though I'm not sure. Implementation TBD, I suppose.

[nwt]

#4806 implements a solution for arrays and sets but for not maps or records.

[philrz]

Here's verification in Zed commit 3a40788 of the map() function added in #4806 that handles this for arrays and sets.

Revisiting the original example above, instead of the over / collect() combination we had to use previously, now we can do:

$ zq -version
Version: v1.10.0-31-g3a40788b

$ echo '[1, 4, 9, 16]' | zq -z 'func mult_two(x): ( x * 2 ) yield map(this, mult_two)' -
[2,8,18,32]

Of course, built-in functions work as well.

$ echo '[1, 4, 9, 16]' | zq -z 'yield map(this, string)' -
["1","4","9","16"]

However, while verifying, I did spot what I think is a bug that's now tracked in #4855. I've also proposed some minor doc fixes in #4856.

Thanks @mattnibs!

[philrz]

Here's an update as of current super commit 4493acd. In addition to having addressed the bug linked from the last comment above, there's now building blocks for a flexible approach to performing this operation on hierarchical inputs that may be a mix of arrays and records. This is shown in the recursive-subquery.yaml test that was recently added as part of #6325 and it will also be highlighted in the updated book docs that are currently being drafted.

Positioning the same values from our examples above as array elements and record "leaf" values at differing locations in a nested structure, we see:

$ super -version
Version: 4493acd53

$ echo '
{
    "x": [
        1,
        4,
        {
            "y": 9
        },
        {
            "a": {
                "b": {
                    "c": [
                        {
                            "d": 16
                        }
                    ]
                }
            }
        }
    ]
}
' | super -J -c "
fn walk(v):
  case kind(v)
    when 'array' then 
      [unnest v | walk(this)]
    when 'record' then 
      unflatten([unnest flatten(v) | {key,value:walk(value)}])
    else v*2
  end
values walk(this)" -

{
    "x": [
        2,
        8,
        {
            "y": 18
        },
        {
            "a": {
                "b": {
                    "c": [
                        {
                            "d": 32
                        }
                    ]
                }
            }
        }
    ]
}

As flatten is not yet supported for map types (#6050) I'll continue to hold this one open until that's addressed and we can demonstrate the same for map types.