Vortex expression partitioning and simplification should support custom partitioning or simplifier-rules

[danking]

Describe the bug

Consumers should have the opportunity to provide simplification rules or to participate in expression partitioning.

---

Vortex expression partitioning (e.g. in Struct Layout) critically relies on the simplifier (typed_simplify) to discover independent parts of the expression.

For example, the following expression is easy to partition and, critically, many columns are not referenced:

merge(
    pack([
        ("a", ...)
        ("b", ...)
    ]),
    pack([
        ("c", ...)
    ]),
)

However, if we insert an arbitrary, user-defined, expression:

merge(
    NEW_EXPR(pack([
        ("a", ...)
        ("b", ...)
    ])),
    pack([
        ("c", ...)
    ]),
)

This entire expression, AFAICT, becomes non-partition-able. I think we read every column from disk and evaluate the whole expression in-memory, which is slow and costly.

To Reproduce

No response

Expected behavior

No response

Additional context

No response

[joseph-isaacs]

We are working on both adding a pluggable simplifier and using the simplifier to partition.