[datafusion-spark] Implement `size` function

[iajoiner]

Is your feature request related to a problem or challenge? Please describe what you are trying to do. There is a user on Discord that wants size akin to the PySpark function with the same name to be implemented.

Describe the solution you'd like A clear and concise description of what you want to happen. Implement size as a BuiltinScalarFunction for ListArray, LargeListArray and MapArray. Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.

[comphead]

btw, there is no size function in PG. There are cardinality function or array_length. Its again should we support both spark/PG syntax ?

[alamb]

Its again should we support both spark/PG syntax ?

I would personally suggest supporting postgres syntax unless postgres doesn't support the feature -- and in that case we could follow spark

[iajoiner]

@alamb Sure! So I guess array_length should be the correct name here.

[iajoiner]

@alamb Wait. In Postgres https://www.postgresql.org/docs/15/functions-array.html it is necessary to have compatibility to have nested arrays so it does make sense to have array_length(array, dimension). For us this makes less sense since we can have arrays that look like [[2, 3], [4, 5, 6]] and even lists of maps. cardinality does make more sense but it doesn't suit our current use case. size from Spark probably makes the most sense here.

[Jefffrey]

I'll rebrand this as a Spark function issue.

Reference: https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.size.html

We want to support size() function on array + map types. Though we do have array_length already, this is meant for arrays so can be confusing to extend it to support maps.

[Jefffrey]

I'll mark as good first issue, some pointers:

• See existing Spark function PRs to see what needs to be done for adding Spark functions, e.g. #18018
• Can see existing array_length implementation in DataFusion for reference as I mentioned above; can ignore all the code around handling dimensions though since Spark size doesn't do that

[CuteChuanChuan]

take

[Yuvraj-cyborg]

@Jefffrey is this issue resolved yet or can I work on it ??

[CuteChuanChuan]

Hi @Yuvraj-cyborg, I am currently working on this. Will submit a PR soon.