spark icon indicating copy to clipboard operation
spark copied to clipboard

[MINOR][SQL] Remove an unnecessary parameter of the PartitionedFileUtil.splitFiles

Open weixiuli opened this issue 2 years ago • 3 comments

What changes were proposed in this pull request?

Remove an unnecessary parameter of the PartitionedFileUtil.splitFiles

Why are the changes needed?

Make code clearer.

Does this PR introduce any user-facing change?

'No'

How was this patch tested?

Pass GitHub Actions

weixiuli avatar Sep 13 '22 09:09 weixiuli

Can one of the admins verify this patch?

AmplabJenkins avatar Sep 14 '22 06:09 AmplabJenkins

Seems OK. There's no reason to expect external code would call this method right?

Although this is not a public api, it is still used by third-party projects based on Spark, for example:

https://github.com/NVIDIA/spark-rapids/blob/fb86a1a8042f241b31d29f2e48ef73820be734d7/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuFileSourceScanExec.scala#L479-L485

https://github.com/oap-project/gluten/blob/2e0f16bbdbba4edd70846123147e24d0b90ce833/jvm/src/main/scala/io/glutenproject/utils/InputPartitionsUtil.scala#L43-L51

LuciferYang avatar Sep 16 '22 14:09 LuciferYang

OK let's leave it if there's any doubt - just not worth messing with libraries that use even non-public APIs

srowen avatar Sep 16 '22 14:09 srowen