delta-io
[WIP] add support for show partitions command
Description
This PR adds support for SHOW PARTITIONS command to list the partition names of a delta table.
Resolves #996
How was this patch tested?
Tests to test this PR were added in ShowDeltaPartitionsSuite
Does this PR introduce any user-facing changes?
No
@zsxwing this PR is not finished yet (the partition spec is not supported yet). But I wanted to double-check the implementation with you since I also introduced some changes to the Describe Detail command to avoid duplicating code. Please let me know if there are changes that I need to do before tackling the partition spec work.
I am not sure if we can not support this sql command using path.
If so, we don't need to extend the antlr grammar and sql parser and can reuse the spark syntax directly. and make the DeltaTableV2 to extend the SupportsPartitionManagement interface.
@YannByron We have to create our own parser as we don't plan to follow Spark's current show partitions format. It's coming from legacy hive format and it's not user friendly. See #996 for the examples I put in the description.
@zsxwing thanks for you explanation. I see delta defines the new output format for show partitions and that's nice.
My team could certainly make use of this feature - is there anything I can do to help move it along ?
@nate-kuhl This PR doesn't seem to be active, you're welcome to pick up this work instead. In my review above I outlined a refactoring I think should be done first before implementing this if you would like to work on it.
@nate-kuhl since we need to do a refactoring (2 PRs) you should create a new PR. Feel free to open the PR early and ask on it if you have any questions