spark-rapids icon indicating copy to clipboard operation
spark-rapids copied to clipboard
verified publisher icon NVIDIA

[FEA] [SPARK-42918] Generalize handling of metadata attributes in FileSourceStrategy

Open revans2 opened this issue 2 years ago • 2 comments

Is your feature request related to a problem? Please describe. Metadata field handling was made more generic. We do interact with these classes in our shim layer and there is a possibility that we will need to update this code.

revans2 avatar Jul 28 '23 20:07 revans2

Need to verify that we fallback properly

mattahrens avatar Aug 08 '23 20:08 mattahrens

The two classes from this change that we rely on are FileFormat and FileSourceMetadataAttribute. We should get these changes for free IMO. @revans2 is my assessment correct?

razajafri avatar Mar 19 '24 16:03 razajafri