Custom Authentication & External File Systems

[andygrove]

What is the problem the feature request solves?

We need to support custom authentication with external file systems

(Access hdfs/hadoop-aws via JNI)

1. HDFS support via fs-hdfs

• [x] Fork into Comet
• [ ] PRs open in fs-hdfs
• [ ] One draft PR
• [ ] Update Comet to use internal fork

2. OpenDAL

• [ ] #2367

Describe the potential solution

No response

Additional context

No response