gravitino icon indicating copy to clipboard operation
gravitino copied to clipboard

Published 20 hours ago • verified publisher icon apache

[FEATURE] Add S3 support for Fileset Hadoop catalog

Open jerryshao opened this issue 9 months ago • 3 comments

Describe the feature

Fileset is a new concept brought in 0.5.0 to manage the non-tabular data, the current implementation uses HCFS to manage the physical data. With HCFS, the Hadoop catalog should support different underlying storage, but currently we only verified local file system and HDFS.

In this issue, we should also support S3, to make the fileset hadoop catalog work with S3 object store.

Motivation

The reason to support S3 is that it is vastly used on the public cloud, we should add this support anyway.

Describe the solution

No response

Additional context

No response

jerryshao avatar May 14 '24 03:05 jerryshao

I think we can change this feature to Support Object Store provided by Cloud Service, so we can add subtask to support Azure Blob and Aliyun OSS

zhoukangcn avatar May 14 '24 04:05 zhoukangcn

@xiaozcy can you please leave a message here, so I can assign the issue to you.

jerryshao avatar Aug 01 '24 11:08 jerryshao

@xiaozcy can you please leave a message here, so I can assign the issue to you.

Sure.

xiaozcy avatar Aug 01 '24 13:08 xiaozcy