Why not use the profile name when initialising the S3FileSystem class?

[wudihero2]

Question

Hi, In #922 standardize some AWS credential names, but I am confused why not use below code to use aws profile name at pyiceberg.io.fsspec.py?

profile_session = AioSession(profile="xxx")
fs = s3fs.S3FileSystem(session=profile_session)

It supports the use of AWS profile names like `glue.profile-name' as in the following code at pyiceberg.catalog.glue.py

profile_name=PropertyUtil.get_first_property_value(properties, GLUE_PROFILE_NAME, DEPRECATED_PROFILE_NAME),

Maybe it would be a better improvement if we could use profile name or is there some concern I haven't considered?

[kevinjqliu]

I think this is a feature gap on the S3 FileIO. It makes sense to support profile_name. We would need to support both fsspec and pyarrow

Is this something you would like to contribute?

[wudihero2]

Hello, I am interested in this, do I need to tag the person who will assign this task to me?

[kevinjqliu]

@wudihero2 assigned to you :)

[sungwy]

Hi folks, I was under the impression that this was something that would need to be addressed in PyArrow S3FileSystem. Please see @HonahX 's earlier comment:

https://github.com/apache/iceberg-python/pull/922#discussion_r1677395543

Previous issue on same topic: https://github.com/apache/iceberg-python/issues/570 PyArrow S3FileSystem: https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html

On that note, I'm in favor of keeping this issue open since this is a frequent question from the community, until we are able to find a solution by perhaps working with the Arrow community.

[kevinjqliu]

https://github.com/apache/iceberg-python/issues/1104#issuecomment-2377397379 this thread is somewhat related

[wudihero2]

Hi all, I checked the code of pyarrow and found that the profile_name parameter is not currently supported. The s3.* related parameters are indeed not suitable for supporting profile_name. It would be great if we could work with the Arrow community to find a solution! This package is great and can help me use iceberg for my current job, thank you !!