new feature: allow inject a customized signer for aws s3

[flaneur2020]

Feature Description

background: https://github.com/apache/iceberg-rust/issues/506

It seems we need to implement this signer in iceberg-rust and pass it as a customized signer to opendal.

Problem and Solution

but currently the signer is a struct AwsV4Signer.

we need make the signer in S3Core a trait before allowing it customizable.

maybe we can defined a trait called S3Signer in opendal, and make AwsV4Signer as it's implementation by default.

Additional Context

No response

Are you willing to contribute to the development of this feature?

• [X] Yes, I am willing to contribute to the development of this feature.

[flaneur2020]

to make the Signer customizable, i've got some difficulties to solely add it in the opendal part.

adding a trait called S3Signer in opendal requires a signature to be object safe, but a signature like fn sign<T>(&self, req: &mut Request<T>, cred: &AwsCredential) -> Result<()> is not object safe because it contains a generic parameter inside it.

so i think there might have two ways to make it possible:

1: add a GenericSigner trait with fn sign(&self, req: &mut dyn SignableRequest, cred: &AwsCredential) -> Result<()> in reqsign.

but as Signer accepts &mut impl SignableRequest, we may have to change the api of Signer to allow it accept trait objects.

2: change Signer in reqsign into an enum like:

pub enum Signer {
  V4(V4Signer),
  Custom(Box<dyn CustomSigner>)
}

this approach can make the api of Signer unchanged, but would introduce a small dispatch cost on every sign.

(i'd prefer the second approach, it seems less intrusive)

@Xuanwo any suggestion about this?

[Xuanwo]

Hi, S3::customized_credential_load should allow you to implement your own logic.

[flaneur2020]

Hi, S3::customized_credential_load should allow you to implement your own logic.

imho customized_credential_load is to customize the credential logic, but here we need customize is the signing logic.

the Signer currently is not a trait like dyn AwsCredentialLoad yet.

[Xuanwo]

Thank you for the clarification. I now understand what the problem is.

My current ideas:

• Add an Sign trait for entire reqsign

  se http;
  
  [async_trait]
  rait Sign: 'static {
  type Credential;
  
     async fn sign(&self, req: &mut http::request::Parts, cred: &Self::Credential) -> Result<()>
  
  

• Expose a Signer from the reqsign

  reqsign will expose a type AwsSigner = Arc<dyn Sign<Credential=AwsCredential>>.

• OpenDAL can accept any impl Sign<Credential=AwsCredential> with AwsV4Signer as default impl.

[rshkv]

@Xuanwo, I'm slightly confused about the the state of reqsign with respect to configurable signers.

The new Sign trait was introduced in https://github.com/Xuanwo/reqsign/pull/459 and then replaced in https://github.com/Xuanwo/reqsign/pull/482 with the Load and Build traits which should achieve the same.

I'm confused about how reqsign is published (new to Rust). The 0.16.1 crate docs have an AwsV4Signer which I don't find in the code. And the 0.16.1 crate docs don't have Load or Build traits. (But I also can't find the reqsign_core and the reqsign_aws_v4 crates published so I think I'm missing something.)

[meteorgan]

@rshkv The code in v0.16.x hasn't been merged into main. Maybe that's the issue.

[Xuanwo]

The new Sign trait was introduced in Xuanwo/reqsign#459 and then replaced in Xuanwo/reqsign#482 with the Load and Build traits which should achieve the same.

Hi, that will be included in the 0.17 release. (which is not ready yet)

[rshkv]

Ah, gotcha. Thank you for the quick reply.