Feature request: support passing customized connector or connector adapter

[wenym1]

Motivation

We are using AWS S3 or any S3 compatible object store as our storage backend. Previously we use aws-sdk-s3 to access the object store, and recently we are migrating from the aws sdk to opendal.

Previously when we use the aws sdk, it supports passing customized http client (see method http_client). In our code, we build our own http client to apply some customized logic. Most logics can be supported in opendal, such as setting nodelay, keepalive. The only thing left is passing a customized connector. When building a hyper client by the build method, it support passing a customized connector that implements the Connect trait. The trait is accessed when creating a new connection. We leverage this feature to monitor the tcp connection, such as number of existing connection, throughput of each connection etc. We implemented our own connector that wraps the original hyper http connector and impl the Connect trait in https://github.com/risingwavelabs/risingwave/pull/11848. However, in opendal this feature is not supported yet.

Requested feature

I think there are two ways to support the feature.

The first one is to support passing a customized http client or a connector like what the aws sdk does. This may leak too much flexibility to our external users, as it depends on users to properly initialize the connector.

The second one is to support passing a connector adapter with the following definition

trait ConnectorAdapter {
    fn wrap(connector: impl Connect) -> impl Connect;
}

With this adapter trait, the opendal can initial the connector internally, and then wrap the initialized connector with this adapter, and in the end pass it to the http client.

Implementation-wise consideration

It seems that opendal is using reqwest to make the http request. I took a rough look at the code of reqwest. Though its underlying http client is also hyper, in here, it seems that the connector used by the http client is fixed. So to support this feature, we shall either modify the reqwest code to support passing a customized connector, or change to make the http request from the raw hyper client.