dolt icon indicating copy to clipboard operation
dolt copied to clipboard

Feat/support aliyun oss store

Open xujihui1985 opened this issue 1 year ago • 7 comments

support aliyun oss as backend store

xujihui1985 avatar Sep 05 '22 14:09 xujihui1985

@zachmu Hi, this PR is to add support for aliyun oss as backend store same as gcs and aws s3, dolt is an amazing project, and hope I can do some contribution for it.

xujihui1985 avatar Sep 06 '22 01:09 xujihui1985

Thanks for the contribution. We'll look at this today. I think @bheni wrote this code.

timsehn avatar Sep 06 '22 15:09 timsehn

Looks pretty good. My only real issue here is that the environment variables limit the oss remotes that can be added. All oss CreateDB calls will use the same endpoint, key_id and key_secret.

@bheni yes, thanks for reviewing, you are right, because aliyun oss doesn't support config file with different profile, I would like to go with aws strategy that connect to oss with a config file with multiple profile configured, for example

[default]
endpoint=<endpoint>
accessKeyID=<id>
accessKeySecret=<secret>

[prod-profile]
endpoint=<endpoint>
accessKeyID=<id>
accessKeySecret=<secret>

the default credentials file will locate at $HOME/.oss/dolt_oss_credentials, environment variable will still be supported for container environment.

The priority order is environment variable > user provide profile > default profile

dolt remote add --oss-creds-profile prod-profile --oss-creds-config origin oss://[bucket]/key

what do you think?

xujihui1985 avatar Sep 07 '22 02:09 xujihui1985

@xujihui1985 I think that works. If customers show up and have issues with it we can look at other solutions.

bheni avatar Sep 07 '22 22:09 bheni

ok, I'm working on it

xujihui1985 avatar Sep 08 '22 01:09 xujihui1985

@xujihui1985 I think that works. If customers show up and have issues with it we can look at other solutions. @bheni It's busy last week, I implement oss store with oss cred file config, please have a look

xujihui1985 avatar Sep 18 '22 09:09 xujihui1985

@bheni Hi bheni, would you like to take a look at this PR and give some advice?

xujihui1985 avatar Sep 22 '22 01:09 xujihui1985

LGTM. I can't really test out the oss specific code. If oss users have issues I'll reach out to you.

of course, I'm glad to be a part of this project, and I will continue to contribute to this project.

xujihui1985 avatar Sep 25 '22 01:09 xujihui1985

Running our tests. If it all passes, I'll merge and it'll go out in the next release. Thanks for the contribution!

timsehn avatar Sep 26 '22 18:09 timsehn

Thank you for the contribution @xujihui1985!! 🙏 Very appreciated!

Test run looked good, so I went ahead and merged this in to main. This will go out with the next release of Dolt, later this week. 🎉

fulghum avatar Sep 26 '22 20:09 fulghum