🚀 Feature: Add support for Cloudflare R2 Storage

[stnguyen90]

🔖 Feature description

Cloudflare R2 Storage is an S3-compatible distributed object storage with generous pricing and a free tier. It would be great if Appwrite could make use of this storage provider.

🎤 Pitch

A storage provider with a free tier would make switching to and testing a cloud storage provider easier.

Related issue:

• https://github.com/utopia-php/storage/issues/52

👀 Have you spent some time to check if this issue has been raised before?

• [X] I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• [X] I have read the Code of Conduct

[gewenyu99]

If it's S3 compatible, can it be supported rn by using the configurable ENV Vars? (I'm genuinely curious as a sanity check).

[stnguyen90]

If it's S3 compatible, can it be supported rn by using the configurable ENV Vars? (I'm genuinely curious as a sanity check).

The way utopia storage works right now, a new Device needs to be implemented to change some parameters:

1. Backblaze
2. DOSpaces
3. Linode
4. Wasabi

The region constants for each provider are also a little nice to make sure you're not using the wrong region.

Then, the Appwrite code needs to be updated to use the right device based on the _APP_STORAGE_DEVICE env var.

That said, there is an issue for creating a generic S3 compatible device. And, I've seen a PR for a generic S3, but it doesn't look like it implements 1 S3 device that you can use for all S3 compatible providers.

[gewenyu99]

Cool, I always learn a lot from talking to you :P

[Meldiron]

@gewenyu99 I think we had PR to do that at some point before 0.14. Not sure why it was not used.

[runesign]

I have the same wish!

[doublevcodes]

Hello, I'd like to have a go at implementing this!

[stnguyen90]

@doublevcodes, thanks for your interest! 🙏 Happy hacking!

[doublevcodes]

So as I understood at it, I need to create 2 PRs? One to utopia-php/storage and one to this repo

[Meldiron]

@doublevcodes That is correct 😇 You can take a look at this Hacktoberfest issue that we created, it's on the same topic but different adapter. You can use it's description to learn more: https://github.com/appwrite/appwrite/issues/3991

[doublevcodes]

Hi, I'm a bit confused on how to implement this because of how Cloudflare R2 bucket URLs are structured.

So far I have something like this:

  public function __construct(string $root, string $accountId, string $accessKey, string $secretKey, string $bucket, string $region = self::AUTO, string $acl = self::ACL_PRIVATE)
  {
      parent::__construct($root, $accessKey, $secretKey, $bucket, $region, $acl);
      $this->headers['host'] = $accountId . '.' . 'r2' . '.' . 'cloudflarestorage' . '.' . 'com' . '/'. $bucket;
  }

However I'm sure that this wouldn't form a valid Host header due to the slash. How would I go about this?

[stnguyen90]

@doublevcodes, oye that's unfortunate Cloudflare does it differently...It looks like the implementation for Cloudflare won't be the same as the other S3 compatible providers like Linode. You'll need to figure out what else needs to be overridden in the base S3 class and implement it in the Cloudflare class.

[doublevcodes]

Ah. Thanks, I'll have a look at what I can do.

[stnguyen90]

@doublevcodes, FYI, virtual-hosted style paths should also be supported in Cloudflare

[pietrodicaprio]

Hi, I'm a bit confused on how to implement this because of how Cloudflare R2 bucket URLs are structured.

So far I have something like this:

  public function __construct(string $root, string $accountId, string $accessKey, string $secretKey, string $bucket, string $region = self::AUTO, string $acl = self::ACL_PRIVATE)
  {
      parent::__construct($root, $accessKey, $secretKey, $bucket, $region, $acl);
      $this->headers['host'] = $accountId . '.' . 'r2' . '.' . 'cloudflarestorage' . '.' . 'com' . '/'. $bucket;
  }

However I'm sure that this wouldn't form a valid Host header due to the slash. How would I go about this?

Hello there, I was looking around for R2 support... You are not forced to keep the bucket at the end of the URL. You should use the URL without the bucket name and then use the name as usually done with standard S3 implementation.
Here a python example using the official boto3 library for AWS: having as URL provided by Cloudflare https://blahblahblahblahblahblah.r2.cloudflarestorage.com/bucket-name-here

s3 = boto3.client('s3',
                      aws_access_key_id=key,
                      aws_secret_access_key=secret,
                      endpoint_url="https://blahblahblahblahblahblah.r2.cloudflarestorage.com")
 with open("some_file_name.png", "rb") as f:
        s3.upload_fileobj(f, "bucket-name-here", "some_file_name.png")

hope this helps a bit

[eldadfux]

Thank you everyone for celebrating Hacktoberfest 22 with us! This issue will now be closed as we're getting ready to celebrate Hacktoberfest 23.

[pietrodicaprio]

Dear @eldadfux I'm not sure this issue is related to an Hacktoberfest. It is a general request for the product's improvement

[gewenyu99]

Dear @eldadfux I'm not sure this issue is related to an Hacktoberfest. It is a general request for the product's improvement

Our team opened this task for Hacktoberfest :P @stnguyen90 is a part of our team. AFAIK, he doesn't actually want this feature :D

[pietrodicaprio]

Dear @eldadfux I'm not sure this issue is related to an Hacktoberfest. It is a general request for the product's improvement

Our team opened this task for Hacktoberfest :P @stnguyen90 is a part of our team. AFAIK, he doesn't actually want this feature :D

Well, https://github.com/utopia-php/storage/pull/76 is a thing and @stnguyen90 requested a review 3 weeks ago so it looks like to be almost ready to be merged?

[gewenyu99]

Dear @eldadfux I'm not sure this issue is related to an Hacktoberfest. It is a general request for the product's improvement

Our team opened this task for Hacktoberfest :P @stnguyen90 is a part of our team. AFAIK, he doesn't actually want this feature :D

Well, utopia-php/storage#76 is a thing and @stnguyen90 requested a review 3 weeks ago so it looks like to be almost ready to be merged?

Yep, that will be a generic adaptor, too. So this will be supported in some future version along with all S3 compatible storage services 🤞

[AndrewBucklin]

Could we re-open this? Cloudflare R2 provides much better costs than AWS and it's S3 compliant. It already use it in other projects by simply entering the Cloudflare R2 storage details where the AWS S3 information is usually entered.

[JosefJezek]

please re-open this

[Tushar98644]

@stnguyen90 Hey , can i take up the issue ?

[Haimantika]

@stnguyen90 Hey , can i take up the issue ?

Assigning it to you. Please remember to update us every 3 days and if you have any questions, do not hesitate to reach us out on Discord

[Tushar98644]

@stnguyen90 @gewenyu99 any updates on my pr?

[tessamero]

@Tushar98644 the PR is still awaiting your follow up before we can merge

[docimin]

any update? I think those 3 days are kind of over

[docimin]

@tessamero Is this implemented yet?

[tessamero]

@docimin I closed the issue as the PR did not provide any proof it works as they wrote the code without Cloudflare access. They need to provide proof of tests passing for it to be merged and they are unable to. We are going to close the PR.

[schneidermr]

@tessamero I'd like to reopen this issue because here is the implementation that can solve also this issue: https://github.com/appwrite/appwrite/pull/7172 I've already tested it with R2 and working as expected.

[tessamero]

@schneidermr , reopened the issue, thank you for submitting a PR :)