🚀 Feature Request: Ability to emulate a public R2 bucket when developing locally with Wrangler 3

[VirtualWolf]

trafficstars

Describe the solution

Under Wrangler 2, when running locally any files you'd upload to a "local" R2 bucket would be saved in a directory with the exact filename you gave it at upload time, which meant you could very easily emulate a public R2 bucket by pointing serve at it or something similar.

Unfortunately this approach doesn't work anymore with Wranger 3 (or I guess Miniflare now?) because the filenames are all hashes now and there's an SQLite database involved as well, and as far as I can tell there's no ability to emulate a public bucket except by actually having a public bucket for real in Cloudflare.

It'd be great to have an extra flag when running wrangler dev, maybe something similar to --assets, that could specify that the R2 bucket should act as though it was public.

[jesseditson]

I'm also curious about AWS features in this scenario - is it implied by this issue that there would be a local S3 API available for buckets as well? Or would that be a separate feature? Currently I'm getting around this by having a special dev-only worker path that just serves the files from the Workers R2 API, but I have a separate need for pre-signed URLs that it appears I won't be able to test locally with wrangler3.

[bryanlatten]

This sounds good. +1 on an actual s3 api to use on local (unless i'm missing something)

[205g0]

But wramgler does still emulate R2 locally or what do I miss here? How is accessing the files via the local file system related to an emulation?

[205g0]

Ok, now I get it, apologies for the double post. You want in a second step to serve those files and not only emulate api calls to the bucket. Valid requirement. So there's nothing in wrangler doing that serving part?

[thexeos]

Since it was removed from the Project, does it mean this is no longer considered?

Having something like wrangler r2 bucket serve <NAME> --host 127.0.0.1 --port 1234 would be very helpful. The current workaround is adding asset-serving functionality to a Worker which (a) would not be executed in production yet the code for it will remain in the Worker, and (b) would likely not replicate the behavior of the Public Domain connected to R2 bucket (e.g., serving the httpMetadata headers of the object) unless developer is very thorough.

[treeder]

I think what we're all looking for here is the localhost version of the public bucket URL:

[treeder]

Anyone from cloudflare aware of this or working on it?

[glassworks-projects]

+1!

[eliahilse]

would appreciate as well

[joefitter]

+1 this would be handy

[shmuli9]

This would be super helpful

[prblackburn]

+1 yes please

[marcbejar]

Does anybody know if they are working on it?

[emilienbidet]

I created a workaround little server using Hono and Bun to serve images from a local Cloudflare R2.

Here is the repo :

https://github.com/emilienbidet/cloudflare-r2-dev-server

[treeder]

Nice one @emilienbidet , thanks, will try it out this week!

[jamaluddinrumi]

I created a workaround little server using Hono and Bun to serve images from a local Cloudflare R2.

Here is the repo :

https://github.com/emilienbidet/cloudflare-r2-dev-server

fantastic! very helpful thank you!

[serikshaikamalov]

Alternative way: A path to the stored file should be: /r2/:key for instance /r2/random-file.png for local development

If you navigate to http://localhost:8788/r2/random-file.png:

1. Get the key from the path. The key is random-file.png
2. Finds the local file by key on R2
3. Returns the response(type is octet-stream) in the form of file

PS: http://localhost:8788 is a local port of wrangler dev server

export default {
    async fetch(req, env, ctx) {
        try {
            const url = new URL(req.url);                       
            if (url.pathname.startsWith('/r2/')) {
                // Local R2 storage                
                const [path, key] = url.pathname.split('/r2/')
                console.log("Local r2: ", {path, key});                
                const file = await env.R2.get(key)                
                const headers = new Headers();
                headers.append('cache-control', 'immutable, no-transform, max-age=31536000');
                headers.append('etag', file.httpEtag);
                headers.append('date', file.uploaded.toUTCString());
                return new Response(file.body, {
                    headers
                })
            }

            // Otherwise, serve the static assets.
            // Without this, the Worker will error and no assets will be served.
            return env.ASSETS.fetch(req);
        } catch (ex) {
            console.error('Worker ex: ', ex)            
            return new Response("Internal server error", { status: 500 })
        }
    },
}

[treeder]

@serikshaikamalov that's awesome, just tried it out. Here's the Hono version:

app.get('/r2/*', async (c) => {
    const key = c.req.path.substring('/r2/'.length)
    console.log("Local r2 key: ", key)
    const file = await c.env.R2.get(key)
    if (!file) throw new APIError("file not found", { status: 404 })
    const headers = new Headers()
    headers.append('etag', file.httpEtag)
    return new Response(file.body, {
        headers,
    })
})

[serikshaikamalov]

@serikshaikamalov that's awesome, just tried it out. Here's the Hono version:

app.get('/r2/*', async (c) => {
    const key = c.req.path.substring('/r2/'.length)
    console.log("Local r2 key: ", key)
    const file = await c.env.R2.get(key)
    if (!file) throw new APIError("file not found", { status: 404 })
    const headers = new Headers()
    headers.append('etag', file.httpEtag)
    return new Response(file.body, {
        headers,
    })
})

Coool👍

[lrapoport-cf]

cc @jonesphillip

[franklin-tina]

If you're using Nuxt (and have the nitro-cloudflare-dev module installed),

/**
* `server/api/assets/[filename].get.ts` OR
* `server/routes/assets/[filename].get.ts` (if you don't want the /api endpoint)
**/

export default defineEventHandler(async (event) => {
  const { filename } = getRouterParams(event);

  if (!filename) {
    throw createError({ status: 400, statusMessage: "Bad Request!" });
  }		

  const r2 = event.context.cloudflare.env;
  const file = await r2.get(filename);

  if (!file) {
    throw createError({
      status: 404,
      statusMessage: "File not found!",
    });
  }

  setHeaders(event, { etag: file.etag });

  return file.body;
});

Enjoy 💚

[multiplehats]

Trying to use Workflows and R2 locally is just plain impossible 👎

[Nalci]

+1 for his feature request. I can't test locally a bucket. A simple browser UI would be also great to simulate file uploads like on the webpage.

[treeder]

Use the code shared above.

[sburgpit]

If you're using Next.js with opennext

app/r2/[...key]/route.ts

import { getCloudflareContext } from '@opennextjs/cloudflare'
import { NextRequest, NextResponse } from 'next/server'

export async function GET(_: NextRequest, { params }: { params: Promise<{ key: string[] }> }) {
  if (process.env.NODE_ENV !== 'development') return new NextResponse('Not Found', { status: 404 })

  const { key: keySegments } = await params
  const key = keySegments.join('/')

  const cloudflare = getCloudflareContext()

  try {
    const file = await cloudflare.env.R2.get(key)

    if (!file) {
      return new NextResponse('File not found', { status: 404 })
    }

    const headers = new Headers()
    headers.append('etag', file.httpEtag)

    return new NextResponse(file.body, {
      headers,
      status: 200,
    })
  } catch (error) {
    console.error(`Error fetching key "${key}" from R2:`, error)
    return new NextResponse('Internal Server Error', { status: 500 })
  }
}

[thejufo]

any update from Cloudflare team? This is a very needed feature please consider it.