Cannot upload any file over 6MB to self-hosted supabase storage via dashboard

[fgh-james]

Bug report

• [x] I confirm this is a bug with Supabase, not with my own application.
• [x] I confirm I have searched the Docs, GitHub Discussions, and Discord.

Describe the bug

When uploading a file greater than 6MB in size to storage via the supabase dashboard in a self-hosted instance I receive the following error:

Failed to upload 6_3mb.pdf: tus: failed to resume upload, caused by [object ProgressEvent], originated from request (method: HEAD, url: http://localhost/upload/resumable/cHN5Y2hvc29jaWFsX2NvbnRlbnQvRkdIIGJ1c2luZXNzIGNhcmRfMjAyNDA4MTkgY29weSA2LnppcC9hMjNmYTM5Yy1kZmI0LTRhMjUtYjQzYS1iMDU2OGY2MGM4NGU, response code: n/a, response text: n/a, request id: n/a)

While the file is uploading, I receive this error multiple times in the browser console

HEAD http://localhost/upload/resumable/cHN5Y2hvc29jaWFsX2NvbnRlbnQvRkdIIGJ1c2luZXNzIGNhcmRfMjAyNDA4MTkgY29weSA2LnppcC9hMjNmYTM5Yy1kZmI0LTRhMjUtYjQzYS1iMDU2OGY2MGM4NGU net::ERR_CONNECTION_REFUSED

I made sure to pull the latest docker images. This does not resolve the error.

To Reproduce

1. Create a file that is 6MB in size
2. Go to the self-hosted supabase dashboard: localhost:8000
3. Go to storage, and upload the file
4. It will show its uploading for a moment, then error out with the error above.

Expected behavior

Files up to 50MB should be able to be uploaded.

Screenshots

While uploading

Error

System information

• OS: macOS
• Browser (if applies): chrome
• Version of supabase docker compose: 3.8
• Version of Node.js: v18.19.0

Additional information

• I have tried this with other files, not just the one I am testing with. I receive the same error for anything over 6MB

[cjlryan]

I'm also having this issue, @fgh-james that your console error is also missing the port used for uploading on local, same as mine (usually it would be http://localhost:54321 rather than just http://localhost).

This looks similar to this bug: https://github.com/supabase/supabase/issues/15703

A potential temporary solution might be to downgrade a couple of versions, mine only stopped working after updating to the latest one (CLI version 1.192.5, Storage API 1.11.0)

[cjlryan]

@fgh-james so... I fixed this for me.

It may be a mismatch between different docker image versions. I deleted then reinstalled all the docker images and it works perfectly again.

I hope that works for you!

[yaschet]

I'm experiencing the same problem as mentioned in this issue: #563 I tried the solution suggested by @cjlryan but it didn't work.

[yaschet]

Update: After a thorough investigation, I successfully resolved the issue in my case. The problem was traced back to the supabase/config.toml file, where TLS was enabled. Disabling TLS definitively fixed the issue for me. You can find the solution here: my comment.

[djsisson]

storage is returning an invalid location header

request sent to https://supabase.domain.com/storage/v1/upload/resumable response contains Location header of http://supabase.domain.com:8000/upload/resumable/............ this is then set in local storage where uploads are then tried to send to:

this then causes an error either due to content mismatch (trying to post to http from https or 404 since its trying to send to an invalid port plus missing /storage/v1)

i assume the bug is in here? https://github.com/supabase/storage/blob/f950fc4a6afd3cbde39d4e7f35b9df42b895b4dc/src/http/routes/tus/lifecycle.ts#L86 the request comes in from kong which is http.. :8000 and no path prefix of /storage/v1

[rickypid]

I also have the same problem reported by @djsisson

[djsisson]

I want to clarify:

This works fine if you call the storage container directly, however if your containers are behind a proxy (like kong, or even a double proxy, then this method of generating a URL doesn't work) The Simplest solution is to have an optional ENV like TUS_URL where we can set the full url ourselves and it just appends the id onto the end.

However there is a work around, depending on your current setup this is what needs to be done:

if you are behind a proxy, you need to move Kong onto port 80 to prevent the code from appending a port onto the url, so add this to your kong compose:

      - 'KONG_PROXY_LISTEN=0.0.0.0:80 reuseport backlog=16384, 0.0.0.0:443 http2 ssl reuseport backlog=16384'

This is the defaults but moved to 80/443

update the following in the studio section

- 'SUPABASE_URL=http://supabase-kong:80'

since kong is no longer on 8000

Next add the following to the storage compose section

- NODE_ENV=production

This will force the url to be https, regardless if the proxy you are using is http or https

Lastly you need to add a section to the kong yaml file to filter /uploads to storage as well

  ## Storage routes: the storage server manages its own auth
  - name: storage-tus
    _comment: 'Storage: /upload/* -> http://supabase-storage:5000/upload/*'
    url: http://supabase-storage:5000/
    routes:
      - name: storage-tus-all
        strip_path: false
        paths:
          - /upload/
    plugins:
      - name: cors

with this, storage now generates valid location header urls (albeit missing /storage/v1) but that is picked up by the added section in kong.yml

[rickypid]

Thank you @djsisson for your quick reply!!!

My situation is as follows: Client Dart -> Ngnix Rever Proxy -> Supabase Self-Hosted [ Kong -> Storage ]

I have the same problem as described above: I make a request on https://myhost/storage/v1/upload/resumable, in the Location Header of the response I get http://myhost/storage/v1/upload/resumable/dGVzdC1leHBsb3Jlci90ZXN0LzQxMTQ3OTctdWhkXzM4NDBfMjE2MF8yNWZwcy5tcDQvODdlZDFkODYtYjZhMS00NTJhLWJkOGEtMmY3MTdlYzk1NTAx

[rickypid]

I solved it by inserting in my proxy rever configuration 🍾 🍾 🍾 :

proxy_redirect http://myhost:8000/ https://myhost/;

[rallisf1]

I spend a full day on this, tried every possible fix I could find. I couldn't get rid of the 8000 port number in the request, so I used it! My dirty hack:

kong.yml

  ## Storage routes: the storage server manages its own auth
  - name: storage-v1
    _comment: 'Storage: /storage/v1/* -> http://storage:5000/*'
    url: http://storage:5000/
    routes:
      - name: storage-v1-all
        strip_path: true
        paths:
          - /storage/v1/
    plugins:
      - name: cors
      - name: request-transformer
        config:
          add:
            headers:
              - 'Forwarded: host=\$(headers.host)/storage/v1;proto=https'

  ## Fix Storage large uploads
  - name: storage-fix
    _comment: 'Storage: /storage/v1:8000/* -> http://storage:5000/*'
    url: http://storage:5000/
    routes:
      - name: storage-fix-all
        strip_path: true
        paths:
          - /storage/v1:8000/
    plugins:
      - name: cors

It's ugly AF but it works and I can finally go to sleep.

P.S. Kong might be a great Enterprise choice but the FOSS version is limited and most people are not familiar with it. I'm pretty sure this bug is left in here on purpose (you can find issues over a year ago!) while the SaaS uses request-transformer-advanced to fix this in a single rule...

[GrizliK1988]

looks like it's already fixed in storage, but has not yet been fully updated to docker-compose of studio.

which this patch upload works for me locally

diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml
index 847aae8cef..47ea2b7465 100644
--- a/docker/docker-compose.yml
+++ b/docker/docker-compose.yml
@@ -225,7 +225,7 @@ services:
   # To use S3 backed storage: docker compose -f docker-compose.yml -f docker-compose.s3.yml up
   storage:
     container_name: supabase-storage
-    image: supabase/storage-api:v1.11.13
+    image: supabase/storage-api:v1.13.3
     depends_on:
       db:
         # Disable this if you are using an external Postgres database
@@ -263,6 +263,7 @@ services:
       GLOBAL_S3_BUCKET: stub
       ENABLE_IMAGE_TRANSFORMATION: "true"
       IMGPROXY_URL: http://imgproxy:5001
+      REQUEST_ALLOW_X_FORWARDED_PATH: "true"
     volumes:
       - ./volumes/storage:/var/lib/storage:z

Hope it works for you.

[callmearta]

I managed to fix it by changing storage block in kong.yml config like this:

  ## Storage routes: the storage server manages its own auth
  - name: storage-v1
    _comment: 'Storage: /storage/v1/* -> http://supabase-storage:5000/*'
    url: http://supabase-storage:5000/
    routes:
      - name: storage-v1-all
        strip_path: true
        paths:
          - /storage/v1/
          - /storage/v1:8000/
          - /storage/v1:8000/storage/v1/
    plugins:
      - name: cors
      - name: request-transformer
        config:
          add:
            headers:
              - 'Forwarded: host=\$(headers.host)/storage/v1;proto=https'

I also applied the storage image version change mentioned by @GrizliK1988 but I don't know if it's required to get it working or not. If this didn't work for you on it's own, try to apply his edits as well.

[djsisson]

if you add the following to kong section:

      - 'KONG_PORT_MAPS=443:8000'

as well as the following to the storage section:

REQUEST_ALLOW_X_FORWARDED_PATH: "true"

it all works without the need to alter kongs ports or alter the kong.yaml

ofc you change port mappings as required

[phatdinhvn]

I have to modify as below to work smoothly: set env KONG_PORT_MAPS is 443:8000 will remove :8000 from http://my.domain.com:8000/ set env REQUEST_ALLOW_X_FORWARDED_PATH is true and add header 'Forwarded: proto=https' will forward http://my.domain.com/ to https://my.domain.com/ and set env TUS_URL_PATH is upload/resumable for storage-api to append url https://my.domain.com/storage/v1/upload/resumable

Restart kong and storage container. Clear cookies, caches on your browser.

[djsisson]

@phatdinhvn for Forwarded: proto=https

you can instead set NODE_ENV=production in storage section of compose it will do the same thing

[callmearta]

I wish they add this discussion to the docs. I spent 4 days trying to fix this issue and there weren't many solutions at the time.

[flogy]

For me just adding NODE_ENV=production and REQUEST_ALLOW_X_FORWARDED_PATH: true to supabase-storage as well as KONG_PORT_MAPS: "443:8000" to supabase-kong did work too. I am using supabase/storage-api:v1.22.17 and kong:2.8.1 together with Traefik as reverse proxy.

Important hint during testing: TUS by default stores your generated upload URLs in browser's local storage. Therefore, when changing configurations and then retry uploading the same file, the previous URL is reused, which therefore might not reflect the changes you just made. So don't forget to delete all tus:* local storage entries before retrying!

[codemonkei]

I have to modify as below to work smoothly: set env KONG_PORT_MAPS is 443:8000 will remove :8000 from http://my.domain.com:8000/ set env REQUEST_ALLOW_X_FORWARDED_PATH is true and add header 'Forwarded: proto=https' will forward http://my.domain.com/ to https://my.domain.com/ and set env TUS_URL_PATH is upload/resumable for storage-api to append url https://my.domain.com/storage/v1/upload/resumable

Restart kong and storage container. Clear cookies, caches on your browser.

this but i had to set the TUS_URL_PATH to /storage/v1/upload/resumable in docker-compose.yml for it to work properly

[morandalex]

i solved as follow:

in kong.yml added:

_format_version: '2.1'
_transform: true


plugins: #--> add this plugin
  - name: request-transformer
    config:
      add:
        headers:
          - x-forwarded-proto:https

...
...


  - name: storage-v1
    _comment: 'Storage: /storage/v1/* -> http://storage:5000/*'
    url: http://storage:5000/
    routes:
      - name: storage-v1-all
        strip_path: true
        paths:
          - /storage/v1/

       ###
      - name: storage-v1-resumable #--> add this route
        strip_path: false # Important: The full path is needed by the storage service
        paths:
          - /upload/resumable/
        ###
    plugins:
      - name: cors

in docker compose I added some environemnt variables

in kong docker service I added :

      KONG_NGINX_PROXY_CLIENT_MAX_BODY_SIZE: 100m
      KONG_TRUSTED_IPS: 0.0.0.0/0,::/0

and in storage docker service I added:

      API_EXTERNAL_URL: ${API_EXTERNAL_URL}
      FILE_SIZE_LIMIT: ${FILE_SIZE_LIMIT}

with .env

FILE_SIZE_LIMIT=100000000

eventually if you are using nginx as reverse proxy

client_max_body_size 100M;

then I restarted all and cleaned the cache on the browser

[laughinggaschambers]

i am working on elest.io

i solved as follow:

in kong.yml added:

_format_version: '2.1' _transform: true

plugins: #--> add this plugin

• name: request-transformer config: add: headers: - x-forwarded-proto:https

... ...

• name: storage-v1 _comment: 'Storage: /storage/v1/* -> http://storage:5000/*' url: http://storage:5000/ routes:

  • name: storage-v1-all strip_path: true paths:
    • /storage/v1/
  • name: storage-v1-resumable #--> add this route strip_path: false # Important: The full path is needed by the storage service paths:
    • /upload/resumable/

  plugins:

  • name: cors in docker compose I added some environemnt variables

in kong docker service I added :

      KONG_NGINX_PROXY_CLIENT_MAX_BODY_SIZE: 100m
      KONG_TRUSTED_IPS: 0.0.0.0/0,::/0

and in storage docker service I added:

      API_EXTERNAL_URL: ${API_EXTERNAL_URL}
      FILE_SIZE_LIMIT: ${FILE_SIZE_LIMIT}

with .env

FILE_SIZE_LIMIT=100000000

eventually if you are using nginx as reverse proxy

client_max_body_size 100M;

then I restarted all and cleaned the cache on the browser

can confirm this fixes it for me

[fenos]

Closing as solved! Thanks a lot for the various solutions posted on the thread! 🥇