googleapis
Uploading multipart/form-data "client intended to send too large body"
We're using a custom docker image with 15.11.0-alpine. We're developing a video upload service for a client.
Large videos 10-20-30GBs goes to our express server via multipart/form-data.
When it lands we have a video service that scans the videos for illegal material before we are liable for storing them; it acts as a temporary in-between. We use ML, and that works great.
Now; the issue is when we try to upload large files, we get: the log client intended to send too large body: 46098280 bytes
I've looked into this, and there seems to be some nasty limitation on the App Engine API.
There are no docs on the BlobStore API for NodeJS which has the same limitation of 32mb. Are clients expected to upload illegal material directly from their browser into our storage buckets? (really bad) It has the potential to break GDPR since if the client can upload from the browser, then leave, that's a serious data storage issue and data fragmentation issue.
Does anyone know a solution that's viable here via App Engine?
We really want to stay inside the GCE but looking like we might have to migrate to Digital Ocean at this rate; 32MB file size limit is crazy stupid.
Greetings! Just to be clear - this issue tracker is specifically for npm modules related to google cloud (usually the @google-cloud/<service> modules. I don't believe we ship any API which supports talking to the BlobStore API, and we really push our users towards Google Cloud Storage.
Can you share which npm module you're trying to use to upload files to blob store? Is it possible for you to instead use @google-cloud/storage, which allows for much, much, much larger file sizes?
@JustinBeckwith yes perhaps I'm in the wrong place; I'm trying to figure out how to upload large files on Google Cloud with NodeJS/Express; but there's some limitation with Nginx defaults on GCloud or with cloud node but I'm not sure tbh the information online is very vague.
Ah, got it. So this is problematic 😆 App Engine and Cloud Run use memory for their non-persistent storage, so any file you want to process locally would have to get stuffed into memory. You probably don't want that. Can you share more about the video scanning service you're using? If it's possible for that service to read from GCS, you can use things like retention policy to help with the compliance aspects.
How we did this usually on Kubernetes (via DigitalOcean) was that the file gets uploaded to the express instance; the instance then runs FFMPEG compress down to 1080p; then we use the Video Intelligence API to scan for porn/violence; after that, the file is finally validated and can be added to local storage.
This is scalable since we can automatically scale instances depending on CPU usage and traffic via App Engine.
We decided to move to a managed service (since maintaining Kubernetes is a nightmare); thing is we keep hitting silly walls like this.
Thing is; I cannot figure for the life of me how to move a 10GB file from the browser to app engine now.
Hi @ollydixon, this issue tracker is better suited for client library issues, rather than API-related questions and/or bugs. I think you might get some more useful information in this tracker (under appengine): https://cloud.google.com/support/docs/issue-trackers
Closing stale bugs, sorry we didn't get to this sooner!
