Submit Code *[as Fork]* generates Payload Too Large

[Martii]

Since we are Content-Type: application/x-www-form-urlencoded encoding a scripts source this generates a larger transfer content size.

For example a script of actual size 782,701 Bytes can get expanded to 1,087,143 Bytes which renders an express page (e.g. not ours) of:

Payload Too Large 

1. We're not handling this failure gracefully... not sure that we can with body-parser :\
2. Unable to fork scripts that are too large. (this is the main bug)
3. Unable to change (too large of) scripts through the site editor... although upload a file works because it isn't urlencoded. (work-around)

Possibilities:

• There is a verify property in body-parser that could be used in our code but we would need to up the limit significantly and check/process in the verify routine if the buffer to UTF-8 string was too big in correlation to the maximum_upload_script_size.
• ... (something else?)

[Martii]

Additional issue found with server accepting ANSI files ... we still do accept them but we should be serving UTF-8 now in the stream with #794 .

[sizzlemctwizzle]

I'll take a closer look later today and see if I can find a solution.

[sizzlemctwizzle]

Okay, so I tried building my own multipart/form-data request with the source of the script pulled out of ACE and sending it to the server via AJAX, but I had no luck getting formidable to parse the request body correctly. Anyway, I stashed my failed changes in a branch^{with this one commit}.

Sorry Marti, that was the best idea (essentially handling a script source submission like a script upload) I could come up with today. Maybe I'll come up with something else later, or maybe you can find a way to make my idea work...I have very little experience with raw request data.

[Martii]

Thanks... I'll have a look at it in a bit. :)

I was thinking about trying formidable too but wasn't quite ready for that just yet... still ironing out our Unicode issues (which is loosely related to this with urlencoding e.g. percent encoding) from #200 ,a tiny step at a time, because it's still present in a small use case manor.

[sizzlemctwizzle]

We could always use socket.io to stream the script source to the server. But that'd be overkill.

[Martii]

Plus we would be excluding any browser (versions included) that doesn't have socket support... not sure we are at that point yet either. Interesting idea though.

[sizzlemctwizzle]

Just for future reference, I never tried capturing the POST I sent using some browser tool. Perhaps that could be enlightening. I'm sure there's one that can parse multipart/form-data and can tell you when your request is property formatted. On Nov 2, 2015 8:34 PM, "Marti Martz" [email protected] wrote:

Plus we would be excluding any browser (versions included) that doesn't have socket support... not sure we are at that point yet either. Interesting idea though.

— Reply to this email directly or view it on GitHub https://github.com/OpenUserJs/OpenUserJS.org/issues/793#issuecomment-153220499 .

[Martii]

Btw assuming it's a site fork, which this issue is, one should be able to fork a script on OUJS using just the metadata blocks to get the required fork status then file uploading (for those that don't have a GH account and repo) the complete actual code. Been a while since I've been in this section of code and there are other upcoming fires to put out.

---

I'm also contemplating upping the size limit but #1548 applies since we don't have WiredTiger for MongoDB yet with a volume for that. This means OUJS needs more donations to keep running as it at least doubles the monthly cost. Not a good idea to do this during the pandemic though without adequate donations.