Formidable monorepo and v4 perp

[tunnckoCore]

Moving to a monorepo and TypeScript.

There will be couple of benefits with that like we can have v1, v2, v3 and others as separate directories and not separate branches. We can make CI builds based on that too.

I played with a lot of variants, so there will be:

• package for each major, like packages/v1 (or similar), v2, v3,
• a v3-trimmed variant,
• benchmarks package,
• v4 built on the old/current parser (the v1/v2/v3 parser is byte by byte streaming which is slow for large files)
• v4 with fresh parser,
• v4 built on @mjackson/multipart-parser fork
• helpers and s3 adapter
• frontend utils

What i just started realizing is that the HTTP Multipart is pretty bad, and there might be an alternative way to handle all that a lot better, like..

What if on the backend user defines formidable(req, options) but on his frontend he also get the formidableClient(e.target, options), the client side part will basically get the FileList and send in parallel each File to a server endpoint that is handled by the formidable(req) and just does the validation and checking on the server.

That way we can accept multiple requests at the same time and validate and stream to a third-party like S3 or the disk. Usually what happens is, no matter how many files and fields there are, on the server end we (any body parser) get just one stream that we gotta process "synchronously", there's no other way - but that's the nature of streams.

eg. in frontend

function handler() {
  const files = event.target.files as FileList;
  const results = await Promise.all(files.map((file: File) => {
    return fetch('/api/upload', {
      method: 'POST',
      body: file
    })
  }))
}

and then, on the endpoint backend, you just get the requests, validate it and send it wherever you want (disk or S3).

[tunnckoCore]

Published v4 RC on formidable@next using a fork of the @mjackson/multipart-parser with several limit options and other tweaks i don't like there. We can't just build directly on top of it for few reasons.

What's cool is the remix-the-web monorepo is thoroughly tested, including the multipart-parser and the headers package that is used under the hood to parse the part's headers and expose it through the part.headers getter.

[tunnckoCore]

The tests in test-node/standalone are not actually working. They all fail at parsing level, erroring with

FormidableError: MultipartParser.end(): stream ended unexpectedly: state = PART_DATA

All tests in general are broken beyond belief. And all tests are basically testing weird stuff with emitters and streams. It's absolute nightmare.

Rather prefer to switch the whole thing under the hood with v4 + compat for limit options, and to actually test that.