File uploads

[jimfranke]

Hi; I like the approach you're taking with Wildcard API. What's you're suggested way of handling file uploads, any idea's on how to do this through RPC?

[brillout]

Hi @jimfranke !

EDIT: This answer is outdated; see the new file upload propsal below.

I'd say go for what you are most comfortable with.

I'm assuming that you are using a modern view library such as React, Vue, or Angular.

You then have two options:

• FormData
• FileReader

If you use FormData then you use Express/Koa/Hapi directly and you bypass Wildcard.

If you use FileReader then you send the file's binary data with Wildcard:

// Node.js

const {endpoints} = require('@wildcard-api/server');

endpoints.uploadImage = async function({imageData}){
   // `imageData` is the binary data of the image
};

// Browser

import React from 'react';
import {endpoints} from '@wildcard-api/client';

function imageUploaderComponent() {
  return (
    <form>
      <input type="file" onChange={e => this.onChange(e)} />
    </form>
  );

  function onChange(ev) {
    const reader = new FileReader();
    reader.onload = async () => {
      const imageData = reader.result;
      await endpoints.uploadImage({imageData});
      console.log('Image upload done!');
    };
    reader.readAsBinaryString(ev.target.files[0]);
  }
}

Personally, I use FileReader. But the FormData way seems to be fine as well.

I like the approach you're taking with Wildcard API

Thanks :-)

Let me know if you have questions!

[brillout]

I'm closing this but let me know if there anything that is not clear :-)

Rom

[DominikSerafin]

If you use FormData then you use Express/Koa/Hapi directly and you bypass Wildcard.

Does this mean that FormData is not supported at all by wildcard? Do you plan to support it in the future?

[brillout]

Hi @DominikSerafin,

Not yet, but I'm open to support it.

What is your use case and tech stack?

[DominikSerafin]

@brillout usually my tech stack consists of Express with React.

As for use case, I was actually thinking about completely replacing Express with Wildcard in production for one of my upcoming projects that will feature various file uploads ranging from tiny to huge.

Unfortunately, lack of support for FormData blocks me in doing that, as using two frameworks at the same time adds unnecessary complexity, so for me it's not a great solution.

I don't want to use FileReader as well because it is heavier on performance (and every bit of it matters for this project) and in general is more hacky (so less overall support) than just using FormData via multipart/form-data.

I'll probably stick to only Express for now, but I think this project is really cool, so I'll revisit it for sure once/if FormData is implemented. :)

[brillout]

The plan regarding file uploads is:

// Browser

await server.submitForm(formData, {someNonFormData: 42});

// Node.js

server.submitForm = function(formData, {someNonFormData}) {
  // By default, files are awaited for and loaded in memory before Wildcard invokes the `submitForm` function
  formData.profilePicture.data;
  formData.memeImage.data;

  console.log(someNonFormData); // prints 42
};

With stream as well as disk mode.

Behind the curtain, Wildcard will use https://github.com/mscdex/busboy which AFAIK is the fastest multipart parser out there and is the one used by Fastify. Also, Wildcard's archticture is designed so that HTTP related libraries are easy to switch out, so that whatever faster multipart comes next, it is going to be the one Wildcard uses.

I was waiting to implemented this until someone needed it, I'm glad someone is knocking at the door :).

completely replacing Express with Wildcard

That's actually the vision here as I believe remote functions can replace all use cases.

I've recently implemented auth sessions. It's not released yet but it's currently running in production and works quite well. (It's a pleasure to be able to simply change the Context upon a endpoints.login() while Wildcard automatically handles the auth cookie for me. I'm looking forward to never have to deal with auth headers anymore while writing apps :).)

I plan to release the auth session thing this week / beginning next week.

You should then be able to completely replace Express.

Edit: removed await; files should be automatically loaded to memory.

[DominikSerafin]

@brillout great to hear :)!

As for file uploads - I think your solution looks great when it comes to reading chunks of files.

However, I wonder if there could be also an additional (optional?) solution where Wildcard could just automatically detect any passed FileList or specific File objects and then just transport them behind the scenes via multipart/form-data and then invoke the server method only when the files are fully streamed in (after end/finish event)?

It would make things super easy & straightforward for some use cases that don't require reading partial chunks of files.

Like so:

// client

const someString = 'Hello, World!';
const someBoolean = true;
let someFile;
let someAnotherFile;

input.addEventListener('change', function(){
  someFile = input.files[0];
  someAnotherFile = input.files[1];
});

// ...

await server.submitForm(someString, someFile, someBoolean, someAnotherFile);


// server

server.submitForm = function(someString, someFile, someBoolean, someAnotherFile) {
  // someFile and someAnotherFile are busboy objects with complete files
}

What do you think?

[brillout]

I agree:

• By default, files should be awaited for and loaded to memory before Wildcard invokes the endpoint function. No need to await, so basically what you proposed.
• Default file upload limit of 20 MB; new error flag fileUploadSizeLimitExceeded to the Wildcard client error object.
• Global option:

  // Change default limit to 50MB
  config.fileUploadSizeLimit = '50MB';
  

• Endpoint option:

  // Second argument has a limit of 100 MB, and third argument's property someAnotherFile a limit of 5MB.
  server.submitForm.fileUploadSizeLimit = [undefined, '100MB', {someAnotherFile: '5MB'}];
  server.submitForm = function(someString, someFile, {someAnotherFile}) {
    // someFile.data
    // someAnotherFile.data
  };
  

• Disk mode:

  // Second argument is saved in a /tmp/* file with max size of 500MB,
  // and third argument's property someAnotherFile is as well saved to disk with a limit of 10GB
  server.submitForm.uploadFileToDisk = [undefined, '500MB', {someAnotherFile: '10GB'}];
  server.submitForm = function(someString, someFile, {someAnotherFile}) {
    // someFile.filePath
    // someAnotherFile.filePath
  };
  

[brillout]

@DominikSerafin is there anything else you'd want?

[brillout]

@DominikSerafin I guess you're not interested anymore (that's fine). I'm deprioritizing this.

If someone is reading and interested in this, let me know and I'll implement it.

[DominikSerafin]

@brillout I had to focus on something else, and I couldn't give a fast reply here, but I'm still interested in wildcard supporting file uploads 💯.

The only remaining feedback I've got is that it's pretty great that my proposal would be a default way how wildcard handles files, but it's also important for me that there is a way to read chunks when you need that. Other than that I think your plan looks great, and I don't have anything else to add at this point from my side.

[brillout]

@DominikSerafin 👍 I'll start implementing file uploads in about a week.

Stream mode:

server.submitForm.uploadToStream = [undefined, true, {someAnotherFile: true}];
server.submitForm = async function(someString, someFile, {someAnotherFile}) {
  // `someFile.stream` is a `ReadableStream`
  for await (const chunk of someFile.stream) {
     // Do something with chunk
  }
  // After the `for await` loop `someFile` is done and fully uploaded
};

[brillout]

Started working on this. The interface will be slightly different. (With several advantages, mostly around TypeScript integration, but also extra flexibility of being able to decide file upload strategy at request-time.)

const { server, context, loadFile } = require('telefunc/server'); // (I'm renaming Wildcard API to Telefunc.)

server.submitForm = async function (someFile) {
  const data = await loadFile(someFile);
}

server.submitAnotherForm = async function (someString, someFile, { someAnotherFile }, thirdFile) {
  const [{data}, {diskPath}, {stream}] = await loadFile([
    {
      file: someFile,
      loadToMemory: '10MB',
    },
    {
      file: someAnotherFile,
      loadToDisk: '100MB',
    },
    {
      file: thirdFile,
      loadToStream: context.user.isAdmin ? Infinity : '1MB',
    }
  ]);
};

[brillout]

Pardon me the delay; I've been working on https://github.com/brillout/vite-plugin-ssr which is fairly high prio since I'm aimaing at shipping the first SSR tool built on top of Vite.

(In case you're curious to see the new Telefunc logo: https://github.com/brillout/wildcard-api/tree/renaming#readme.)

[xgenvn]

IMHO, we should detach from the main API for file uploads using the signed URL uploading approach. In previous projects, integrate file uploading into Express seems not good in terms of scale and performant, and 90% of file uploading requirements are to integrate with the 3rd blob storage services. So I think it's the form-data support instead of the explicit file uploading feature.

[brillout]

integrate file uploading into Express seems not good in terms of scale and performant

Can you elaborate?

90% of file uploading requirements are to integrate with the 3rd blob storage services

I'm thinking of plugins for things like GraphQL but also for third-party file uploads.

The plugins are built directly on top of HTML and expose a user interface in an Wildcard idiosyncratic way.

server.submitForm = async function (someFile) {
  // `loadToS3` is defined by a plugin and returns the URL of the file.
  const url = await loadToS3(someFile);
}

(@DominikSerafin Expect further delayed but vite-plugin-ssr is becoming more and more stable and requires less and less work; I'll then work on Wildcard/Telefunc again.)

[xgenvn]

integrate file uploading into Express seems not good in terms of scale and performant

We did use busboy to handle uploads but the requests body was an issue while running on some gateways/serverless service. For scale, my previous project also needs to scale only the upload API, not the whole API, so it's also a common issue if someone is going to do a microservice.

[xgenvn]

@brillout for the plugins, I think we don't need to bind too closely with the frontend, the interface to config the signed URL is good though, but that depends on requirements and easily done with the current version of wildcard-api.

No strong opinion here, it's my thought that we don't want something sophisticated but not practical (I also understand that people want to serve uploads API through express).

[brillout]

@xgenvn 👍 I will take this in consideration. Also, note that Wildcard is only a middleware: you don't have to use Wildcard for everything, and you can use your custom HTTP handling instead of using Wildcard when you need something specific that Wildcard doesn't do.