[misc]: Processing files in middleware

[didinewlander]

I understand that this issue may be closed if it should be filed in another category

• [X] I understand

My issue

I have this documentUploader route, that gets word,pdf and image files:

documentUploader: f({
    pdf: { maxFileSize: "8MB", maxFileCount: 4 },
    image: { maxFileSize: "8MB", maxFileCount: 4 },
    "application/msword": { maxFileSize: "8MB", maxFileCount: 4 },
  })
    .middleware(async ({ files }) => {
      await auth();

      const processedFiles = [...files];

      const images = files.filter((file) =>
        file.name.match(/\.(png|jpg|jpeg)$/i)
      );
      for (const file of images) {
        const resizedBuffer = await sharp(file) // << Problematic Line
          .resize(100, 100)
          .toBuffer();

        const resizedFile = {
          ...file,
          buffer: resizedBuffer,
          name: `small-${file.name}`,
        };
        processedFiles.push(resizedFile);
      }
      return { files: processedFiles };
    })
    .onUploadComplete(() => {}),
} satisfies FileRouter;

Disclaimer: I got to this version using ChatGPT, but the overall idea is from 'Josh Tried Coding' channel https://www.youtube.com/watch?v=7hS9b6n7HrM for image compression optimized for nextjs applications.

The Issue

The problem is that in the middleware I haven't figured out how to work with the buffer itself, only with the FileUploadData class, which provides only the name, size, and type of the file, meaning I can't actually make any modifications or create more files in the middleware.

The Solution

Enable in the core.ts file, where all the routes are handled, an option to work with the files themselves BY REQUEST and not by default, so only if there is a BE file optimizer or handler the developer could process the files by selection.

Use cases

• Image compression
• File parsers and validators
• Automation processes and file forwarding

[markflorkowski]

The file buffers aren't available in the middleware to avoid server ingress/egress costs. To achieve what you need, you'd either need to process files client-side or have your own server endpoint ingest, modify, and then upload them using UTApi functions.

We're considering an alternative that would allow optional server-side file handling, despite the added costs, but it's not yet in development. Your feedback is quite helpful here, as it highlights demand for this feature.

cc @t3dotgg

[romoguill]

@markflorkowski I had a similar need but I it could also be done in the onUploadComplete callback. For instance geting an image width and height. The file prop doesn't have the buffer from the file but just some metadata. The only solution I found is to make a request to the ufsUrl and then work with the proper file. Would this be the correct approach? It's an extra request which could be prevented and I don't know if what you said about ingress/egress costs apply to the onUploadComplete callback too.

[markflorkowski]

@romoguill Alternatively, you can pass that information in the input from the client, however it is possible a malicious user could mess with that data. docs

For a guarantee of the integrity, what you described of fetching the file in onUploadComplete would be a valid method, though you would be paying that ingress fee on your server as mentioned.