uppy
uppy copied to clipboard
transloadit
NextJS example
More of a "tutorial request", but it would be amazing if there was a NextJS example for Uppy.
I'm working through the Uppy docs (which are great) but sometimes a simple code example can make a big difference.
NextJS is relatively popular (44K stars) and it's fundamentally ReactJS, so hopefully this would give a lot of exposure to Uppy for little work
Would you be able/willing to take notes while you figure it out and contribute such an example?
Sure thing @kvz - I'll detail down implementation if I end up using Uppy (I hit a few roadblocks with companion)
Appreciated! <3
Are you hitting bugs or something? Do let us know here on on the community forum :ok_hand:
On 11/27/19 3:13 PM, Copple wrote:
Sure thing @kvz https://github.com/kvz - I'll detail down implementation if I end up using Uppy (I hit a few roadblocks with companion)
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/transloadit/uppy/issues/1958?email_source=notifications&email_token=AAAGRAHLW64IQHCUZPWR7MTQVZ6BJA5CNFSM4JSCUZJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFJUB4Q#issuecomment-559104242, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAGRAGL32AAGBTDHZKZAVTQVZ6BJANCNFSM4JSCUZJQ.
I didn't end up using it @oyeanuj, so no example on my side (i found it too much effort to set up the transloadit server)
You can’t run Transloadit yourself. You can let our company Transloadit host components like Tusd and Companion. That would require no setup, we take care of monitoring, upgrades, scaling, encoding, global distribution so the endpoints are close to your users. Costs $49/mo. Free plans for testing, open source, charity, students, teachers, startups at accelerators that we partner with.
We probably need to do a better job at underscoring that there are hosted versions available in the docs.
Probably you struggled running Companion in production? If you want to run Companion and Tusd yourself, that’s possible too but does require handling said things yourself. Feel free to paste errors and such on our community forum.
As for Next.js, I was interested in learning about the integration but haven’t toyed with it myself. Couldn’t you just use the react integration for this tho? If not I would like to learn why (honest Q, my experience with Next.js is limited to following a few pages of their interactive tutorial :)
Sent from mobile, pardon the brevity.
On 17 Feb 2020, at 01:42, Copple [email protected] wrote:
I didn't end up using it @oyeanuj, so no example on my side (i found it too much effort to set up the transloadit server)
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.
Probably you struggled running Companion in production
Yes you're right! Sorry it was a while back. Yes, most of the implementation could be pulled from the React example, there are just some Next.js specific things that should be added (eg, the env vars in next.config.js). Also Next.js can be SSR with getInitialProps, so there may be some added value there.
Sorry i'm a bit busy to dig up the specific errors that I had with Companion. We may need this in my next project - if so I will try again
That would be great Ethan! 🙌
Sent from mobile, pardon the brevity.
On 30 Apr 2020, at 03:58, Ethan Willis [email protected] wrote:
@kvz I know Next.js pretty well. I could work on this example.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.
@ethanwillis @kiwicopple were either of you able to cook up a nextjs -> transloadit example using uppy/robodog? I'm struggling with getting this to work (mostly with the react Dashboard component and styling) and an example would be very helpful. if not, all good but figured I'd ask!
One more in need of a NextJS example with Companion. I'm trying to setup things but having problems in figuring out what goes where and if it's at all possible.
I'm trying to setup a new API middleware to use Companion but I'm getting errors:
POST http://localhost:3000/api/uppy/url/meta 404 (Not Found)
OPTIONS http://localhost:3000/api/uppy/url/meta 404 (Not Found)
This is my pages\api\uppy\index.js nextjs code:
import session from 'express-session';
import companion from '@uppy/companion';
const options = {
server: {
host: 'localhost:3000',
protocol: 'http',
},
filePath: '/Users/myUser/testFolder/',
secret: 'sdaghdhgsasdajg',
};
const middlewares = [bodyParser.json(), session({ secret: 'somesecretysecret' }), companion.app(options)];
//NEXTJS
// Helper method to wait for a middleware to execute before continuing
// And to throw an error when an error happens in a middleware
function runMiddleware(req, res, fn) {
return new Promise((resolve, reject) => {
fn(req, res, (result) => {
if (result instanceof Error) {
return reject(result);
}
return resolve(result);
});
});
}
async function handler(req, res) {
// Run the middleware
await Promise.all(middlewares.map((m) => runMiddleware(req, res, m)));
res.header('Access-Control-Allow-Methods', 'OPTIONS, GET, POST, PATCH, PUT');
res.json({ message: 'ok' });
}
export default handler;
Any idea on what I'm missing?
Addressing the various example combinations in this branch: https://github.com/ethanwillis/uppy/tree/ethan/nextjs-examples
Examples will be under uppy/examples/next*
@Jmales setting up a Companion middleware in my opinion is a slightly different task than the original ask of this issue. Can you create and link a new issue related to your existing code?
@Jmales I am as well trying to integrate the Companion server into my Next.js project to allow uploading to S3.
Did you solve your problem or found any alternative solution?
@baba43 @zxl777 I've put together an alternate solution which doesn't use Companion and instead allows you to simply forward all the calls from a NextJS API endpoint
https://github.com/daniel-centore/uppy-next-s3-multipart
+1 Has anyone found a good solution to integrate companion directly in the nextJs api? the solution provided by @daniel-centore would have worked, but unfortunately it says that doesn't support uppy v3!
+1 Has anyone found a good solution to integrate companion directly in the nextJs api? the solution provided by @daniel-centore would have worked, but unfortunately it says that doesn't support uppy v3!
Sadly we had to switch everything to a different free front-end library that's more actively maintained and does not require a companion server. Won't link to it directly because I don't want to be disrespectful toward Uppy, but I believe the team should consider putting more emphasis on modern frameworks like Next.js if they are still maintaining this.
Uppy is actively maintained :) There are integrations for React, Vue, Svetle, etc. I am not sure how we can better support Next.js. Companion is meant to be run standalone or as a part of an Express.js server. It would help if someone could clarify what is unclear with Uppy+Next.js integration.
Uppy is actively maintained :) There are integrations for React, Vue, Svetle, etc. I am not sure how we can better support Next.js. Companion is meant to be run standalone or as a part of an Express.js server. It would help if someone could clarify what is unclear with Uppy+Next.js integration.
@arturi NextJS does not use the Express server, it is an independent server implementation. Incorporating the Express server into a NextJS app significantly complicates things and also makes NextJS incompatible with Vercel, which is one of the main advantages of using NextJS.
Last year I built a shim which allows a limited subset of Uppy v2 to work with NextJS ( https://github.com/daniel-centore/uppy-next-s3-multipart ), basically re-implementing the Companion server endpoints in a way which could easily be integrated with NextJS endpoints. This apparently broke with Uppy v3.
The personal project I was using this for is abandoned for the moment and I won't have time to fix this for a while (if ever). Ideally, Uppy would provide either their own NextJS integration, or at least a server-agnostic method of implementing the endpoints so consumers can directly call the backend functions from whatever server endpoints they have.
@daniel-centore thanks for the clarification! Why not run Companion standalone and call its APIs from NextJS, what is the benefit of making Companion part of your NextJS app?
@daniel-centore thanks for the clarification! Why not run Companion standalone and call its APIs from NextJS, what is the benefit of making Companion part of your NextJS app?
Why run an extra server, likely a paid Heroku instance, when it could be all part of one app?
@altechzilla because Companion is a relatively complex standalone proxy server that handles oauth for Google Drive, Instagram, Dropbox, etc + file signing and uploading for XHR, S3 and tus protocols. Easy solution for not running it yourself is to use the hosted one we provide.
Maybe I’m missing something and we could provide a middleware or similar to make working with NextJS easier, but I don’t think we can turn Companion itself into a NextJS app, since it’s just one backend framework for making website and apps, out of many. Would this mean that to make integration with Rails and Django easier, we’d have to rewrite Companion in Ruby and Python?
I don't have experience with NextJs, but I think Next can also do normal fetch/xmlhttprequest calls, see https://nextjs.org/docs/basic-features/data-fetching/client-side - so if companion is running next to nextjs, it should just work. Although I haven't tested it so I don't know if there are some issues with running Uppy from Next. Has anyone tested that?
I think it makes sense to try support running Uppy in Next because Next is such a popular framework, so that it something we should do. If we could make our e2e tests run Uppy with Next instead of Vite, maybe that would be even better.
As for integration Companion into Next, I think that involves:
- rewriting all express/HTTP endpoints to something else. We would have to make an abstraction that can make all the endpoints pluggable into express as well as other http frameworks like next. But Companion is currently tightly coupled with Express and uses a lot of
req.objassignments as well asexpress-sessionetc. - rewriting all our Oauth authentication code to work without express
- we use grant for oauth - is it even possible to use in nextjs?
My gut feel is that rewriting Companion to be integratable into Next, is a full rewrite.
You should be able to run any Node.js server in Next.js. You can for instance run tus-node-server in Next.js, which needs some specific Next.js settings seen here:
https://github.com/tus/tus-node-server/tree/main/packages/server#example-integrate-tus-into-nextjs
But indeed Companion is Express specific, which may make it hard indeed, but may be possible with some research.
You should be able to run any Node.js server in Next.js
@Murderlon Doing this eliminates a lot of the benefits of using NextJS. From https://nextjs.org/docs/advanced-features/custom-server :
Before deciding to use a custom server, please keep in mind that it should only be used when the integrated router of Next.js can't meet your app requirements. A custom server will remove important performance optimizations, like serverless functions and Automatic Static Optimization.
Note: A custom server cannot be deployed on Vercel.
These are blockers for many, if not most projects using NextJS
The idea is to do it inside an API route without a custom server, then you don't loose any benefits. If that's not possible, I wouldn't try to integrate Companion indeed.
in case this is helpful to anyone, this is how I managed to do s3 multipart upload, I know its not perfect and that the code sucks but I just got it into this state and I will make it much better now that I know how to do this:
import { env } from "@/env/server.mjs";
import { prisma } from "@/server/db";
import queue from "@/server/queue";
import { s3 } from "@/utils/s3";
import {
CompleteMultipartUploadCommand,
CreateMultipartUploadCommand,
UploadPartCommand,
} from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import { getAuth } from "@clerk/nextjs/server";
import status from "http-status";
import { NextApiRequest, NextApiResponse } from "next";
import { v4 as uuidv4 } from "uuid";
export default async function handler(
req: NextApiRequest & { session?: { userId?: string } },
res: NextApiResponse
) {
const { userId } = getAuth(req);
if (!userId) {
res.status(status.UNAUTHORIZED).json({ message: "Unauthorized" });
return;
}
if (req.method === "POST") {
const {
filename,
contentType,
operation,
key,
uploadId,
partNumber,
parts,
size,
} = req.body;
if (
(operation === "createMultipartUpload" && (!filename || !contentType)) ||
(operation === "prepareUploadPart" &&
(!key || !uploadId || !partNumber)) ||
(operation === "completeMultipartUpload" && (!key || !uploadId || !parts))
) {
res.status(status.BAD_REQUEST).json({ message: "Missing parameters" });
return;
}
let result;
if (operation === "createMultipartUpload" && filename && contentType) {
const Key = uploadId || uuidv4();
result = await s3.send(
new CreateMultipartUploadCommand({
Bucket: env.AWS_S3_BUCKET,
Key: Key,
ContentType: contentType,
Metadata: {
userId,
file: JSON.stringify({
id: Key,
size: size as number,
metadata: {
name: filename,
size,
filename,
filetype: contentType,
},
}),
},
})
);
res.status(status.OK).json({
uploadId: result.UploadId,
key: Key,
});
} else if (
operation === "prepareUploadPart" &&
key &&
uploadId &&
partNumber
) {
// @ts-ignore
const signedUrl = await getSignedUrl(
s3,
new UploadPartCommand({
Bucket: env.AWS_S3_BUCKET,
Key: key,
UploadId: uploadId,
PartNumber: partNumber,
})
);
res.status(status.OK).json({ url: signedUrl });
} else if (
operation === "completeMultipartUpload" &&
key &&
uploadId &&
parts
) {
result = await s3.send(
new CompleteMultipartUploadCommand({
Bucket: env.AWS_S3_BUCKET,
Key: key,
UploadId: uploadId,
MultipartUpload: {
Parts: parts,
},
})
);
res.status(status.OK).json({
location: result.Location,
});
await prisma?.upload.create({
data: {
size: size as number,
offset: 0,
createdAt: new Date(),
transcodedAt: null,
id: key,
},
});
await prisma.videoMetadata.create({
data: {
name: filename as string,
type: contentType,
fileType: contentType,
fileName: filename,
relativePath: key,
uploadId: key,
},
});
await queue.add("moveUpload", {
uploadId: key,
fileName: filename as string,
});
} else {
res
.status(status.BAD_REQUEST)
.json({ message: "Invalid operation or missing parameters" });
}
} else {
res
.status(status.METHOD_NOT_ALLOWED)
.json({ message: "Method not allowed" });
}
}
the frontend part uses uppy and looks like this, I also did the TUS part but it cant work with vercel because of the payload limit
const uppy = React.useMemo(() => {
const uppyInstance = new Uppy();
if (isResumable) {
uppyInstance.use(Tus, {
id: "uppy-tus",
endpoint: "/api/upload",
retryDelays: [0, 1000, 3000, 5000],
chunkSize: 10_485_760, // 10 MB
onBeforeRequest: async (req, _file) => {
const token = await getToken();
req.setHeader("Authorization", `Bearer ${token}`);
},
});
} else {
uppyInstance.use(AwsS3Multipart, {
id: "uppy-s3-multipart",
companionUrl: "/api",
createMultipartUpload(file) {
return getToken().then((token) =>
fetch("/api/get-signed-url", {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: `Bearer ${token}`,
},
credentials: "include",
body: JSON.stringify({
filename: file.name,
filetype: file.type,
type: file.type,
contentType: file.type,
size: file.size,
operation: "createMultipartUpload",
}),
})
.then((res) => res.json())
.then((data) => {
return { uploadId: data.uploadId, key: data.key };
})
);
},
signPart(file, partData) {
return getToken().then((token) =>
fetch("/api/get-signed-url", {
method: "POST",
credentials: "include",
headers: {
"Content-Type": "application/json",
Authorization: `Bearer ${token}`,
},
body: JSON.stringify({
...partData,
operation: "prepareUploadPart",
}),
})
.then((res) => res.json())
.then((data) => {
return { url: data.url };
})
);
},
completeMultipartUpload(file, data) {
return getToken().then((token) =>
fetch("/api/get-signed-url", {
method: "POST",
credentials: "include",
headers: {
"Content-Type": "application/json",
Authorization: `Bearer ${token}`,
},
body: JSON.stringify({
key: data.key,
uploadId: data.uploadId,
parts: data.parts,
size: file.size,
filename: file.name,
contentType: file.type,
operation: "completeMultipartUpload",
}),
})
.then((res) => res.json())
.then((data) => {
return { location: data.location };
})
);
},
});
}
return uppyInstance;
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [isResumable]);
if you wanna see the entire thing I'm about to move 100% to transloadit as I consider it an excellent service and aws s3 multipart is the way to go my project is https://github.com/sicksid/pugtube so hopefully it helps anyone as its def possible
