turborepo-remote-cache icon indicating copy to clipboard operation
turborepo-remote-cache copied to clipboard

Published 20 hours ago • verified publisher icon ducktors

Large build output cannot be uploaded

Open asquithea opened this issue 2 years ago • 3 comments

I found that artifacts from the end of my build pipeline (just before deploy) are not cached.
Bit of a show-stopper as it causes a full redeploy!

It looks like Fastify has a default body size limit of 1 MiB, and so I'm speculating this is the reason.

Could we have a configuration option to bump this considerably higher?

A backport to the last stable version (so 1.1.2 I guess) would be really helpful, BTW.

asquithea avatar Jun 27 '22 06:06 asquithea

This is strange. We already have the limit increased from default here: https://github.com/fox1t/turborepo-remote-cache/blob/main/src/plugins/remote-cache/index.ts#L18

fox1t avatar Jun 28 '22 10:06 fox1t

Hmm. I'm building individually packaged lambda functions using esbuild, so the packages are varying between about 1 MB and 5 MB. But there are quite a lot of them in one service... the total size of the service build folder containing the ZIP files is 228 MB! This probably explains it if they're all being uploaded together as a single artifact: no reason to think your limit is ineffective.

100 MB does seem like a reasonable limit if the data is being buffered in memory. Do you think it would be possible to stream the request to storage? Along the lines of https://github.com/fastify/fastify/issues/534 ? I appreciate this is probably harder to implement, though it might yield a performance benefit.

asquithea avatar Jun 28 '22 21:06 asquithea

I think we can support arbitrary significant artifacts using a strategy similar to one I used here: https://github.com/fox1t/fastify-multer (multer is only for form-data/multipart).

The idea is to have a content parser that flags the incoming request as "interesting" and then handles the upload in a preHanlder function, piping the incoming stream directly to a writable stream (regardless of the local file or remote bucket).

Would you like to open a PR to support it?

fox1t avatar Jun 29 '22 13:06 fox1t

This might also be related to https://github.com/vercel/turbo/issues/2096 and https://github.com/vercel/turbo/issues/2280

tm1000 avatar Oct 27 '22 17:10 tm1000

After further testing and creating a PR in turborepo (https://github.com/vercel/turbo/pull/2428)

I can confirm that I was able to upload an artifact of 92mb to the ducktors/turborepo-remote-cache server implementation. It took about 1min

tm1000 avatar Oct 27 '22 20:10 tm1000

I have an issue where my artifacts are larger than the 104mb file size limit.

Would it be possible to increase the default, or add an environment variable to set this limit for the server?

simonjpartridge avatar Oct 31 '22 11:10 simonjpartridge

I'd like the same as @simonjpartridge. An environment variable would be great

tm1000 avatar Oct 31 '22 23:10 tm1000