upload-artifact Archive uploads instead of uploading individual files

I have a 900MB node_modules directory with 86k files. Using actions/upload-artifact as-is takes about an hour and a half -- the logs appear to indicate that files are uploaded individually.

If I tar/gzip the directory first, the tar takes about 30 seconds, reducing it to 175MB, then the upload takes about 25 seconds.

Example of how actions/cache does it: https://github.com/actions/cache/blob/master/src/save.ts

Nov 21 '19 16:11 jry-anchor

~You should not use actions/upload-artifact for this, instead you should use actions/cache - see https://github.com/actions/cache~

Edit Dec 2020: I was wrong. actions/cache can cache the process of running npm install, but doesn't make sense to share the node_modules directory between steps.

Nov 25 '19 12:11 g105b

@g105b in that case the Passing data between jobs in a workflow doc needs an update, since that page doesn't mention limitations or actions/cache at all. Also Comparing artifacts and dependency caching. How many files and/or which size of directory warrants using caching instead of archiving?

Also calling it a cache for some "artifacts" but not others is a bit of a misnomer, since I need the exact contents of the directory in all the jobs in the workflow.

Nov 25 '19 14:11 jry-anchor

I understand and share your frustration. I hope this is made a bit simpler soon.

Nov 25 '19 14:11 g105b

Hi, I'm trying to have a build process spit out a fully working artifact that includes everything it needs to run (node_modules, dist, etc). But I'm facing the same problem: uploading 16k files takes longer than actually building the whole application.

If I understand this correct, actions/cache would only help if you need to share the node_modules between workflow runs?

Is there any workaround to force actions/upload-artifact to somehow tar the files instead of uploading them all individually? I also tried using actions/create-zip-file, which speeds everything up, but when downloading the artifact there now is a zip inside a zip, and I feel like that's unnecessary. I'm guessing #69 would fix this problem, but it doesn't seem like that will be a thing soon

Jul 17 '20 10:07 Clemens-E

You can get a massive boost by taring the folder before uploading it (Zip also works, but then you're wasting time on compression). The resulting file being zipped again is unfortunate, but the speedup is worth it. Edit: see also #39