actions-setup icon indicating copy to clipboard operation
actions-setup copied to clipboard

Published 20 hours ago • verified publisher icon earthly

Investigate caching buildkit cache volume

Open ingwarsw opened this issue 3 years ago • 7 comments

After @kevinschoonover added caching in #2 maybe it would be possible to cache in GHA whole buildkit cache volume.

That should give huge speed boost cause it would work more/less as perfect case of advanced docker caching in earthly. And speed of GH cache should be really good.

ingwarsw avatar May 31 '22 18:05 ingwarsw

This would be nice to have. Long-term, I'd like to move over to one of Earthly's paid CI features (Satellites or Earthly CI), but I can't do that until I've convinced my team how great Earthly is :)

shepherdjerred avatar Mar 02 '23 01:03 shepherdjerred

I've made a little proof-of-concept for this using actions/cache here: https://github.com/theomessin/example/blob/master/.github/workflows/push.yaml

I think it works great!

image

theomessin avatar Mar 17 '23 02:03 theomessin

That looks great!!

Any idea how this interacts with GitHub's 10GB per-repo cache limit? If the volume is >10GB, will the entire cache be evicted, or just individual layers/files

shepherdjerred avatar Mar 17 '23 05:03 shepherdjerred

@shepherdjerred I'm pretty sure the entire cache would be evicted. However, I think that's okay, it's just a cache after all. I don't mind clearing the cache and re-running all steps every once in a while. Now if your earthly-cache is more than 10GB after running earthly once, I can't help you 😆

theomessin avatar Mar 17 '23 12:03 theomessin

this looks cool!

I wonder if /var/lib/docker/volumes/earthly-cache could be backed without the use of a docker run? I've never looked into if it's possible.

alexcb avatar Mar 17 '23 16:03 alexcb

Now if your earthly-cache is more than 10GB after running earthly once, I can't help you 😆

The cache size is 10GB per repo, not workflow :/

Unfortunately, I think this would lead to a lot of evictions making the cache not very effective. The evictions might be tolerable if it's per layer, but if the entire volume is evicted then I think it'll just be wasted bandwidth, at least in my case.

Great work though! I'm sure this would be useful for many others.

shepherdjerred avatar Mar 17 '23 18:03 shepherdjerred

Unfortunately, I think this would lead to a lot of evictions making the cache not very effective. The evictions might be tolerable if it's per layer, but if the entire volume is evicted then I think it'll just be wasted bandwidth, at least in my case.

I've no idea how the earthly-cache is structured, but nothing's stopping us from saving multiple separate caches. I'll try to find some time to test this 😃

theomessin avatar Mar 17 '23 19:03 theomessin