[FEATURE REQUEST] Enable users to delete the buildpack cache per app

[svkrieger]

Issue

Once every while we have users whose buildpack cache gets so big that they can't push their app anymore, because it exceeds the 4GB limit of the staging containers (together with the package and droplet).

Context

Currently there are two possibilities to fix this issue:

1. Delete the app (which is often not an option for productively running apps)
2. Delete the global buildpack cache of the whole cf installation with the /v3/admin/actions/clear_buildpack_cache endpoint. This is also not optimal, because the user first has to contact an operator to delete the buildpack cache globally. Also the next push of all apps will be slower, because dependencies and so on need to be downloaded again.

It would be favourable to enable users to delete the buildpack cache for their app. This could be e.g. done by implementing an endpoint like /v3/apps/:guid/actions/clear_buildpack_cache.

If there is a need, in the future it could be also added to the staging process itself e.g. by adding query parameters to the /v3/builds endpoint like no-cache or clear-cache or having an app feature which persists such a choice.

Expected result

The user can delete the buildpack cache of their app by themselves.

Current result

An operator has to delete the global buildpack cache, which removes the cache of all apps.

Possible Fix

Implement an /v3/apps/:guid/actions/clear_buildpack_cache endpoint, which deletes the buildpack cache for a single app.

[jpalermo]

We thought of an alternative idea that we might consider.

Add a global configuration option in Cloud Controller for max_buildpack_cache_size. When diego is done staging, it uploads the new cache to Cloud Controller, and if the cache is larger than the max size, Cloud Controller will automatically clear the cache for that app.

This would prevent app developers from having to worry about the buildpack cache for their app and let the platform manage it for them.

Thoughts?

[svkrieger]

I like the idea of having the platform deal with it instead of the users. This idea is worth to be analysed further.

Just one thing which might cause problems is that buildpacks can be of different size. While I think that the smallest buildpacks like binary and static don't have any cache, the largest difference between buildpacks with a cache I could see, was between the r-buildpack (763MB) and the ruby-buildpack (440MB). So when having a global max_buildpack_cache_size, applications with a smaller buildpack would be disadvantaged, because theoretically they could have a bigger cache size, without exceeding the 4GB limit. I'm not 100% sure though, whether the buildpack even counts into the 4GB limit, as the documentation states that "app files, compiled droplet, and buildpack cache cannot total more than 4 GB of space during staging" see here.

[Gerg]

I'm currently facing a similar issue with a Node app:

              Uploading droplet, build artifacts cache...  
              Uploading build artifacts cache...  
              Uploading droplet...  
              Uploaded droplet (205.7M)  
              Failed to upload payload for build artifacts cache  
              Uploading failed  
           FAILED  

Being a modern Node app, it has ~500MB of npm packages, which the Node buildpack puts in the buildpack cache. After a few rounds of updating npm package versions, it's easy for the buildpack cache to have > 1GB of cached packages, which exceeds the 1GB cc.app_bits_max_body_size configuration for my environment.

I worked around this by bumping the environment's max body size to 2GB, but this was clearly not a long-term solution, since the cache size will probably eventually reach that magnitude. Deleting the app and pushing again (maybe as part of a blue/green deploy?) is another blunt tool that can help.

---

Ideally, app developers shouldn't need to know or care about the buildpack cache, since it is just an optimization. Even them having to manually clear it feels like a bad experience (but better than the current situation).

Also, I don't think this error should be able to fail the build. The app/droplet are totally fine and should work. It's a shame that it's being blocked by the cache. IMO, failing to upload the cache should be a warning at most.

[kathap]

Possible fix was implemented, new endpoint /v3/apps/:guid/actions/clear_buildpack_cache is available with https://github.com/cloudfoundry/cloud_controller_ng/pull/3848