Cache management facilities

[cgils]

Please consider adding some way of setting a maximum cache size. At the moment our project's build caches grow unbounded leading to us discarding them periodically.

A facility with which to display cache information would be nice as well. Something like the output of ccache -s, for example:

  cache directory                     /home/user/.ccache
  cache hit (direct)                 34993
  cache hit (preprocessed)             153
  cache miss                         23479
  compile failed                         3
  ccache internal error                  1
  preprocessor error                     1
  cache file missing                     1
  unsupported source language          382
  no input file                       7322
  files in cache                     70445
  cache size                          16.4 Gbytes
  max cache size                      90.0 Gbytes

[dslomov]

@cgils, are you talking about project "cache" (it is not really a cache, it is just your output directory) or stuff that gets into ~/.cache/bazel?

[cgils]

Stuff that gets put into ~/.cache/bazel.

[glinscott]

I vote for this to be higher priority :). Bazel is commonly filling up hard drives on our project from the ~/.cache/bazel directory growing unbounded.

[kavehv]

+1

[jianga]

+1

[jiayiliu]

+1, esp. if the home dir is on NFS.

[lefromage]

+1

[burnzzz]

+1

[softprops]

+1

[llhe]

+1

[dahlstrom-g]

+1

[hlopko]

@buchgr

[buchgr]

@jin do you have an idea to which component to assign this issue?

[jin]

@buchgr team-Local-Exec sounds like a good fit.

[jmmv]

There are several kinds of data to be managed here:

• Bazel "installation files". This is https://github.com/bazelbuild/bazel/issues/2109 and they should be auto-purged. No question.

• Output trees. These are just the build artifacts of a given build. They are not a cache. Bazel discarding them automatically is a bad idea as we'd be throwing away people's data. bazel clean removes these on a workspace basis. (I think these files living under ~/.cache/ is also a mistake.)

• Fetch/remote cache (whatever it's called). bazel fetch can download a lot of stuff and this stuff is shared across workspaces. This is a cache and could be automatically cleaned. However, pruning stuff from here affects the ability to work offline, so I'd be wary of doing that too. It'd be pretty annoying to fetch everything before hopping into a plane, only to discover later that some random stuff was pruned and you cannot do anything any longer.

Separately, having a command to dump all files Bazel knows about, grouped by category/project and with a summary of their size, would be awesome. Tag each entry with an ID so you can tell bazel clean some-id and it'd be even better.

[edrumwri]

+1

[jmmv]

Per #2765, one more thing to consider if implementing a smarter clean command is bazel clean //target. Noting here because this would possibly conflict with the bazel clean some-id proposal above.

[ozio85]

+1

[jonatanj]

+1

[dchichkov]

It had been some time since 2016, it'd be great for this to get higher priority.

To give some context, I'm on 2TB drive and have to keep fighting with Bazel for living space. My ~/.cache/bazel is consistently eating up to 0.7TB of free space (in 2-3 weeks). I have to purge it then. And that results in multi-GB downloads (CUDA, etc).

It'd be great to add a limit and an LRU-type of behavior to the cache. Alternative - remove "{Fast, Correct} - Choose two" advertisement from the web page. Eating all the space on the disk is not correct. Making a user to purge cache completely is not fast.

[MilesCranmer]

This issue should really be P1, not P2. It is extremely bad that on my system there have been nearly a million files generated in the ~/.cache that aren't automatically cleared.

On many personal computers, people might just not realize this is happening, and have their entire filesystem slowed down due to the additional effort needed by their system to index everything.

Then, on some shared computers where there are hard file limit constraints, bazel is effectively unusable because it generates so many files. (This is the only reason I noticed this in the first place – otherwise I would have just had a slower filesystem without knowing why)

[Ryang20718]

+1

[gosha-frost]

+1

[StilesCrisis]

+1

[tjgq]

We acknowledge the problem discussed here; however, for the sake of clarity, I'm going to close this issue in favor of the following narrower-scoped ones:

https://github.com/bazelbuild/bazel/issues/2109 https://github.com/bazelbuild/bazel/issues/5139 https://github.com/bazelbuild/bazel/issues/22515 https://github.com/bazelbuild/bazel/issues/22516