Default compiler_check setting does not seem optimal for Github Actions

[cmelchior]

This is the default config for ccache:

Run ccache --show-config
(default) base_dir = 
(/home/runner/.ccache/ccache.conf) cache_dir = /home/runner/work/realm-kotlin/realm-kotlin/.ccache
(default) cache_dir_levels = 2
(default) compiler = 
(default) compiler_check = mtime
(/home/runner/.ccache/ccache.conf) compression = true
(default) compression_level = 6
(default) cpp_extension = 
(default) debug = false
(default) depend_mode = false
(default) direct_mode = true
(default) disable = false
(default) extra_files_to_hash = 
(default) hard_link = false
(default) hash_dir = true
(default) ignore_headers_in_manifest = 
(default) keep_comments_cpp = false
(default) limit_multiple = 0.8
(default) log_file = 
(default) max_files = 0
(/home/runner/.ccache/ccache.conf) max_size = 500.0M
(default) path = 
(default) pch_external_checksum = false
(default) prefix_command = 
(default) prefix_command_cpp = 
(default) read_only = false
(default) read_only_direct = false
(default) recache = false
(default) run_second_cpp = true
(default) sloppiness = 
(default) stats = true
(default) temporary_dir = 
(default) umask = 

But using compiler_check = mtime is suboptimal for CI as the timestamp changes on every checkout. Shouldn't the default setting be compiler_check = content?

[jonashaag]

If you’re using Git then I think you are getting “constant” mtimes because Git keeps track of them.

[cmelchior]

We are using Git and actions/checkout@v3, but I was seeing a cache hit rate of 0%, it went up to 100% (as expected) after adding this change.

[jonashaag]

Interesting. Would you mind creating a minimal reproducing example that we could use as a test for that fix?

[cmelchior]

My experiments are Open Source. The branch is here: https://github.com/realm/realm-kotlin/pull/881/files#diff-15c806aa509538190832852f439e9921a23bec2da81f95ed0e4bf13c14e5b160R52. You should be able to clone the repo and just run it, but the full build is currently ~35 minutes.

If that isn't working for you I can try to cut it down to something more manageable?

[jonashaag]

That would be great! Would be perfect if you could boil it down to using a single file in this repo (or if that doesn’t work, a demo repo)

[hendrikmuhs]

According to the docs, this setting influences how the cache matches the compiler binary, but not the source file. So it is about the mtime of the compiler, which should only change if you change the base image for your workflow.

[hendrikmuhs]

@cmelchior

I think this is a misunderstanding of ccache. The setting controls how the compiler binary is matched. I had a quick look at your PR, you seem to use the ootb ubuntu compilers. Using content means that ccache creates a checksum of the binary on every invocation, that slows down caching.

I don't know the reason why your didn't see cache hits, the reason must be something else. It would be surprising, I haven't received similar feedback. Note that caches of PR branches are isolated from e.g. the main branch, therefore I am not sure how good caching works in a PR branch, e.g. they might get evicted faster than for main.

If you want to follow up on this, I need something smaller and isolated, e.g. a you could create a small project.

[cmelchior]

Thank you for the clarification. I'll see if I can reproduce it on a much simpler project. For what it is worth, I got the hint from the React Native docs here: https://reactnative.dev/docs/build-speed#using-this-approach-on-a-ci

[jonashaag]

Seems like a mistake on that page. Care to open a bug report there?

[cmelchior]

I tried to reproduce it in a simple project here: https://github.com/cmelchior/ccache-action-bug/pull/1 but I cannot reproduce the behavior there and the action seems to work as you describe.

So right now my guess is that it is a problem with how the Android NDK build interacts with ccache, but I need to investigate further.

You can probably close this issue, I will open a new issue if I discover the root cause.

Since React Native also uses the Android NDK, maybe they are seeing the same as I am, so I will wait a little before opening an issue with them as their fix actually seem to work even if it is suboptimal.

[whoozle]

Just hit the same issue on the private project (just pure gh actions, no docker, matrix.os ubuntu.latest).

Ccache computes compiler hash by including compiler's mtime/size, for compiler_check=content, it's reading compiler binary instead. This would slow down cache, but if you're not using

I think you might provide and option to switch compiler_check to content instead.

[mathiaswagner]

I hit a similar issues seeing 0% direct hits and then changing to content it goes to 100% direct hits. Overhead seems to be negligible as compile times for individual files are long enough that hashing the compiler binary is not an issue. Not sure why mtime does not work for a default Ubuntu 22.04 gcc 12 but as setting it to content fixes it ... (Weirdly I had better hit rates with clang but also not direct but just preprocessed ...).

[hendrikmuhs]

@mathiaswagner do you have a link or is this a private repo?

default Ubuntu 22.04 gcc 12

The default gcc in 22.04 is gcc 11. How do you install gcc 12? Do you "apt install" it as part of your build?

It is strange that the mtime is different for every run, to further debug this, can you e.g. run stat /usr/bin/gcc-12 and share the results?

just installed it, mtime is set according to package creation, not installation date:

stat /usr/bin/gcc-12 
  File: /usr/bin/gcc-12 -> x86_64-linux-gnu-gcc-12
  Size: 23              Blocks: 0          IO Block: 4096   symbolic link
Device: fd00h/64768d    Inode: 49042184    Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-07-19 08:13:12.000000000 +0200
Modify: 2022-05-13 13:11:55.000000000 +0200
Change: 2023-07-19 08:13:19.045350880 +0200
 Birth: 2023-07-19 08:13:19.013350315 +0200