[REQUEST] include sccache in the SDK, make easy to use with AGP

[khirowatari]

NDK_CCACHE feature is very effective for build speed. For Mac OS X / Ubuntu, I can easy to install ccache by homebrew / apt. But on windows, I can't find convenient binary package. ( I found many of old binary or broken binary) Finally I build from source by mingw.

It's inefficient for many windows developer. I want google to provide prebuilt ccache for windows.

[rcdailey]

Huge bump here... but is ccache with NDK likely not going to happen at this point, seeing as this ask was 2 years ago and was marked unplanned?

[enh]

given that https://ccache.samba.org/ has moved to GPLv3, it seems unlikely.

this is probably broken in r18 too, since GCC is gone now:

        # Unfortunately, we can just do CC="$NDK_CCACHE $CC" because some
        # configure scripts are not capable of dealing with this properly
        # E.g. the ones used to rebuild the GCC toolchain from scratch.
        # So instead, use a wrapper script
        CC=$NDK_BUILDTOOLS_ABSPATH/ndk-ccache-gcc.sh
        CXX=$NDK_BUILDTOOLS_ABSPATH/ndk-ccache-g++.sh

seems like goma (https://chromium.googlesource.com/infra/goma/client/) supports Mac and Windows, but there's no publicly available server?

[rcdailey]

I got a tip from Jomo at Google regarding an issue related to build performance when using CMake. He had suggested looking into ccache to improve build performance where static libraries shared between multiple Gradle modules are redundantly rebuilt. However, I was not able to find an easy way to incorporate ccache into CMake + Gradle on Windows platform. I think NDK support for that would be essential, especially for general usability, and given that that likely won't happen, it doesn't seem like ccache is a viable option for that specific issue.

Thanks for your response @enh!

[DanAlbert]

He had suggested looking into ccache to improve build performance where static libraries shared between multiple Gradle modules are redundantly rebuilt.

Fixing gradle so this isn't necessary sounds like a better investment that getting ccache into the NDK anyway.

[nathpete-msft]

@rcdailey here's a stripped down example of how we have ccache running as part of our Gradle builds on Windows. This is just showing the bits specific to Windows, but it's fairly easy to make more general as our CMake scripts are shared between Linux and macOS builds (including via Xcode). Since there is no official build of ccache for Windows, we use one installed via mingw64. It works reasonably well standalone (outside mingw64), but a few features don't work on Windows (hardlinks being one example). In addition to this, we also set the CMake build staging directory to be the same for all Gradle modules, so that we reuse outputs for static libs that are shared across modules. Unfortunately, Gradle also tries to "help" here by re-routing the final library outputs, making each invocation to appear distinct causing Gradle to again "helpfully" blow away the staging directory. You can work around this by hooking into the generateJsonModel* tasks with your own dependent task that manually patches up android_gradle_build.json on the fly to correct the path automatically set for CMAKE_LIBRARY_OUTPUT_DIRECTORY to be rooted in the staging directory you want.

CMakeLists.txt:

project(foo)

set(CCACHE_PROGRAM "path/to/ccache.exe")
set(C_LAUNCHER "${CCACHE_PROGRAM}")
set(CXX_LAUNCHER "${CCACHE_PROGRAM}")

set(c_launcher_template_path "ccache/launch-c.bat.in")
set(cxx_launcher_template_path "ccache/launch-cxx.bat.in")
set(c_launcher_path "build/launch-c.bat")
set(cxx_launcher_path "build/launch-cxx.bat")

configure_file("${c_launcher_template_path}" "${c_launcher_path}")
configure_file("${cxx_launcher_template_path}" "${cxx_launcher_path}")

set(CMAKE_C_COMPILER_LAUNCHER "${c_launcher_path}")
set(CMAKE_CXX_COMPILER_LAUNCHER "${cxx_launcher_path}")

launch-c.bat.in:

@echo off
setlocal
    set CCACHE_CPP2=true
    start "" /b "${C_LAUNCHER}" %*
endlocal

launch-cxx.bat.in:

@echo off
setlocal
    set CCACHE_CPP2=true
    start "" /b "${CXX_LAUNCHER}" %*
endlocal

Credit for the general ccache+CMake approach goes to @audiofanatic from his blog post on the topic.

[craigscott-crascit]

@nathpete-msft For MSVC, there's also clcache which has a similar aim to ccache. It may be possible to use clcache in a similar way with the launcher-based approach, but I can't confirm whether this works or not. Not really relevant for NDK which uses gcc or clang though I guess.

@enh ccache isn't restricted to GCC, it works for clang as well. If you use the launcher-based approach like that above, it will use ccache and still let you specify the compilers separately via a toolchain file, etc. It shouldn't be necessary to hard-code the compiler choice

[enh]

@audiofanatic my GCC reference actually talking about building the NDK's GCC. https://android-review.googlesource.com/c/platform/ndk/+/718560 removes that, since we don't use it. this doesn't affect the ndk-build/cmake ccache support. but we won't be distributing GPLv3 ccache binaries.

[rcdailey]

I don't see a need for the intermediate script. With CMake 3.12, I let CMake manage usage of ccache and it uses the correct, expected compiler. Here is what I do:

# Do not issue compiler warnings when unrecognized options are specified. This prevents
# compilation failures due to treating warnings as errors. This is important in a ccache
# context, per the following issues:
#
# * http://petereisentraut.blogspot.com/2011/05/ccache-and-clang.html
# * https://github.com/android-ndk/ndk/issues/171
string( APPEND CMAKE_CXX_FLAGS " -Qunused-arguments" )
string( APPEND CMAKE_C_FLAGS " -Qunused-arguments" )

# ccache allows caching of object files for speedier builds. This is especially important with
# Gradle, as of v3.3 of Android Studio, they still do not consolidate CMAKE_BINARY_DIR builds
# between modules, which causes redundant builds of object files for shared/common static
# libraries. See: https://issuetracker.google.com/issues/67493964
find_program( CCACHE ccache )
if( CCACHE )
    message( STATUS "Using ccache: ${CCACHE}" )
    set( ENV{CCACHE_BASEDIR} ${CMAKE_BINARY_DIR} )
    set( CMAKE_CXX_COMPILER_LAUNCHER ${CCACHE} )
    set( CMAKE_C_COMPILER_LAUNCHER ${CCACHE} )
endif()

On Windows, however, the same logic fails:

C:/msys64/usr/bin/ccache.exe C:\android\android-ndk-r17b\toolchains\llvm\prebuilt\windows-x86_64\bin\clang.exe --target=armv7-none-linux-androideabi --gcc-toolchain=C:/android/android-ndk-r17b/toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64 --sysroot=C:/android/android-ndk-r17b/sysroot -DANDROID -DSQLITE_DISABLE_LFS -IE:/code/frontend2/source/Core/ThirdParty/sqlite/. -isystem C:/android/android-ndk-r17b/sysroot/usr/include/arm-linux-androideabi -D__ANDROID_API__=15 -g -DANDROID -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -mthumb -Wa,--noexecstack -Wformat -Werror=format-security -Qunused-arguments -Os -DNDEBUG -Wno-inconsistent-missing-override -MD -MT Core/ThirdParty/sqlite/CMakeFiles/sqlite.dir/sqlite3.c.o -MF Core\ThirdParty\sqlite\CMakeFiles\sqlite.dir\sqlite3.c.o.d -o Core/ThirdParty/sqlite/CMakeFiles/sqlite.dir/sqlite3.c.o -c E:/code/frontend2/source/Core/ThirdParty/sqlite/sqlite3.c ccache: error: Could not find compiler "C:\android\android-ndk-r17b\toolchains\llvm\prebuilt\windows-x86_64\bin\clang.exe" in PATH

I guess the mingw64 version of ccache doesn't like the absolute path or something? I'm not sure what isn't working. Anyone know how to fix this problem so I can use mingw ccache with NDK?

[rcdailey]

ccache seems to have problems with backslashes in the paths. I've tried everything I can to get it working on Windows, but CMake injects backslashes for a few of the parameters. Not sure how anyone ever got ccache working on Windows...

[nathpete-msft]

@rcdailey, that's interesting, because it's working fine for me, regardless of back/forward slashes:

E:/tools/ccache/ccache.exe E:\linked\android\ndk\Android.Ndk.17.1.4828580\toolchains\llvm\prebuilt\windows-x86_64\bin\clang.exe --target=aarch64-none-linux-android --gcc-toolchain=E:/linked/android/ndk/Android.Ndk.17.1.4828580/toolchains/aarch64-linux-android-4.9/prebuilt/windows-x86_64 --sysroot=E:/linked/android/ndk/Android.Ndk.17.1.4828580/sysroot -Iexternal -I../../../../external -I../../../../external/sqlite -isystem E:/linked/android/ndk/Android.Ndk.17.1.4828580/sysroot/usr/include/aarch64-linux-android -D__ANDROID_API__=21 -g -DANDROID -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -Wa,--noexecstack -Wformat -Werror=format-security -Wall -Wextra -Wno-unused-command-line-argument -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-comment -Werror -pthread -fuse-ld=gold.exe -O0 -fno-limit-debug-info -g -gdwarf-4 -DDEBUG -fPIC -MD -MT external/CMakeFiles/libsqlite3.dir/sqlite/sqlite3.c.o -MF external\CMakeFiles\libsqlite3.dir\sqlite\sqlite3.c.o.d -o external/CMakeFiles/libsqlite3.dir/sqlite/sqlite3.c.o -c E:\proj\cdp\external\sqlite\sqlite3.c

It looks like the only real difference between ours is that I'm using the launcher scripts, which really shouldn't matter. Just to be sure this wasn't due to something shell specific, I just tried building in CMD, PowerShell, and MinGW64, but the build worked fine across all three. I'm not sure what else could causing the issue you're running into.

Also, you're right that you probably don't need to intermediate launcher script. We're using it to help abstract away differences between generators, since this CMake script is shared for Android, iOS, and Linux builds.

[fplk0]

@rcdailey for you it doesn't work because you're using msys2 ccache (i.e. C:/msys64/usr/bin/ccache.exe ), I'm using ccache from mingw64 and it works ( in by build.gradle it's set to ndk.ccache=C:/msys64/mingw64/bin/ccache.exe ). You can easily google how to install package for mingw as well. ...However, with mingw version I'm having another problem - for me it complains about ccache: error: Failed to create temporary file for CMakeFiles/<removed>/C_/<path_removed>/<path_removed>/Source/<path_removed>.cpp.o: No such file or directory Anyone saw that before?

[DanAlbert]

FYI we're investigating shipping sccache in the SDK (https://github.com/android/ndk/issues/1790 shows how to use it yourself, but definitely read the readme in the sccache project that explains why what I've done there is not suitable for production). Retitling this FR to cover that since there's no point in having both: sccache is strictly better afaik.