GraphVisitor.usesResByRes is extremely slow. (Stuck in computeActualUsageDebug for many minutes)

[juliocbcotta]

trafficstars

Build scan link

Plugin version 1.31.0

Gradle version 8.6

JDK version 21.0.2 (BellSoft 21.0.2+14-LTS)

(Optional) Kotlin and Kotlin Gradle Plugin (KGP) version 1.9.23

(Optional) Android Gradle Plugin (AGP) version 8.3.2

Describe the bug The task computeActualUsageDebug gets stuck for a long time. This module contains only android resources(images, vector drawables, translations and theme definitions), no code, so I don't understand why it takes so long. It has around 250 files on it and it takes around 5-10 min to run in a M1 machine doing no other heavy compilation. The same task in other big modules with lots of code and dependencies takes way less time. I checked the memory usage in the gradle daemon and it seemed fine (not a lot of GCs happening).

Is there any debug flag I could pass to the build that could give me infos about what is happening?

Expected behavior To run the analysis faster?

Additional context My gradle.properties has, if I remove this flag, computeActualUsageRelease also gets stuck. dependency.analysis.android.ignored.variants=release

[autonomousapps]

Thanks for the issue. My first thought was memory, and it still might be the case. Do you have a build scan you can share?

You could use gradle-profiler to profile the build for CPU and memory usage and see what seems to be taking so long. I would run just the task in question to keep the profile useful. I'm happy to take a look if you're willing to share the reports. I have done so in the past.

[juliocbcotta]

hey @autonomousapps , I sent you some data by email a few days ago using your email listed on your profile. Did you check that? I will try to find time for a better analyses if it is not enough.

[ParaskP7]

👋 @autonomousapps !

I too I am currently experiencing similar issues when running the buildHealth task. I am working on this woocommerce-android open source project and trying to set-up the plugin. It seems to be taking too long, the computeActualUsage* per se. Previously, about a year ago, when I tried running the plugin again, everything worked as expected (much faster).

You will find these changes within the build/dependency-analysis-android-gradle-plugin branch, just in case you would like to clone/pull the repo/branch and experienced it for yourself.

[!NOTE] Pulling this branch you will also notice that in order to make buildHealth more efficient, I had to exclude all other variants using a custom configuration (commit). This is because the dependency.analysis.android.ignored.variants=variant1,variant2,variant3 flag is not working as expected for this project. I'll leave another comment about it within #945.

---

Build scan link (GRADLE_PROFILER) https://gradle.com/s/j73i5752pltng

Plugin version 1.32.0

Gradle version 8.2.1

JDK version openjdk 17.0.11 2024-04-16

(Optional) Kotlin and Kotlin Gradle Plugin (KGP) version 1.9.22

(Optional) Android Gradle Plugin (AGP) version 8.1.0

---

You could use gradle-profiler to profile the build for CPU and memory usage and see what seems to be taking so long.

I am also providing you with a profile.log report, which I generated running the below gradle-profile command, you'll find the build scans links within:

gradle-profiler --profile buildscan --project-dir ../woocommerce-android buildHealth

profile.log

---

I hope all this info will somewhat help you with your investigation, and thank you! 🙏

[autonomousapps]

hey @autonomousapps , I sent you some data by email a few days ago using your email listed on your profile. Did you check that? I will try to find time for a better analyses if it is not enough.

I did see the email, thanks!

[ParaskP7]

👋 @autonomousapps !

To follow-up on this, today I was trying older versions on the plugin and the problem persisted till version 1.30.0.

Now, that doesn't mean that version 1.29.0 worked, actually this version produced a could not determine the dependencies of task kind of failure (see dependency-analysis-failure-v1.29.0.txt).

But then, version 1.28.0 worked as expected, with the buildHealth task no longer being stuck in computeActualUsage phase for minutes. 🤔

---

I hope the above will help you somehow and many thanks for all the work with this plugin! 🥇

[autonomousapps]

--profile buildscan isn't super userful for investigating this, unfortunately. I need to know what's happening in the CPU -- the build scan is too coarse grained. Can someone run gradle-profiler --profile async-profiler on their project?

[ParaskP7]

👋 @autonomousapps and apologies for the wait, you'll find a .zip file with the profile-out when using gradle-profiler --profile async-profiler on WCAndroid. 🍀

profile-out.zip

Let me know if you need anything else. 🙏

[autonomousapps]

@ParaskP7 thanks! Could you do that again, but this time only target the computeActualUsageDebug task (or whichever variant of it you're having trouble with)?

Looking at the data you sent, the CPU flamegraph is dominated by Gradle infrastructure. I think if we focus on just the specific task, then it might be easier to see where the issue is.

[ParaskP7]

👋 @autonomousapps sure thing, you'll find another .zip file with the profile-out when using gradle-profiler --profile async-profiler on WCAndroid. 🍀

profile-out.zip

Let me know if you need anything else. 🙏

FYI: I used the computeActualUsageJalapenoDebug.

PS: Below all the candidates when you configure the project to ignore (*) everything else but the debug build type and the jalapenoDebug product flavor:

• computeActualUsageDebug (this candidate is actually fast) ✅
• computeActualUsageDebugTest (test candidate, ignored) ✔️
• computeActualUsageDebugandroidTest (test candidate, ignored) ✔️
• computeActualUsageJalapenoDebug (this one is the slow one) ❌
• computeActualUsageJalapenoDebugTest (test candidate, ignored) ✔️
• computeActualUsageJalapenoDebugandroidTest (test candidate, ignored) ✔️

(*) dependency.analysis.android.ignored.variants=release,vanillaDebug,vanillaRelease,wasabiDebug,wasabiRelease,jalapenoRelease

[autonomousapps]

@ParaskP7 I'd like to make (hopefully just) one more request 😅

When I look at the log included in your zip, I see this

> Task :WooCommerce-Wear:computeActualUsageJalapenoDebug UP-TO-DATE

And that corroborates what I'm seeing in the flamechart, which is mostly Gradle infrastructure. Could you run this profiler one more time, either with ./gradlew :WooCommerce-Wear:computeActualUsageJalapenoDebug --rerun or ./gradlew :WooCommerce-Wear:clean :WooCommerce-Wear:computeActualUsageJalapenoDebug --no-build-cache. We need the clean --no-build-cache (or --rerun) to ensure the task actually runs.

[ParaskP7]

👋 @autonomousapps and apologies for the delay, you'll now find a couple of .zip files with the profile-out when using gradle-profiler --profile async-profiler on WCAndroid. 🍀

scenario.1

compute_actual_usage {
    tasks = [":WooCommerce-Wear:computeActualUsageJalapenoDebug"]
    gradle-args = ["--rerun-tasks"]
}

profile-out.zip

PS: As expected the above run was quite fast, it targeting the wear module and all. As such, I run the below as well. Hope it helps. 🤞

scenario.2

compute_actual_usage {
    tasks = ["computeActualUsageJalapenoDebug"]
    gradle-args = ["--rerun-tasks"]
}

profile-out.zip

Let me know if you need anything else. 🙏

[autonomousapps]

This is perfect -- a smoking gun. Take a look at the slow version:

Clearly dominated by GraphVisitor.usesResByRes. By contrast, here's the fast version:

The little bit highlighted in purple near the middle is GraphVisitor.usesResByRes. Big difference!

I never would have been able to just figure this out based on first principles. Having the actual profile from a real project is critically important. Thank you!

[autonomousapps]

For the record, the code in question has been around for 2 years :)

[autonomousapps]

I just merged a change that should improve the performance of that algorithm. Could someone test the latest snapshot and verify? Please wait a few hours from this message posting.

[ParaskP7]

This is awesome @autonomousapps , you rock, thanks for all this work, will test the latest snapshot asap and report back here! 🙇 ❤️ 🚀

[ParaskP7]

👋 @autonomousapps !

I did test with the latest snapshot (1.32.1-SNAPSHOT) via https://oss.sonatype.org/content/repositories/snapshots/, but, unfortunately, I am still seeing the same delay... 🤔 Maybe I am using the wrong snapshot? 🤷

Here the .zip files with the profile-out when using gradle-profiler --profile async-profiler on WCAndroid using this latest snapshot.

profile-out.zip

Let me know if I am using the wrong snapshot or doing something wrong. 🙏

[autonomousapps]

It's possible my change just wasn't sufficient. The algorithm is inherently very complex and I tried a shortcut to make it faster without attempting a fundamental rethink. I think it's roughly O(N^3), where N is the number of resource symbols in a project. So... that's a lot.

Unfortunately, I have limited time available for this kind of work. It's non-trivial (I won't say "impossible"), and also non-essential (for some value of that term). I may be open to further work on a sponsored basis.

Re-opening for now.

[ParaskP7]

Thanks for all your work on that Tony, and for all the details you shared, yea, this problem doesn't look to be trivial, I very much appreciated all the effort here, thanks for re-opening, FYI, I am a big fan of this tool you created! 🙏 ❤️ 🙇