Memory usage increased significantly in 1.9.10

[sgrimm]

Describe the bug Dokka 1.9.10 requires over 30% more memory than 1.9.0.

Expected behaviour The memory requirements stay roughly the same across patch versions.

To Reproduce Example: https://github.com/terraware/terraware-server/

This project's gradle.properties sets the maximum JVM heap size to 4096MB.

./gradlew dokkaHtml succeeds on both x86 and ARM systems using Dokka 1.9.0.

Upgrade to Dokka 1.9.10 as in https://github.com/terraware/terraware-server/pull/1420 and ./gradlew dokkaHtml fails with a "Java heap space" error.

Editing gradle.properties to set the maximum heap size to 5.5GB (-Xmx5632m) causes Dokka to run successfully.

Dokka configuration Configuration of dokka used to reproduce the bug

tasks.withType<DokkaTask>().configureEach {
  dokkaSourceSets {
    named("main") {
      outputDirectory = file("docs/dokka")
      moduleName = "Terraware Server"
      includes.from(fileTree("src/main/kotlin") { include("**/Package.md") })
      sourceLink {
        localDirectory = file("src/main/kotlin")
        remoteUrl =
            URI("https://github.com/terraware/terraware-server/tree/main/src/main/kotlin").toURL()
        remoteLineSuffix = "#L"
      }
    }
  }
}

Installation

• Operating system: macOS 13.5.2 (ARM)
• Build tool: Gradle 8.4
• Dokka version: 1.9.10
• JVM version: Corretto 20.0.2.9.1

Additional context The example project uses a Dokka plugin to add Mermaid support, but commenting out that plugin in build.gradle.kts doesn't help.

Are you willing to provide a PR? Yes, but I have no idea where to start looking.

[ianbrandt]

I just had to raise my Gradle Daemon max heap from 11G to 16G due to Dokka 1.9.20 OutOfMemoryErrors:

[ianbrandt]

Just an FYI, I noticed this past comment on #1405 from @Kordyjan:

Dokka is indeed memory-intensive in some parts, which are not yet optimized, but it should only affect heap space.

[whyoleg]

Hey @sgrimm and @ianbrandt! In Dokka 2.0.0 we slightly improved memory usage, could you check how it affects your projects? For reference, the PR with some measurements is #3800 Also, one more fix with potential improvements to memory usage will be available in next version (#4008)

[ianbrandt]

Hi @whyoleg,

A colleague looked into upgrading our project from Dokka 1.9.20 to 2.0.0. Unless there was a misunderstanding, they found that the dokka(project(...)) dependencies in our report aggregation project didn't result in transitive project dependencies being included (like they are with Gradle's Test Report Aggregation Plugin and JaCoCo Report Aggregation Plugin).

Is that correct regarding how Dokka 2.0.0 documentation aggregation currently works?

We have over 450 subprojects in our build. Maintaining a flat list of the ones that should be included in documentation aggregation wouldn't be very practical for us. I believe any scripting to try to accumulate subprojects that have the Dokka plugin applied may end up violating the requirements for Gradle's upcoming Project Isolation and Parallel Configuration features. We wouldn't want to do that because parallel configuration is probably going to offer us significant build performance gains.

[whyoleg]

@adam-enko, do you remember that we discussed resolving transitive dependencies before, and I was advocating that it would not really help? :) Looks like I was wrong. Do you think that we will be able to provide the same level of support for transitive dependencies as Gradle?

Coming back to the original question:

Is that correct regarding how Dokka 2.0.0 documentation aggregation currently works?

Yes, Dokka 2.0.0 with DGPv2 works this way. Let's wait for @adam-enko to return from vacation and hope he will have some insights on whether it's possible and how hard it would be to support transitive dependencies for Dokka.

I believe any scripting to try to accumulate subprojects that have the Dokka plugin applied may end up violating the requirements for Gradle's upcoming https://github.com/gradle/build-tool-roadmap/issues/76 and https://github.com/gradle/build-tool-roadmap/issues/88 features.

So far, we haven't found a solution to somehow automate this inclusion in a way that will support both Gradle features and all types of projects. But in specific cases, you can try to do something like this in the aggregation module:

dependencies {
  // list of all modules which should not be included in aggregated Dokka HTML
  val excluded = setOf(
    ":m1",
    ":m2",
  )
  // iterating over subprojects is safe: https://docs.gradle.org/current/userguide/isolated_projects.html#build_logic_constraints
  subprojects.forEach {
    if (it.path !in excluded) dokka(it)
  }
}

Depending on how your project is configured, you might want to change subprojects to something else, still compatible with Isolated Projects.

Also, @ianbrandt, JIC, currently you can update to Dokka 2.0.0 without migrating to the new Gradle Plugin, but the new Dokka Gradle Plugin (DGPv2) will be enabled by default in future versions.

[adam-enko]

@adam-enko, do you remember that we discussed resolving transitive dependencies before, and I was advocating that it would not really help? :) Looks like I was wrong. Do you think that we will be able to provide the same level of support for transitive dependencies as Gradle?

It's been a while so I only vaguely remember!

(Just to check, because I'm a bit lost because this issue is about performance, but you're asking now about making it easier to aggregate Dokka modules, which is unrelated to performance, right?)

We discussed making aggregation easier in kotlinlang #dokka recently. There are a few workarounds described in the thread. https://kotlinlang.slack.com/archives/C0F4UNJET/p1736763794337139?thread_ts=1736521419.942349&cid=C0F4UNJET (archive, if you don't have Slack.)

Handling easier aggregation is a missing feature in Gradle (thumbs-up this issue https://github.com/gradle/gradle/issues/29403). Until then, we could try and update DGP to make aggregation easier, but I think it's always going to be difficult/awkward/flawed until Gradle handles this use-case natively.

Until then, the dependencies { subprojects.forEach { dokka(it) } } should work and be compatible with Isolated Projects (reading immutable data like the project path is fine.) Dokka should filter out non-compatible projects without DPG, or they can be excluded manually. Please report an issue if there are problems.

Here are some of the things DGP could look at.

• Dokka could provide a BuildService, although these have a flaw that can be very awkward to workaround. We could make a settings plugin - but these also have flaws.

  • Gradle plugin portal just displays them as regular Project plugins,
  • they aren't commonly used so may be confusing requiring more documentation,
  • and settings plugins cannot interact with other plugins, unless the other plugins are added to the settings classpath, which could require a significant refactoring of buildscripts - would this be worth it just to make Dokka aggregation easier. There are no critical flaws, but I am reluctant and sceptical.

  Perhaps we could introduce a more limited Dokka aggregation settings plugin. Very few projects have a huge amount of subprojects, so we could 'hide' it from most users. Don't publish it to GPP to avoid confusion.

  I would like to turn Dokkatoo into a hub for 'community supported' Dokka tools. We could add a dokka-aggregation settings plugin there.

• We could make the dokka Configuration extend from 'regular' Kotlin dependencies, so the dependencies get propagated transitively.

  For example, if a build has subprojects :docs, :lib-A, :lib-B, :lib-C, and :app. If :app depends on all the :lib-* projects (e.g. dependencies { implementation(project(":lib-A")) }, then subproject :dokka would just need to define dependencies { dokka(project(":app")) }, and the :lib-* subprojects would be shared transitively.

  Pro: easy, and suits most use cases. Cons: Excluding projects becomes difficult and hard to configure.

[whyoleg]

In Dokka 2.1.0-Beta, we have again made several improvements in the Dokka generator memory consumption. I will close this specific issue, as many things have changed between 1.9.0 and 2.1.0-Beta inside Dokka, and the original project from the issue no longer uses Dokka.

@ianbrandt, if you will still have problems with memory usage on Dokka 2.1.0-Beta+, please create a separate issue.